| 2014-09-18
      
      § | 
    
  | 19:57 | <Jeff_Green> | iridium.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:54 | <Jeff_Green> | magnesium.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:50 | <Jeff_Green> | sodium.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:48 | <reedy> | Synchronized php-1.24wmf22/extensions/Flow/: (no message) (duration: 00m 16s) | [production] | 
            
  | 19:45 | <Jeff_Green> | iodine.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:44 | <Jeff_Green> | polonium.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:35 | <Jeff_Green> | lead.wm.o exim conf checked, puppet reenabled | [production] | 
            
  | 19:22 | <reedy> | Synchronized php-1.24wmf22: (no message) (duration: 00m 57s) | [production] | 
            
  | 19:16 | <Jeff_Green> | disabling puppet on polonium, lead, sodium, iridium, magnesium, and iodine to monitor rollout of https://gerrit.wikimedia.org/r/155753 | [production] | 
            
  | 19:05 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: rest of group0 to 1.24wmf22 | [production] | 
            
  | 19:01 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf21 | [production] | 
            
  | 18:59 | <bblack> | restarting apache on fenari | [production] | 
            
  | 18:49 | <reedy> | Finished scap: testwiki to 1.24wmf22 and build l10n cache (duration: 30m 23s) | [production] | 
            
  | 18:44 | <Jeff_Green> | testing exim configuration change on lead.wm.o | [production] | 
            
  | 18:19 | <reedy> | Started scap: testwiki to 1.24wmf22 and build l10n cache | [production] | 
            
  | 17:49 | <reedy> | Started scap: testwiki to 1.24wmf22 and build l10n cache | [production] | 
            
  | 17:08 | <cmjohnson1> | replacing failed disk es1005 | [production] | 
            
  | 17:05 | <yurik> | Finished scap: (no message) (duration: 23m 26s) | [production] | 
            
  | 16:43 | <yurikR> | yurik scaping zero - partner needs an l10n message asap | [production] | 
            
  | 16:42 | <yurik> | Started scap: (no message) | [production] | 
            
  | 15:38 | <hashar> | restarting Zuul just to be safe | [production] | 
            
  | 15:06 | <anomie> | Synchronized php-1.24wmf21/resources/src/mediawiki.action/mediawiki.action.view.redirectPage.css: SWAT: mediawiki.action.view.redirectPage: Correct a CSS selector [[gerrit:161239]] (duration: 00m 23s) | [production] | 
            
  | 15:02 | <anomie> | Synchronized php-1.24wmf21/extensions/Wikidata/: SWAT: Update Wikidata to fix broken xml api output [[gerrit:161232]] (duration: 00m 38s) | [production] | 
            
  | 11:40 | <apergos> | forgot to log this earlier: manually started salt minion on radon, elastic1015, searchidx1001, it wasn't running there | [production] | 
            
  | 09:00 | <godog> | updated authdns to 0c2225d | [production] | 
            
  | 08:56 | <springle> | xtrabackup clone db1016 to db2010 | [production] | 
            
  | 07:48 | <godog> | re-enabled icinga notifications for ms-be1001 | [production] | 
            
  | 07:10 | <bblack> | removing pybal cfg "eqiad/misc_web_https" (unused now, https://gerrit.wikimedia.org/r/161183) | [production] | 
            
  | 06:53 | <bblack> | removing pybal cfg "esams/wikimedialbsecure" (unused, points at maerlant) | [production] | 
            
  | 06:47 | <bblack> | removing pybal symlink "$site/ipv6", also unused (old ipv6 protoproxying) | [production] | 
            
  | 06:45 | <bblack> | removing pybal symlink "$site/text-varnish", seems to be a remnant no longer in use | [production] | 
            
  | 04:21 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu Sep 18 04:20:56 UTC 2014 (duration 20m 55s) | [production] | 
            
  | 03:09 | <LocalisationUpdate> | completed (1.24wmf21) at 2014-09-18 03:09:44+00:00 | [production] | 
            
  | 02:53 | <yurik> | Synchronized wmf-config/CommonSettings.php: (no message) (duration: 01m 53s) | [production] | 
            
  | 02:52 | <yurikR> | yurik Fixing graph ext namespace name - otherwise get screen of WMF death on graph: ns visits | [production] | 
            
  | 02:36 | <LocalisationUpdate> | completed (1.24wmf20) at 2014-09-18 02:36:46+00:00 | [production] | 
            
  | 00:32 | <marktraceur> | Finished scap: [SWAT] Move things out of assets/ and into resources/assets/ (duration: 35m 28s) | [production] | 
            
  
    | 2014-09-17
      
      § | 
    
  | 23:57 | <marktraceur> | Started scap: [SWAT] Move things out of assets/ and into resources/assets/ | [production] | 
            
  | 23:47 | <marktraceur> | Synchronized wmf-config/InitialiseSettings.php: [SWAT] Enable Graph on metawiki and labswiki (duration: 00m 10s) | [production] | 
            
  | 23:42 | <marktraceur> | Synchronized php-1.24wmf21/extensions/Graph/: [SWAT] Update Graph to master (duration: 00m 08s) | [production] | 
            
  | 23:41 | <marktraceur> | Synchronized php-1.24wmf20/extensions/Graph/: [SWAT] Update Graph to master (duration: 00m 07s) | [production] | 
            
  | 23:35 | <marktraceur> | Synchronized php-1.24wmf21/extensions/MultimediaViewer/: [SWAT] Fix reuse dropdown message weirdness (duration: 00m 07s) | [production] | 
            
  | 23:29 | <marktraceur> | Synchronized php-1.24wmf20/extensions/MultimediaViewer/: [SWAT] Fix reuse dropdown message weirdness (duration: 00m 08s) | [production] | 
            
  | 23:10 | <marktraceur> | Synchronized php-1.24wmf21/extensions/UploadWizard/: [SWAT] Fix EventLogging schema declarations for UploadWizard (duration: 00m 11s) | [production] | 
            
  | 21:41 | <mutante> | fixing updates on planet feeds - file permissions | [production] | 
            
  | 21:11 | <manybubbles> | restarting rebuilding cirrus's enwiki index now that I've found the reason it wasn't working before - the new index was putting too many shards on an already full node and overwhelming it.  silly allocation algorithm!  thats a bad idea! | [production] | 
            
  | 21:08 | <yurik> | Synchronized php-1.24wmf21/extensions/ZeroPortal/: (no message) (duration: 01m 05s) | [production] | 
            
  | 20:19 | <godog> | rebooting ms-be1006 | [production] | 
            
  | 19:00 | <Krinkle> | jenkins-slave tmpfs on lanthanum was filling up (> 500MB). I purged tmp dbs for old jobs. We should get these purged automatically and also increase the size as 500MB is too little. | [production] | 
            
  | 18:59 | <robh> | disabled icinga alerts for ms-be1001, rebooting it to look at its raid bios settings for codfw deployment mirroring | [production] |