| 2016-04-13
      
      § | 
    
  | 23:35 | <dereckson@tin> | Synchronized php-1.27.0-wmf.20/extensions/Echo/modules/ooui/mw.echo.ui.FooterNoticeWidget.js: Fixes for Echo ([[Gerrit:282714]] + [[Gerrit:282715]]) (duration: 00m 26s) | [production] | 
            
  | 23:25 | <Dereckson> | Ran mwscript updateArticleCount.php --wiki=plwikisource --update | [production] | 
            
  | 23:23 | <dereckson@tin> | Synchronized wmf-config/InitialiseSettings.php: Consider all pages as valid content articles on pl.wikisource (Task T131771, [[Gerrit:283349]]) (duration: 00m 35s) | [production] | 
            
  | 23:11 | <dereckson@tin> | Synchronized wmf-config/InitialiseSettings.php: Add signature edit button for the Comments namespace to ru.wikinews (Task T132241, [[Gerrit:283347]]) (duration: 00m 38s) | [production] | 
            
  | 22:23 | <hoo@tin> | Synchronized php-1.27.0-wmf.21/./extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.statementview.RankSelector.js: touch (duration: 00m 26s) | [production] | 
            
  | 22:11 | <hoo@tin> | Finished scap: Update Wikibase to master (wmf21) (duration: 32m 06s) | [production] | 
            
  | 21:39 | <hoo@tin> | Started scap: Update Wikibase to master (wmf21) | [production] | 
            
  | 21:09 | <bd808> | https://tools.wmflabs.org/sal missing entries since 2016-04-13T09:21. Needs to be backfilled | [production] | 
            
  | 20:20 | <bblack> | re-pooling ulsfo traffic T128424 | [production] | 
            
  | 19:56 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: (no message) | [production] | 
            
  | 19:24 | <ori@tin> | Synchronized php-1.27.0-wmf.21/includes/libs/IEUrlExtension.php: Live-hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 33s) | [production] | 
            
  | 19:23 | <ori@tin> | Synchronized php-1.27.0-wmf.20/includes/libs/IEUrlExtension.php: Live-hack IEUrlExtension::haveUndecodedRequestUri() to always return true (duration: 00m 33s) | [production] | 
            
  | 18:48 | <gehel> | activating cache headers for WDQS | [production] | 
            
  | 18:46 | <bblack> | lvs4002 - enable->run puppet post-reboot | [production] | 
            
  | 18:45 | <bblack> | lvs4001 - enable->run puppet post-reboot | [production] | 
            
  | 18:37 | <gehel> | activated maintenance page for wdqs1002 (data load in progress) | [production] | 
            
  | 18:16 | <bblack> | shutdown lvs400[12] | [production] | 
            
  | 18:06 | <mutante> | bast1001 back with jessie | [production] | 
            
  | 18:03 | <robh> | the cp sysetms in ulsfo will be rebooting into maint mode regularly for the next few hours.  I'll be scheduling downtime for each host as I get to them, but not echoing every cp host reboot in SAL | [production] | 
            
  | 18:03 | <robh> | the cp sysetms in ulsfo will be rebooting into maint mode regularly for the next few hours.  I'll be scheduling for each host as I get to them, but not echoing every cp host reboot in SAL | [production] | 
            
  | 18:00 | <bblack> | disable puppet, stop pybal on lvs400[12] (maint shutdown imminent, depooled from DNS since yesterday) | [production] | 
            
  | 17:46 | <robh> | lvs4003 rebooted and back online, lvs4004 offlining for maint. | [production] | 
            
  | 17:41 | <mutante> | tungsten stop and remove rsync package and config | [production] | 
            
  | 17:33 | <Krinkle> | mwscript deleteEqualMessages.php --wiki cywiki (T45917) | [production] | 
            
  | 17:27 | <mutante> | bast1001 - revoke puppet cert, delete salt key, reinstall with jessie | [production] | 
            
  | 17:15 | <mutante> | rebooting bast1001 into PXE | [production] | 
            
  | 17:14 | <robh> | lvs4003 going offline for maint (icinga has been silenced, i think ;) | [production] | 
            
  | 16:58 | <ori@tin> | Synchronized wmf-config/CommonSettings-labs.php: I5a0abcdc: Make MathML rendering default in labs (duration: 00m 39s) | [production] | 
            
  | 16:54 | <volans> | completed TLS upgrade for s1 - T111654 | [production] | 
            
  | 16:35 | <ottomata> | rebuilding raid1 array on aqs1001 after hot swapping sdh | [production] | 
            
  | 16:32 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1057 after TLS upgrade on s1 - T111654 (duration: 00m 26s) | [production] | 
            
  | 16:18 | <andrewbogott> | rebooting mw1139 — OOM | [production] | 
            
  | 16:11 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.20/extensions/CentralNotice: SWAT: Update CentralNotice [[gerrit:283205]] (duration: 00m 30s) | [production] | 
            
  | 16:10 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.21/extensions/CentralNotice: SWAT: Update CentralNotice [[gerrit:283206]] (duration: 00m 33s) | [production] | 
            
  | 15:56 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.20/extensions/WikidataPageBanner: SWAT: Attempt at fixing table of contents problem [[gerrit:282994]] (duration: 00m 28s) | [production] | 
            
  | 15:21 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.21/extensions/WikidataPageBanner: SWAT: Attempt at fixing table of contents problem [[gerrit:282995]] (duration: 00m 29s) | [production] | 
            
  | 15:10 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix typo in newikibooks namespaces [[gerrit:283183]] (duration: 00m 30s) | [production] | 
            
  | 14:53 | <volans> | start upgrading TLS for cross-dc replica on shards s1 - T111654 | [production] | 
            
  | 14:53 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1057 to upgrade TLS on s1 - T111654 (duration: 00m 26s) | [production] | 
            
  | 14:48 | <elukey> | Yarn nodemanager Xmx size bumped up from 1000m to 2048 on all the analytics* hosts to overcome the OutOfMemory errors. | [production] | 
            
  | 14:38 | <ottomata> | restarting hadoop-yarn-nodemanager on all hadoop worker nodes one by one to apply increase in heap size | [production] | 
            
  | 14:35 | <godog> | start cleanup on restbase100[569] - T128107 | [production] | 
            
  | 14:25 | <volans> | completed upgrading TLS for cross-dc replica on shards s2 - T111654 | [production] | 
            
  | 13:52 | <jzerebecki@tin> | Finished scap: php-1.27.0-wmf.21: Update Wikidata to wmf/1.27.0-wmf.21 (duration: 29m 56s) | [production] | 
            
  | 13:22 | <jzerebecki@tin> | Started scap: php-1.27.0-wmf.21: Update Wikidata to wmf/1.27.0-wmf.21 | [production] | 
            
  | 13:14 | <volans> | start upgrading TLS for cross-dc replica on shards s2 - T111654 | [production] | 
            
  | 11:46 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Reduce db1050 weight - T111654 (duration: 00m 30s) | [production] | 
            
  | 11:22 | <volans> | completed upgrading TLS for cross-dc replica on shards s6 and s7 - T111654 | [production] | 
            
  | 10:56 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1050 and db1041 after TLS upgrade - T111654 (duration: 00m 42s) | [production] | 
            
  | 09:21 | <volans> | start upgrading TLS for cross-dc replica on shards s6 and s7 - T111654 | [production] |