| 2016-03-31
      
      § | 
    
  | 15:17 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.19/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: SWAT: Ignore ResultSets that do not return pages [[gerrit:280669]] (duration: 00m 38s) | [production] | 
            
  | 14:01 | <volans> | manually running puppet on es2018 to double verify merged changes | [production] | 
            
  | 13:55 | <mutante> | hooft - reboot to pxe, one more time | [production] | 
            
  | 13:49 | <mutante> | netmon1001 - re-enabled puppet (was for torrus issue earlier) | [production] | 
            
  | 13:44 | <elukey> | forced puppet agent on netmon1001 | [production] | 
            
  | 13:27 | <ema> | repooling cp1044, upgraded to varnish 4 (T122880) | [production] | 
            
  | 13:17 | <moritzm> | installing rsync security updates | [production] | 
            
  | 13:09 | <ema> | depooling cp1044 for varnish 4 upgrade (T122880) | [production] | 
            
  | 12:02 | <moritzm> | removed rsync 3.10-2ubuntu0.1~wmf1 from carbon. this backport was only needed when the hosts used a mix of precise and trusty. this is no longer the case, so remove the backport and allow to use stock Ubuntu updates on precise again | [production] | 
            
  | 10:42 | <elukey> | stopping torrus-common on netmon1001 to try https://wikitech.wikimedia.org/wiki/Torrus#Deadlock_problem | [production] | 
            
  | 10:35 | <elukey> | restarted torrus-common on netmon1001 | [production] | 
            
  | 10:24 | <moritzm> | logstash on logstash100[1-3] is now using systemd | [production] | 
            
  | 09:50 | <moritzm> | disabled puppet on logstash1002/1003 (to activate the new logstash systemd unit in steps) | [production] | 
            
  | 09:00 | <volans> | Forced WriteBack cache policy mode on db1047 RAID | [production] | 
            
  | 06:11 | <Krinkle> | mwscript deleteEqualMessages.php --wiki zh_min_nanwiki (T45917) | [production] | 
            
  | 05:31 | <krinkle@tin> | Synchronized docroot/wikipedia.org/speed-tests/: (no message) (duration: 00m 33s) | [production] | 
            
  | 05:13 | <krinkle@tin> | Synchronized php-1.27.0-wmf.19/extensions/MobileFrontend/: Iaa5ed38c712b19e (duration: 00m 31s) | [production] | 
            
  | 05:10 | <krinkle@tin> | Synchronized php-1.27.0-wmf.18/extensions/MobileFrontend/: Iaa5ed38c712b19e (duration: 00m 42s) | [production] | 
            
  | 02:48 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.19) (duration: 10m 26s) | [production] | 
            
  | 02:27 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.18) (duration: 11m 52s) | [production] | 
            
  | 00:30 | <mutante> | argon - removing apache and config | [production] | 
            
  | 00:04 | <krinkle@tin> | Synchronized errorpages/: (no message) (duration: 00m 33s) | [production] | 
            
  
    | 2016-03-30
      
      § | 
    
  | 23:42 | <maxsem@tin> | Synchronized php-1.27.0-wmf.19/extensions/Kartographer/: (no message) (duration: 00m 27s) | [production] | 
            
  | 23:35 | <yuvipanda> | cleaned salt keys for analytics1017 and 1021 | [production] | 
            
  | 23:34 | <yuvipanda> | cleaned out puppet cert for analytics 1017 and 1021 | [production] | 
            
  | 23:10 | <maxsem@tin> | Synchronized php-1.27.0-wmf.18/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/280600/ (duration: 00m 31s) | [production] | 
            
  | 23:08 | <maxsem@tin> | Synchronized php-1.27.0-wmf.19/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/280601/ (duration: 00m 34s) | [production] | 
            
  | 23:06 | <mutante> | torrus - follow instruction for deadlock problem | [production] | 
            
  | 23:06 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280449/ (duration: 00m 43s) | [production] | 
            
  | 22:13 | <mutante> | smokeping/netmon - remove hooft/bast3001, restart service | [production] | 
            
  | 21:05 | <maxsem@tin> | Synchronized php-1.27.0-wmf.19/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#/c/280549/ (duration: 00m 40s) | [production] | 
            
  | 20:20 | <subbu> | finished deploying parsoid version a20ef276 | [production] | 
            
  | 20:15 | <Pchelolo> | finished update restbase to 1b52276f | [production] | 
            
  | 20:11 | <subbu> | synced code; restarted parsoid on wtp1004 as a canary | [production] | 
            
  | 20:10 | <Pchelolo> | starting update restbase to 1b52276f | [production] | 
            
  | 20:07 | <subbu> | starting parsoid deploy | [production] | 
            
  | 20:03 | <Pchelolo> | update restbase to 1b52276f canary deploy to restbase1005 | [production] | 
            
  | 19:48 | <mutante> | hooft revoking puppet cert, salt key, reboot into PXE | [production] | 
            
  | 19:43 | <mutante> | hooft is going to be reinstalled and renamed, affects esams bastion and ganglia during the install | [production] | 
            
  | 19:39 | <mutante> | hooft - updating root password | [production] | 
            
  | 19:07 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.19 | [production] | 
            
  | 18:58 | <Pchelolo> | doing a puppet run in codfw restbase nodes and performing a rolling restart of restbase | [production] | 
            
  | 18:55 | <Pchelolo> | finished restbase update to 9fe8676d | [production] | 
            
  | 18:20 | <Pchelolo> | starting restbase update to 9fe8676d | [production] | 
            
  | 17:46 | <hoo@tin> | Synchronized wmf-config/interwiki.php: Sync changes on meta (duration: 00m 42s) | [production] | 
            
  | 17:22 | <akosiaris> | restarting grrrit-wm | [production] | 
            
  | 16:24 | <ostriches> | gerrit: restarting to clean up mismatched jvm versions | [production] | 
            
  | 15:41 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.18/extensions/Echo/modules/ooui/mw.echo.ui.NotificationBadgeWidget.js: SWAT: Change threshold for survey invitation from 2 unread notifs to 1 [[gerrit:280369]] (duration: 00m 28s) | [production] | 
            
  | 15:32 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.19/extensions/Translate/resources/js/ext.translate.editor.js: SWAT: Fix regressions in insertables placement [[gerrit:280414]] (duration: 00m 27s) | [production] | 
            
  | 15:26 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.19/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: Try to avoid JS error [[gerrit:280389]] (duration: 00m 29s) | [production] |