| 2017-07-12
      
      ยง | 
    
  | 21:28 | <demon@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: MinervaNeue config (duration: 00m 46s) | [production] | 
            
  | 21:27 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: MinervaNeue config (duration: 00m 47s) | [production] | 
            
  | 21:22 | <demon@tin> | Finished scap: Rebuilding l10n cache for new skin (duration: 38m 47s) | [production] | 
            
  | 20:43 | <demon@tin> | Started scap: Rebuilding l10n cache for new skin | [production] | 
            
  | 20:16 | <bsitzmann@tin> | Finished deploy [mobileapps/deploy@3f90bf1]: Update mobileapps to d30dae2 (T169930, T170225) (duration: 05m 00s) | [production] | 
            
  | 20:11 | <bsitzmann@tin> | Started deploy [mobileapps/deploy@3f90bf1]: Update mobileapps to d30dae2 (T169930, T170225) | [production] | 
            
  | 19:37 | <XioNoX> | adding ignore-l3-incompletes to all peering/transit interfaces - T163542 | [production] | 
            
  | 19:27 | <thcipriani@tin> | Synchronized php: promote php symlink group1 wikis to 1.30.0-wmf.9 (duration: 00m 45s) | [production] | 
            
  | 19:25 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.9 | [production] | 
            
  | 19:10 | <demon@tin> | Synchronized php-1.30.0-wmf.9/skins/MinervaNeue/: Latest code (duration: 00m 47s) | [production] | 
            
  | 19:09 | <demon@tin> | Synchronized php-1.30.0-wmf.7/skins/MinervaNeue/: Latest code (duration: 00m 48s) | [production] | 
            
  | 18:16 | <mobrovac@tin> | Finished deploy [recommendation-api/deploy@7fd10f2]: Use the domain parameter as the target language - T170439 (duration: 00m 40s) | [production] | 
            
  | 18:15 | <mobrovac@tin> | Started deploy [recommendation-api/deploy@7fd10f2]: Use the domain parameter as the target language - T170439 | [production] | 
            
  | 18:15 | <andrewbogott> | depooling labvirt1015, deleting a bunch of stuck contintcloud instances | [production] | 
            
  | 18:14 | <demon@tin> | Synchronized static/images/project-logos/: Fixing srwikiquote logos (duration: 00m 48s) | [production] | 
            
  | 17:52 | <chasemp> | labnodepool1001:~# sudo puppet agent --enable | [production] | 
            
  | 17:29 | <chasemp> | labnodepool1001:~# service nodepool stop | [production] | 
            
  | 17:21 | <_joe_> | rolling restart of pybal on low-traffic LVS in eqiad,codfw | [production] | 
            
  | 17:19 | <chasemp> | labnet1001:~# service nova-api restart | [production] | 
            
  | 17:15 | <chasemp> | labcontrol1001:~# service rabbitmq-server restart | [production] | 
            
  | 17:13 | <robh@puppetmaster1001> | conftool action : set/pooled=no; selector: name=wdqs2002.codfw.wmnet | [production] | 
            
  | 17:13 | <robh@puppetmaster1001> | conftool action : set/pooled=no; selector: name=wdqs2003.codfw.wmnet | [production] | 
            
  | 17:07 | <moritzm> | upgrading nginx on mwdebug* | [production] | 
            
  | 16:52 | <godog> | roll-restart and upgrade thumbor in eqiad | [production] | 
            
  | 16:47 | <oblivian@puppetmaster1001> | conftool action : set/pooled=true; selector: dnsdisc=recommendation-api | [production] | 
            
  | 16:36 | <oblivian@puppetmaster1001> | conftool action : set/pooled=yes; selector: service=recommendation-api,dc=eqiad | [production] | 
            
  | 16:15 | <oblivian@puppetmaster1001> | conftool action : set/pooled=yes; selector: service=recommendation-api,dc=codfw | [production] | 
            
  | 16:14 | <ema> | downgrade pybal to 1.13.6 on lvs1010 T82747 T154759 (1.13.7 throwing exceptions) | [production] | 
            
  | 16:09 | <godog> | upload thumbor 1.0-1 to install1002 | [production] | 
            
  | 16:06 | <mobrovac@tin> | Finished deploy [recommendation-api/deploy@eb2fef3]: (no justification provided) (duration: 00m 33s) | [production] | 
            
  | 16:06 | <ema> | lvs1006, lvs1010: upgrade pybal to 1.13.7 T82747 T154759 | [production] | 
            
  | 16:05 | <mobrovac@tin> | Started deploy [recommendation-api/deploy@eb2fef3]: (no justification provided) | [production] | 
            
  | 15:57 | <mobrovac@tin> | Finished deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001, take #3 - T165760 (duration: 00m 46s) | [production] | 
            
  | 15:56 | <mobrovac@tin> | Started deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001, take #3 - T165760 | [production] | 
            
  | 15:52 | <mobrovac@tin> | Finished deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001, take #2 - T165760 (duration: 00m 06s) | [production] | 
            
  | 15:52 | <mobrovac@tin> | Started deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001, take #2 - T165760 | [production] | 
            
  | 15:51 | <mobrovac@tin> | Finished deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001 - T165760 (duration: 00m 15s) | [production] | 
            
  | 15:51 | <mobrovac@tin> | Started deploy [recommendation-api/deploy@ed41fc4]: Initial deploy on canary scb2001 - T165760 | [production] | 
            
  | 15:07 | <ema> | lvs2006: upgrade pybal to 1.13.7 T82747 T154759 | [production] | 
            
  | 14:56 | <marostegui> | Run redact_sanitarium on db1069 and db1095 for maiwikimedia - T168788 | [production] | 
            
  | 14:55 | <marostegui> | Run redact_sanitarium on db1069 and db1095 for maiwikimedia - T169510 | [production] | 
            
  | 14:37 | <moritzm> | installing apache security updates on californium / horizon.wikimedia.org | [production] | 
            
  | 14:28 | <addshore@tin> | Synchronized wmf-config/extension-list-labs: [[gerrit:362393|Add Newsletter to extension-list]] PT1/2 (duration: 00m 46s) | [production] | 
            
  | 14:27 | <addshore@tin> | Synchronized wmf-config/extension-list: [[gerrit:362393|Add Newsletter to extension-list]] PT1/2 (duration: 00m 47s) | [production] | 
            
  | 14:17 | <jynus> | restarting labsdb1005 (toolsdb) | [production] | 
            
  | 14:14 | <madhuvishy> | Disable icinga notifications  and event handler checks for labsdb1005 | [production] | 
            
  | 14:07 | <marostegui> | Run redact_sanitarium on db1069 for dinwiki - T169193 | [production] | 
            
  | 13:23 | <marostegui> | Deploy alter table s1 - db1072 - T166204 | [production] | 
            
  | 13:21 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1072 - T166204 (duration: 00m 46s) | [production] | 
            
  | 13:17 | <zeljkof> | EU SWAT finished! | [production] |