| 2017-09-21
      
      § | 
    
  | 08:32 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet | [production] | 
            
  | 08:32 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet | [production] | 
            
  | 08:32 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:31 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:31 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:31 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:31 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:31 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:30 | <akosiaris@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:30 | <akosiaris@puppetmaster1001> | conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet | [production] | 
            
  | 08:25 | <mobrovac@tin> | Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s) | [production] | 
            
  | 08:22 | <mobrovac@tin> | Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests | [production] | 
            
  | 08:21 | <mobrovac@tin> | Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s) | [production] | 
            
  | 08:20 | <mobrovac@tin> | Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests | [production] | 
            
  | 08:19 | <mobrovac@tin> | Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s) | [production] | 
            
  | 08:14 | <mobrovac@tin> | Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests | [production] | 
            
  | 08:08 | <mobrovac@tin> | Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s) | [production] | 
            
  | 07:57 | <mobrovac@tin> | Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 | [production] | 
            
  | 07:31 | <_joe_> | restarted aphlict on phab1001 | [production] | 
            
  | 07:27 | <mobrovac@tin> | Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s) | [production] | 
            
  | 07:27 | <mobrovac@tin> | Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 | [production] | 
            
  | 07:09 | <ema> | bounce pybal on lvs1003 to clear stale alert T176388 | [production] | 
            
  | 06:59 | <ema> | bounce pybal on lvs1006 to clear stale alert | [production] | 
            
  | 06:55 | <ema> | bounce pybal on lvs1009 to clear stale alert | [production] | 
            
  | 03:24 | <bblack> | depooled cp4026 | [production] | 
            
  | 03:12 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s) | [production] | 
            
  | 03:06 | <l10nupdate@tin> | scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s) | [production] | 
            
  | 02:29 | <l10nupdate@tin> | scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s) | [production] | 
            
  | 00:41 | <eileen> | update civicrm from da833ac561f7586a0b64fbef2cdf2963bc4055c3 to fc56901efd386ca725bde971ae000d798e1e2fca | [production] | 
            
  | 00:23 | <ebernhardson@tin> | Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s) | [production] | 
            
  | 00:18 | <ebernhardson@tin> | Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s) | [production] | 
            
  | 00:09 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s) | [production] | 
            
  | 00:02 | <catrope@tin> | Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s) | [production] | 
            
  
    | 2017-09-20
      
      § | 
    
  | 23:49 | <catrope@tin> | Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s) | [production] | 
            
  | 23:47 | <catrope@tin> | Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s) | [production] | 
            
  | 23:40 | <mutante> | phabricator: aphlict (notification service) now running on prod server | [production] | 
            
  | 23:36 | <mutante> | phab1001 - re-enable puppet, run after follow-up fix, adding notification server config | [production] | 
            
  | 23:31 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s) | [production] | 
            
  | 23:29 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s) | [production] | 
            
  | 23:21 | <catrope@tin> | Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s) | [production] | 
            
  | 23:14 | <mutante> | phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group | [production] | 
            
  | 23:13 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s) | [production] | 
            
  | 23:11 | <mutante> | phab2001 - groupadd aphlict,temp adduser aphlict  to debug user creation issue | [production] | 
            
  | 22:22 | <ebernhardson> | elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb | [production] | 
            
  | 21:54 | <mutante> | phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001 | [production] | 
            
  | 21:45 | <eileen> | update civicrm from 0fb46767f0c86b3fc85c44e228fd7c1a7ae75e46 to da833ac561f7586a0b64fbef2cdf2963bc4055c3 | [production] | 
            
  | 21:09 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet | [production] | 
            
  | 20:58 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet | [production] | 
            
  | 20:40 | <bsitzmann@tin> | Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s) | [production] | 
            
  | 20:36 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet | [production] |