| 2017-09-21
      
      § | 
    
  | 07:31 | <_joe_> | restarted aphlict on phab1001 | [production] | 
            
  | 07:27 | <mobrovac@tin> | Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s) | [production] | 
            
  | 07:27 | <mobrovac@tin> | Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 | [production] | 
            
  | 07:09 | <ema> | bounce pybal on lvs1003 to clear stale alert T176388 | [production] | 
            
  | 06:59 | <ema> | bounce pybal on lvs1006 to clear stale alert | [production] | 
            
  | 06:55 | <ema> | bounce pybal on lvs1009 to clear stale alert | [production] | 
            
  | 03:24 | <bblack> | depooled cp4026 | [production] | 
            
  | 03:12 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s) | [production] | 
            
  | 03:06 | <l10nupdate@tin> | scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s) | [production] | 
            
  | 02:29 | <l10nupdate@tin> | scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s) | [production] | 
            
  | 00:41 | <eileen> | update civicrm from da833ac561f7586a0b64fbef2cdf2963bc4055c3 to fc56901efd386ca725bde971ae000d798e1e2fca | [production] | 
            
  | 00:23 | <ebernhardson@tin> | Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s) | [production] | 
            
  | 00:18 | <ebernhardson@tin> | Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s) | [production] | 
            
  | 00:09 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s) | [production] | 
            
  | 00:02 | <catrope@tin> | Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s) | [production] | 
            
  
    | 2017-09-20
      
      § | 
    
  | 23:49 | <catrope@tin> | Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s) | [production] | 
            
  | 23:47 | <catrope@tin> | Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s) | [production] | 
            
  | 23:40 | <mutante> | phabricator: aphlict (notification service) now running on prod server | [production] | 
            
  | 23:36 | <mutante> | phab1001 - re-enable puppet, run after follow-up fix, adding notification server config | [production] | 
            
  | 23:31 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s) | [production] | 
            
  | 23:29 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s) | [production] | 
            
  | 23:21 | <catrope@tin> | Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s) | [production] | 
            
  | 23:14 | <mutante> | phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group | [production] | 
            
  | 23:13 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s) | [production] | 
            
  | 23:11 | <mutante> | phab2001 - groupadd aphlict,temp adduser aphlict  to debug user creation issue | [production] | 
            
  | 22:22 | <ebernhardson> | elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb | [production] | 
            
  | 21:54 | <mutante> | phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001 | [production] | 
            
  | 21:45 | <eileen> | update civicrm from 0fb46767f0c86b3fc85c44e228fd7c1a7ae75e46 to da833ac561f7586a0b64fbef2cdf2963bc4055c3 | [production] | 
            
  | 21:09 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet | [production] | 
            
  | 20:58 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet | [production] | 
            
  | 20:40 | <bsitzmann@tin> | Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s) | [production] | 
            
  | 20:36 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet | [production] | 
            
  | 20:35 | <ebernhar|lunch> | increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster | [production] | 
            
  | 20:34 | <bsitzmann@tin> | Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) | [production] | 
            
  | 20:33 | <demon@tin> | Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s) | [production] | 
            
  | 19:55 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet | [production] | 
            
  | 19:54 | <eevans@tin> | Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s) | [production] | 
            
  | 19:54 | <eevans@tin> | Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster | [production] | 
            
  | 19:53 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet | [production] | 
            
  | 19:48 | <eevans@tin> | Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s) | [production] | 
            
  | 19:47 | <eevans@tin> | Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release | [production] | 
            
  | 19:39 | <bblack@neodymium> | conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet | [production] | 
            
  | 19:26 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19 | [production] | 
            
  | 19:24 | <demon@tin> | Synchronized php: symlink swap for wmf.19 (duration: 00m 49s) | [production] | 
            
  | 19:02 | <catrope@tin> | Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s) | [production] | 
            
  | 18:40 | <catrope@tin> | Started scap: Patches for T176302, T175962, T173533 | [production] | 
            
  | 17:29 | <hoo> | Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed. | [production] | 
            
  | 17:29 | <hoo> | Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed. | [production] | 
            
  | 17:13 | <hoo> | Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes | [production] | 
            
  | 17:06 | <ejegg> | removed limit from donation queue consumer | [production] |