| 2016-04-16
      
      § | 
    
  | 14:24 | <andrewbogott> | rebooting mw1134 — OOM | [production] | 
            
  | 10:34 | <jynus> | recreating _counters table on s3 to solve replication issues | [production] | 
            
  | 10:08 | <jynus> | recreating hitcounter table on s3 to solve replication issues | [production] | 
            
  | 09:31 | <volans> | Skipped on dbstore1002 query from replica: 'DELETE FROM `bawiktionary`.`hitcounter`' | [production] | 
            
  | 09:31 | <jynus> | importing sys schema to db1075 | [production] | 
            
  | 06:44 | <akosiaris> | restart fermium to apply extra vcpus assignment | [production] | 
            
  | 06:23 | <volans> | Stopped and restarted replica on dbstore2002 for s3 to "unstuck" the replica | [production] | 
            
  | 02:31 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Sat Apr 16 02:31:33 UTC 2016 (duration 9m 11s) | [production] | 
            
  | 02:22 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.21) (duration: 09m 03s) | [production] | 
            
  | 02:16 | <urandom> | Bootstraping restbase1009-a.eqiad.wmnet : T95253 | [production] | 
            
  
    | 2016-04-15
      
      § | 
    
  | 22:28 | <mutante> | rebooting bast4001 - debugging install issue | [production] | 
            
  | 21:04 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Repool new db1075,1077,1078 after TLS upgrade on s3 - T111654 (duration: 00m 36s) | [production] | 
            
  | 16:08 | <demon@tin> | Synchronized wmf-config/CommonSettings.php: more liberal initialisesettings invalidation (duration: 00m 42s) | [production] | 
            
  | 15:34 | <godog> | remove cassandra metrics for restbase100[1234]* restbase100[789] restbase2004 - T132771 | [production] | 
            
  | 14:40 | <gehel> | reindexing all wikis after the switch to codfw (T132762) | [production] | 
            
  | 14:06 | <urandom> | start decommission of restbase1009-a.eqiad.wmnet : T95253 | [production] | 
            
  | 12:15 | <gehel> | increase max request size for elasticsearch | [production] | 
            
  | 12:14 | <volans> | Re-arrange s3 replica topology: making codfw replicate from db1075 (this time for real) - T111654 | [production] | 
            
  | 12:08 | <bblack> | experimenting on carbon HTTP config (apt/mirrors/ubuntu.wm.o) - watch out for installer / package-update issues! | [production] | 
            
  | 12:00 | <gehel> | remove maintenance from wdqs1002 | [production] | 
            
  | 10:50 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Depool new db1075,1077,1078 to upgrade TLS on s3 - T111654 (duration: 00m 41s) | [production] | 
            
  | 10:37 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1042 after TLS upgrade on s4 - T111654 (duration: 00m 30s) | [production] | 
            
  | 10:21 | <_joe_> | restarted zuul, zuul-merger on gallium | [production] | 
            
  | 10:10 | <TimStarling> | on helium: scheduled restore of home_pmtpa to bast4001 | [production] | 
            
  | 09:23 | <volans> | Re-arrange s3 replica topology: making codfw replicate from db1075 - T111654 | [production] | 
            
  | 09:01 | <gehel> | reenabling wdqs1002 in varnish rotation after reinstall (T132387) | [production] | 
            
  | 08:09 | <volans@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1042 to upgrade TLS on s4 - T111654 (duration: 00m 36s) | [production] | 
            
  | 08:05 | <volans> | starting TLS upgrade for shard s4 T111654 | [production] | 
            
  | 08:00 | <gehel> | removing empty log archives from Fluorine (T132324) | [production] | 
            
  | 07:40 | <moritzm> | rebooting oresrdb* to Linux 4.4 | [production] | 
            
  | 07:17 | <moritzm> | rebooting oxygen for kernel update to 4.4 | [production] | 
            
  | 02:30 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Apr 15 02:30:21 UTC 2016 (duration 8m 55s) | [production] | 
            
  | 02:21 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.21) (duration: 08m 28s) | [production] | 
            
  
    | 2016-04-14
      
      § | 
    
  | 23:13 | <demon@tin> | Synchronized portals: (no message) (duration: 00m 29s) | [production] | 
            
  | 23:12 | <demon@tin> | Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 29s) | [production] | 
            
  | 23:11 | <demon@tin> | Synchronized php-1.27.0-wmf.21/includes/resourceloader/ResourceLoaderSpecialCharacterDataModule.php: I3e26d08a (duration: 00m 30s) | [production] | 
            
  | 23:08 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: Add images.unsplash.com to $wgCopyUploadsDomains (duration: 00m 33s) | [production] | 
            
  | 23:06 | <demon@tin> | Synchronized portals/: updating to master, removing top-links A/B test (duration: 00m 45s) | [production] | 
            
  | 21:57 | <bd808@tin> | Synchronized php-1.27.0-wmf.21/extensions/ContentTranslation: Enable europeana2802016 campaign (T125626) (duration: 00m 34s) | [production] | 
            
  | 21:44 | <csteipp@tin> | Synchronized wmf-config/CommonSettings-labs.php: Syncing labs config change (duration: 00m 27s) | [production] | 
            
  | 21:43 | <csteipp@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: Syncing labs config change (duration: 00m 34s) | [production] | 
            
  | 20:31 | <ostriches> | gerrit restarting | [production] | 
            
  | 19:35 | <godog> | force puppet run on cache_upload in eqiad | [production] | 
            
  | 19:26 | <godog> | force puppet run for cache_upload in esams | [production] | 
            
  | 19:11 | <godog> | depool upload/eqiad | [production] | 
            
  | 19:01 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.21 | [production] | 
            
  | 18:58 | <godog> | force puppet run on cache_upload in codfw | [production] | 
            
  | 18:51 | <godog> | forcing puppet run on cache_upload in eqiad and codfw | [production] | 
            
  | 18:36 | <demon@tin> | Finished scap: sync some symlink removals (duration: 04m 54s) | [production] | 
            
  | 18:31 | <demon@tin> | Started scap: sync some symlink removals | [production] |