| 2014-03-20
      
      § | 
    
  | 23:38 | <catrope> | synchronized php-1.23wmf18/resources/mediawiki/mediawiki.inspect.js | [production] | 
            
  | 23:22 | <catrope> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 23:22 | <catrope> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 23:22 | <catrope> | synchronized docroot/noc/createTxtFileSymlinks.sh | [production] | 
            
  | 23:22 | <catrope> | updated /a/common to {{Gerrit|Ia08c65d40}}: Enable Flow on Hovercards Beta Features | [production] | 
            
  | 21:43 | <ori> | synchronized wmf-config/InitialiseSettings.php  'If51eda243: Follow up Id6222f4db to amend sort order in feed URL' | [production] | 
            
  | 21:43 | <ori> | updated /a/common to {{Gerrit|If51eda243}}: Follow up Id6222f4db to amend sort order in feed URL | [production] | 
            
  | 21:20 | <ori> | synchronized wmf-config/InitialiseSettings.php  'Id6222f4d: Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist' | [production] | 
            
  | 21:20 | <ori> | updated /a/common to {{Gerrit|Id6222f4db}}: Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist | [production] | 
            
  | 21:06 | <reedy> | synchronized docroot and w | [production] | 
            
  | 21:03 | <reedy> | updated /a/common to {{Gerrit|Iaa99d2162}}: Add Wikibase repoSiteName setting for client | [production] | 
            
  | 20:56 | <bd808> | Updated scholarships.wikimedia.org to cb2ef4c (fix for bug 62464) | [production] | 
            
  | 20:14 | <mutante> | DNS update - remove ssl1-4 | [production] | 
            
  | 20:08 | <mutante> | DNS update - remove sq67-70, former varnish testing | [production] | 
            
  | 19:35 | <akosiaris> | created a 50G LV for /var/log on zirconium, stopped all services, moved data to it, mounted it and restarted all services | [production] | 
            
  | 19:29 | <reedy> | synchronized wmf-config/  'Wikibase config updates' | [production] | 
            
  | 19:23 | <reedy> | Finished scap: Rebuild 1.23wmf19 l10n cache for wikibase (duration: 12m 01s) | [production] | 
            
  | 19:10 | <reedy> | Started scap: Rebuild 1.23wmf19 l10n cache for wikibase | [production] | 
            
  | 19:08 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf19 | [production] | 
            
  | 19:02 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf18 | [production] | 
            
  | 19:00 | <mutante> | disk full on zirconium - gzipping some etherpadlite.sql dump i found | [production] | 
            
  | 17:41 | <reedy> | synchronized php-1.23wmf19  'Update Wikidata and WikimediaMessages' | [production] | 
            
  | 17:06 | <Krinkle> | Reloading Zuul to deploy Ie800ed90b51c47d5a1 | [production] | 
            
  | 16:58 | <mutante> | repooling mw1163 (it's back in dsh as well) | [production] | 
            
  | 16:38 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf18 till window | [production] | 
            
  | 16:05 | <bd808> | Ran /usr/local/bin/sync-common && /usr/local/bin/scap-rebuild-cdbs on mw1163. Should not repool until it's back in the dsh group. Should me manually synced just before repooling. | [production] | 
            
  | 15:36 | <reedy> | Finished scap: testwiki to 1.23wmf19 and build l10n cache (duration: 15m 32s) | [production] | 
            
  | 15:21 | <reedy> | Started scap: testwiki to 1.23wmf19 and build l10n cache | [production] | 
            
  | 15:03 | <maxsem> | synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/118328' | [production] | 
            
  | 15:00 | <bd808> | logstash stopped ingesting logs at 2014-03-19T22:37:54.000Z. | [production] | 
            
  | 14:57 | <bd808> | restarted logstash on logstash1001.eqiad.wmnet | [production] | 
            
  | 03:23 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu Mar 20 03:23:37 UTC 2014 (duration 23m 35s) | [production] | 
            
  | 02:32 | <LocalisationUpdate> | completed (1.23wmf18) at 2014-03-20 02:32:10+00:00 | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.23wmf17) at 2014-03-20 02:17:34+00:00 | [production] | 
            
  | 01:26 | <Tim> | gave Sam Reed access to racktables | [production] | 
            
  | 00:26 | <bd808> | scap-slave rsync servers have "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22"; missing new 10.64.48.0/22 subnet | [production] | 
            
  | 00:21 | <bd808> | Subnet for row D still not in rsync server config? | [production] | 
            
  | 00:21 | <bd808> | scaps failed because "@ERROR: access denied to common from mw1202.eqiad.wmnet (10.64.48.34)" | [production] | 
            
  | 00:16 | <bd808> | finished dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' | [production] | 
            
  | 00:13 | <bd808> | dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' | [production] | 
            
  
    | 2014-03-19
      
      § | 
    
  | 23:57 | <spage> | synchronized wmf-config  'Config change to enable extension Popups (Hovercards) on mediawikiwiki' | [production] | 
            
  | 23:30 | <spage> | Finished scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 07m 15s) | [production] | 
            
  | 23:22 | <spage> | Started scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet | [production] | 
            
  | 23:22 | <spage> | scap aborted: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 69m 26s) | [production] | 
            
  | 23:20 | <bd808> | restarted elasticsearch on logstash1002 | [production] | 
            
  | 23:18 | <bd808> | scap by spage looks hung with 50 hosts not finished rsyncing | [production] | 
            
  | 22:38 | <bd808> | Manually ran sync-common and scap-rebuild-cdbs on mw1163 | [production] | 
            
  | 22:33 | <mutante> | depooling mw1163 | [production] | 
            
  | 22:28 | <ori> | synchronized wmf-config/mc.php  'Set serializer to php for production memcache' | [production] | 
            
  | 22:12 | <spage> | Started scap: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet | [production] |