| 2014-10-28
      
      § | 
    
  | 18:52 | <reedy> | Finished scap: Split Cite extension, scap to build l10n cache for CiteThisPage (duration: 33m 52s) | [production] | 
            
  | 18:29 | <bd808|LUNCH> | restarted elasticsearch node on logstash1003 | [production] | 
            
  | 18:18 | <reedy> | Started scap: Split Cite extension, scap to build l10n cache for CiteThisPage | [production] | 
            
  | 18:11 | <reedy> | Synchronized wmf-config/: Set = true for Italian Wikipedia in November-December 2014 (duration: 00m 14s) | [production] | 
            
  | 18:05 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf5 | [production] | 
            
  | 17:27 | <ori> | Synchronized wmf-config/CommonSettings.php: I56082795: Modify $wmgAddWikiNotify for use by notifyNewProjects (duration: 00m 05s) | [production] | 
            
  | 17:00 | <bd808> | restarted logstash on logstash1002 to try and get gelf input events into kibana again | [production] | 
            
  | 16:58 | <bd808> | disk utilization on logstash100[123]  greater than 80% | [production] | 
            
  | 16:57 | <cscott> | no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) | [production] | 
            
  | 16:57 | <andrew> | Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 03s) | [production] | 
            
  | 16:56 | <bd808> | No new logs in /var/log/elasticsearch for logstash100[123] since Sep 30 06:25 | [production] | 
            
  | 16:55 | <andrew> | Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 02s) | [production] | 
            
  | 16:44 | <reedy> | Synchronized php-1.25wmf4/extensions/OpenStackManager: (no message) (duration: 00m 14s) | [production] | 
            
  | 15:47 | <andrewbogott> | running sync-common on virt1000 | [production] | 
            
  | 15:47 | <manybubbles> | Synchronized php-1.25wmf4/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) | [production] | 
            
  | 15:46 | <manybubbles> | Synchronized php-1.25wmf5/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) | [production] | 
            
  | 15:24 | <manybubbles> | Synchronized wmf-config/: SWAT cirrus regex queues too small? (duration: 00m 05s) | [production] | 
            
  | 15:11 | <manybubbles> | Synchronized php-1.25wmf5/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) | [production] | 
            
  | 15:03 | <manybubbles> | Synchronized wmf-config/: SWAT cirrus config updates - (hopefully) faster regexes (duration: 00m 06s) | [production] | 
            
  | 14:51 | <godog> | rolling-restart of eqiad ms-fe* after https://gerrit.wikimedia.org/r/#/c/167310/ | [production] | 
            
  | 14:04 | <godog> | reload swift frontend in eqiad after password rotation | [production] | 
            
  | 14:04 | <demon> | Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 04s) | [production] | 
            
  | 13:48 | <manybubbles> | Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 05s) | [production] | 
            
  | 13:47 | <manybubbles> | Synchronized php-1.25wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 11s) | [production] | 
            
  | 01:01 | <demon> | Synchronized wmf-config/InitialiseSettings.php: Turn Cirrus back on basically everywhere. If Elasticsearch freaks out again just revert I73ae276e to get back to lsearchd again (duration: 00m 04s) | [production] | 
            
  | 00:43 | <ori> | Synchronized php-1.25wmf4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I4adffaa26: Actually unset the HHVM cookie (duration: 00m 03s) | [production] | 
            
  | 00:43 | <ori> | Synchronized php-1.25wmf5/extensions/WikimediaEvents/WikimediaEventsHooks.php: I4adffaa26: Actually unset the HHVM cookie (duration: 00m 03s) | [production] | 
            
  | 00:27 | <awight> | reenabling recurring GlobalCollect job | [production] | 
            
  | 00:07 | <awight> | updated crm from 9bb50403616d80aa8d39a89ab59965f53e9e3f3d to ffa543cab3eb508fa38b94c6de2643d168b0d507 | [production] | 
            
  
    | 2014-10-27
      
      § | 
    
  | 23:52 | <bd808> | Restarted logstash service on logstash1001 because I was not seeing any events from MW make it into kibana | [production] | 
            
  | 23:27 | <maxsem> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/169229/ for reals now (duration: 00m 04s) | [production] | 
            
  | 23:27 | <Reedy> | restarted logstash on logstash1001 | [production] | 
            
  | 23:23 | <maxsem> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/169229/ (duration: 00m 04s) | [production] | 
            
  | 23:22 | <Tim> | on mw1114: disabled puppet, enabled Eval.PerfPidMap, restarted hhvm | [production] | 
            
  | 23:21 | <awight> | updated crm from 5b395c37dc596736ecafceeb156221e3751bfe37 to 9bb50403616d80aa8d39a89ab59965f53e9e3f3d | [production] | 
            
  | 23:21 | <awight> | disabling recurring globalcollect job | [production] | 
            
  | 23:20 | <maxsem> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/168771/ (duration: 00m 04s) | [production] | 
            
  | 23:17 | <maxsem> | Synchronized php-1.25wmf4/extensions/VisualEditor/: (no message) (duration: 00m 04s) | [production] | 
            
  | 23:16 | <maxsem> | Synchronized php-1.25wmf4/extensions/MobileFrontend/: (no message) (duration: 00m 05s) | [production] | 
            
  | 23:14 | <maxsem> | Synchronized php-1.25wmf5/extensions/MobileFrontend/: (no message) (duration: 00m 04s) | [production] | 
            
  | 23:14 | <maxsem> | Synchronized php-1.25wmf5/extensions/VisualEditor/: (no message) (duration: 00m 05s) | [production] | 
            
  | 23:14 | <maxsem> | Synchronized php-1.25wmf5/extensions/Wikidata/: (no message) (duration: 00m 10s) | [production] | 
            
  | 23:13 | <maxsem> | Synchronized php-1.25wmf3/extensions/Wikidata/: (no message) (duration: 00m 12s) | [production] | 
            
  | 23:06 | <maxsem> | Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/169192/ (duration: 00m 04s) | [production] | 
            
  | 22:58 | <awight> | reenabling recurring globalcollect job | [production] | 
            
  | 22:54 | <awight> | rollback civicrm from 9bb50403616d80aa8d39a89ab59965f53e9e3f3d to 5b395c37dc596736ecafceeb156221e3751bfe37 | [production] | 
            
  | 22:53 | <awight> | updated civicrm from 5b395c37dc596736ecafceeb156221e3751bfe37 to 9bb50403616d80aa8d39a89ab59965f53e9e3f3d | [production] | 
            
  | 22:50 | <aaron> | Synchronized wmf-config/flaggedrevs.php: Removed $wgFlaggedRevsProtectQuota for enwiki (duration: 00m 03s) | [production] | 
            
  | 22:46 | <awight> | disabling recurring GlobalCollect job | [production] | 
            
  | 22:45 | <Tim> | activated heap profiling on mw1114 | [production] |