| 2015-02-25
      
      § | 
    
  | 22:51 | <andrewbogott> | rebooting virt1005 in anticipation of an exprimental upgrade to Trusty.  (There are no VMs on virt1005 other than a testing host) | [production] | 
            
  | 22:44 | <twentyafterfour> | Done deploying - uploaded release notes for 1.25wmf19 | [production] | 
            
  | 22:36 | <twentyafterfour> | Purged l10n cache for 1.25wmf17 | [production] | 
            
  | 22:32 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf19 | [production] | 
            
  | 22:26 | <robh> | finished with my deployment window for redirects, tested and is now live with no issues (so far) | [production] | 
            
  | 22:25 | <twentyafterfour> | Synchronized ./wmf-config/InitialiseSettings.php: mw1204 still logging errors (duration: 00m 05s) | [production] | 
            
  | 22:22 | <twentyafterfour> | Synchronized ./wmf-config/InitialiseSettings.php: flooding logs (duration: 00m 07s) | [production] | 
            
  | 22:19 | <twentyafterfour> | Synchronized ./wmf-config/InitialiseSettings.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 22:16 | <robh> | localtesting of change on mw1001 shows no issues, so pushing out to rest of apaches | [production] | 
            
  | 22:13 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf18 | [production] | 
            
  | 10:00 | <_joe_> | restarted hhvm on mw1229, stuck in __lll_lock_wait from HPHP::hphp_session_init | [production] | 
            
  | 07:04 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Feb 25 07:03:41 UTC 2015 (duration 3m 40s) | [production] | 
            
  | 06:22 | <greg-g> | 06:20 < twentyaft> that log was bogus, just me testing but not actually syncing | [production] | 
            
  | 06:17 | <twentyafterfour> | Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 02s) | [production] | 
            
  | 04:38 | <springle> | s/db1001/dbproxy1001/g on zirconium drupal contacts. seems unpuppetized | [production] | 
            
  | 02:35 | <LocalisationUpdate> | completed (1.25wmf18) at 2015-02-25 02:34:23+00:00 | [production] | 
            
  | 02:34 | <l10nupdate> | Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 03s) | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.25wmf17) at 2015-02-25 02:21:53+00:00 | [production] | 
            
  | 02:21 | <l10nupdate> | Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s) | [production] | 
            
  
    | 2015-02-24
      
      § | 
    
  | 23:37 | <springle> | killed runaway RecentChangesUpdateJob::purgeExpiredRows transactions from jobrunners. db1033 db1038 db1040 db1052 db1058 | [production] | 
            
  | 22:59 | <aaron> | Synchronized php-1.25wmf17/includes/jobqueue/jobs/RecentChangesUpdateJob.php: 6f6d7e57be0ccff8ed2473b7d250e77703c7a6dd (duration: 00m 09s) | [production] | 
            
  | 22:43 | <aude> | Synchronized docroot/mediawiki/xml/: Add sitelist export-import docs (duration: 00m 07s) | [production] | 
            
  | 22:28 | <aude> | Synchronized wikidataclient.dblist: Enable Wikibase Client on Wikibooks (duration: 00m 06s) | [production] | 
            
  | 22:27 | <aude> | Synchronized wmf-config/InitialiseSettings.php: Wikibase config for Wikibooks (duration: 00m 06s) | [production] | 
            
  | 22:25 | <aude> | Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata (duration: 00m 06s) | [production] | 
            
  | 22:20 | <aude> | Synchronized wmf-config/Wikibase.php: Enable Wikibooks sitelinks on Wikidata (duration: 00m 06s) | [production] | 
            
  | 22:08 | <aude> | Finished scap: Updates for enabling Wikibase on Wikibooks (duration: 16m 03s) | [production] | 
            
  | 21:52 | <aude> | Started scap: Updates for enabling Wikibase on Wikibooks | [production] | 
            
  | 21:42 | <twentyafterfour> | rebuilt wikiversions.cdb and synchronized wikiversions files: sync-wikiversions group1 to 1.25wmf18 | [production] | 
            
  | 21:25 | <twentyafterfour> | Synchronized php-1.25wmf18/extensions/Popups: hotfix for Popups (see https://gerrit.wikimedia.org/r/#/c/192465/) (duration: 00m 06s) | [production] | 
            
  | 21:24 | <ottomata> | increased Hadoop nodemanager cpu-vcores to facter $processcount - 1, this should increase hadoop cluster utilization | [production] | 
            
  | 20:31 | <paravoid> | restarting cr1-eqiad/re1 chassis-control; should not be traffic-disrupting | [production] | 
            
  | 20:05 | <qchris> | Updated gerrit plugin its-phabricator-from-bugzilla to 03b936b2cd8fa6adfdbee0ef68eb4b31944936c2 | [production] | 
            
  | 20:05 | <qchris> | Updated gerrit plugin its-phabricator to 25a34d7564cffb90a87110a971782195ba2db467 | [production] | 
            
  | 17:55 | <Coren> | Shutting down labstore1001 - planned outage for expansion | [production] | 
            
  | 14:59 | <godog> | rolling restart cassandra to pick up metrics configuration | [production] | 
            
  | 07:55 | <andrewbogott> | ‘nova reset-state —active’ and ‘nova reboot’ for EVERY instance on virt1012 | [production] | 
            
  | 07:13 | <andrewbogott> | suspending all instances on virt1012 | [production] | 
            
  | 06:40 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Feb 24 06:39:18 UTC 2015 (duration 39m 17s) | [production] | 
            
  | 06:29 | <YuviPanda> | stopped nova-compute on virt1005 | [production] | 
            
  | 06:28 | <YuviPanda> | starting nova-compute on virt1005 | [production] | 
            
  | 04:07 | <springle> | Synchronized wmf-config/db-eqiad.php: raise db1066 load (duration: 00m 07s) | [production] | 
            
  | 02:50 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1066, warm up (duration: 00m 06s) | [production] | 
            
  | 02:19 | <LocalisationUpdate> | completed (1.25wmf18) at 2015-02-24 02:18:03+00:00 | [production] | 
            
  | 02:18 | <l10nupdate> | Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s) | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.25wmf17) at 2015-02-24 02:16:30+00:00 | [production] | 
            
  | 02:16 | <l10nupdate> | Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s) | [production] | 
            
  | 01:52 | <ori> | Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Reverting live-hack (duration: 00m 07s) | [production] | 
            
  | 01:47 | <ori> | Synchronized php-1.25wmf17/extensions/MobileFrontend/includes/modules/MobileUserModule.php: Testing a theory for T90411 with a live-hack to MobileFrontend. Will revert momentarily. (duration: 00m 07s) | [production] | 
            
  | 00:56 | <Tim> | on osmium, removing the packages I just installed since I will do it in a chroot instead | [production] |