| 2016-03-11
      
      ยง | 
    
  | 15:03 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: moving large.dblist back to wmf.16, did not help | [production] | 
            
  | 14:45 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: moving largest.dblist back to wmf.15 for now | [production] | 
            
  | 14:33 | <godog> | run swiftrepl thumbs eqiad -> codfw with concurrency 96 | [production] | 
            
  | 14:25 | <godog> | run swiftrepl thumbs eqiad -> codfw with concurrency 128 | [production] | 
            
  | 14:12 | <moritzm> | installed security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw1017 (other canaries will be upgraded later on if all is well, the rest of mw* on Monday) | [production] | 
            
  | 13:50 | <apergos> | on fluorine truncated archive/redis.log-20160311 and archive/JobQueueFederated.log-20160311 to 5mb each, they were each about 500gb | [production] | 
            
  | 13:26 | <mobrovac> | restbase deploy end of 3bedb8f5c42 | [production] | 
            
  | 12:56 | <mobrovac> | restbase deploy start of 3bedb8f5c42 on canary restbase1001 | [production] | 
            
  | 12:36 | <mobrovac> | mathoid deploying 7a282a4181a4 | [production] | 
            
  | 12:10 | <moritzm> | uploaded backport of linux-tools 4.4-4 for jessie-wikimedia to carbon (provides kbuild and perf amonst others) | [production] | 
            
  | 11:48 | <akosiaris> | restarting gerrit on ytterbium | [production] | 
            
  | 11:27 | <jynus> | running authdns-update to solve a dns typo | [production] | 
            
  | 11:25 | <jynus> | killed some long-running git-upload-pack tasks | [production] | 
            
  | 10:26 | <hashar> | Gerrit task management documented on https://wikitech.wikimedia.org/wiki/Gerrit#Tasks_management | [production] | 
            
  | 10:11 | <hashar> | Gerrit: killed stuck "git-upload-pack '/mediawiki/core.git'" tasks | [production] | 
            
  | 09:46 | <elukey> | Rebooting rdb200[56].codfw for kernel upgrade | [production] | 
            
  | 09:13 | <ema> | umounted /sys/kernel/debug/tracing on cp1067 | [production] | 
            
  | 09:12 | <hashar> | Enabling puppet on gallium.wikimedia.org . Been disabled since ~ Mon Mar 7 10:26:01 UTC 2016 | [production] | 
            
  | 09:07 | <moritzm> | uploaded linux-meta 1.9 for jessie-wikimedia to carbon (which now defaults linux-meta to installing Linux 4.4) | [production] | 
            
  | 08:49 | <ema> | cp1067: apt-get removed linux-image-3.16.0-4-amd64 and linux-image-4.4.0-1-amd64-dbg to free up some disk space | [production] | 
            
  | 08:20 | <moritzm> | restarted hhvm on mw1246, mw1258 | [production] | 
            
  | 08:18 | <moritzm> | restarted hhvm on mw1220, mw1244 | [production] | 
            
  | 08:17 | <moritzm> | restarted hhvm on mw1174, mw1186, mw1211 | [production] | 
            
  | 07:31 | <kartik@tin> | Synchronized php-1.27.0-wmf.16/extensions/ContentTranslation: Deploying 276700 for ContentTranslation (duration: 00m 43s) | [production] | 
            
  | 06:44 | <_joe_> | restarted hhvm on mw1166 | [production] | 
            
  | 04:50 | <ori> | Uninstalled HHVM on nobelium; not puppetized. | [production] | 
            
  | 04:15 | <papaul> | rdb200[5-6]  installation complete | [production] | 
            
  | 03:45 | <papaul> | rdb200[5-6] signing puppet certs, salt-key, initial run | [production] | 
            
  | 02:57 | <krenair@tin> | Synchronized php-1.27.0-wmf.16/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWTemplatePage.js: touch - see comment on https://gerrit.wikimedia.org/r/#/c/274120/ (duration: 00m 31s) | [production] | 
            
  | 02:54 | <papaul> | installing rdb200[5-6] | [production] | 
            
  | 02:39 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Mar 11 02:39:01 UTC 2016 (duration 8m 45s) | [production] | 
            
  | 02:31 | <mutante> | cygnus/technetium: puppet cert clean, salt-key -d (neodymium), puppetstoredconfigclean.rb (rm from Icinga),  gnt-instance remove (destroy VMs) | [production] | 
            
  | 02:30 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.16) (duration: 12m 33s) | [production] | 
            
  | 02:23 | <mutante> | cygnus - poweroff (for variety) | [production] | 
            
  | 02:21 | <mutante> | technetium - shutdown -h now | [production] | 
            
  | 01:25 | <krinkle@tin> | Synchronized w/static.php: 6da604f and 49c07ac (duration: 00m 39s) | [production] | 
            
  | 01:06 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Un-revert Flow change on mw.org (duration: 00m 26s) | [production] | 
            
  | 01:05 | <catrope@tin> | Synchronized wmf-config/CommonSettings.php: Un-revert Flow change on mw.org (duration: 00m 29s) | [production] | 
            
  | 01:04 | <MaxSem> | Made mediawiki/php/wikidiff read-only in Gerrit | [production] | 
            
  | 00:46 | <catrope@tin> | Synchronized portals: (no message) (duration: 00m 26s) | [production] | 
            
  | 00:45 | <catrope@tin> | Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 26s) | [production] | 
            
  | 00:43 | <catrope@tin> | Synchronized php-1.27.0-wmf.16/extensions/Echo/Hooks.php: Try fixing the thank-you-edit bug again (duration: 00m 26s) | [production] | 
            
  | 00:41 | <catrope@tin> | Synchronized php-1.27.0-wmf.16/includes/diff/DifferenceEngine.php: Convert timing to milliseconds (duration: 00m 26s) | [production] | 
            
  | 00:32 | <RoanKattouw> | Running populateContentModel.php on mediawikiwiki so I can un-revert the Flow change | [production] | 
            
  | 00:28 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Revert Flow change on mw.org (duration: 00m 27s) | [production] | 
            
  | 00:28 | <catrope@tin> | Synchronized wmf-config/CommonSettings.php: Revert Flow change on mw.org (duration: 00m 27s) | [production] | 
            
  | 00:24 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable cross-wiki notifications beta feature on all wikis (duration: 00m 27s) | [production] | 
            
  | 00:22 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Clean up old Flow occupy stuff (duration: 00m 26s) | [production] | 
            
  | 00:22 | <catrope@tin> | Synchronized wmf-config/CommonSettings.php: Clean up old Flow occupy stuff (duration: 00m 25s) | [production] | 
            
  | 00:19 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Make Flow the default in talk namespaces on mediawikiwiki (duration: 00m 38s) | [production] |