| 2016-01-13
      
      § | 
    
  | 02:57 | <Krinkle> | Manually killed uwsgi graphite-web child processes on graphite1001. Service recovered itself from there. | [production] | 
            
  | 02:44 | <Krinkle> | Graphite is down. Consistently returns HTTP 502 Bad Gateway for any/all requests | [production] | 
            
  | 02:34 | <mwdeploy@tin> | mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 13s) | [production] | 
            
  | 01:33 | <yurik> | deployed tilerator maps service | [production] | 
            
  | 01:19 | <krenair@tin> | krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Echo/Resources.php: https://gerrit.wikimedia.org/r/#/c/263645/ (duration: 00m 32s) | [production] | 
            
  | 01:18 | <krenair@tin> | krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Flow/modules/editor/editors/visualeditor/mw.flow.ve.Target.js: https://gerrit.wikimedia.org/r/#/c/263644/ (duration: 00m 31s) | [production] | 
            
  | 01:03 | <krenair@tin> | krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/263770/ - after having done the submodule update this time (duration: 00m 31s) | [production] | 
            
  | 00:37 | <krenair@tin> | krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/263770/ (duration: 00m 33s) | [production] | 
            
  | 00:31 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/261994/ (duration: 00m 31s) | [production] | 
            
  | 00:28 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/262895/ (duration: 00m 32s) | [production] | 
            
  | 00:25 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/262894/ (duration: 00m 30s) | [production] | 
            
  | 00:17 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263237/ (duration: 00m 31s) | [production] | 
            
  | 00:15 | <krenair@tin> | krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/262999/ (duration: 00m 31s) | [production] | 
            
  | 00:10 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 30s) | [production] | 
            
  | 00:08 | <yurik> | switched all maps kartotherian servers to v5, restarted | [production] | 
            
  | 00:06 | <krenair@tin> | krenair@tin Synchronized images/mobile/wikivoyage.png: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 31s) | [production] | 
            
  | 00:05 | <krenair@tin> | krenair@tin Synchronized images/mobile/wikidata.png: https://gerrit.wikimedia.org/r/#/c/263201/ (duration: 00m 32s) | [production] | 
            
  
    | 2016-01-12
      
      § | 
    
  | 21:58 | <ori> | Restarting jobchron / jobrunner / HHVM on all job runners for I44990808 | [production] | 
            
  | 21:07 | <hoo@tin> | hoo@tin Synchronized php-1.27.0-wmf.10/extensions/Math/: Introduce a "MathEnableWikibaseDataType" config (duration: 00m 32s) | [production] | 
            
  | 20:52 | <hoo@tin> | hoo@tin Synchronized wmf-config/: Set $wgMathEnableWikibaseDataType to false (duration: 01m 29s) | [production] | 
            
  | 20:44 | <twentyafterfour@tin> | twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.10 | [production] | 
            
  | 20:34 | <thcipriani@tin> | thcipriani@tin Finished scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache (duration: 54m 42s) | [production] | 
            
  | 20:14 | <mobrovac> | restbase switching restbase200x to node 4.2 | [production] | 
            
  | 20:13 | <mobrovac> | restbase switch of restbase100[1-4] to node 4.2 completed | [production] | 
            
  | 20:10 | <mobrovac> | restbase switching restbase100[1-4] to node 4.2 | [production] | 
            
  | 19:39 | <thcipriani@tin> | thcipriani@tin Started scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache | [production] | 
            
  | 19:31 | <dduvall@tin> | dduvall@tin scap failed: CalledProcessError Command 'sudo -u www-data -n -- /bin/mktemp' returned non-zero exit status 1 (duration: 00m 42s) | [production] | 
            
  | 19:30 | <dduvall@tin> | dduvall@tin Started scap: testwiki to php-1.27.0-wmf.10 and rebuild l10n cache | [production] | 
            
  | 19:26 | <YuviPanda> | import new r-base package into carbon | [production] | 
            
  | 18:15 | <marxarelli> | cutting MW branch 1.27.0-wmf.10 | [production] | 
            
  | 17:37 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263632/ (duration: 00m 31s) | [production] | 
            
  | 16:53 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on gu.wikipedia [[gerrit:258441]] (duration: 00m 29s) | [production] | 
            
  | 16:48 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Get rid of old unused $wgAllowed* variables [[gerrit:256853]] (duration: 00m 29s) | [production] | 
            
  | 16:47 | <_joe_> | restarted salt-minion on tin | [production] | 
            
  | 16:44 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add portal namespace to ps.wikipedia.org [[gerrit:255519]] (duration: 00m 30s) | [production] | 
            
  | 16:41 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove proxyunbannable [[gerrit:254842]] (duration: 00m 30s) | [production] | 
            
  | 16:37 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysop to grant and revoke transwiki on gu.wikipedia [[gerrit:258474]] (duration: 00m 29s) | [production] | 
            
  | 16:33 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on pa.wikipedia [[gerrit:258436]] (duration: 00m 29s) | [production] | 
            
  | 16:22 | <thcipriani@tin> | thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on my.wikipedia [[gerrit:258442]] (duration: 00m 30s) | [production] | 
            
  | 15:56 | <godog> | reprovision ms-fe3001 with jessie | [production] | 
            
  | 14:54 | <ema> | added myself to ops and wmf ldap groups | [production] | 
            
  | 11:57 | <_joe_> | enabling auth on the production etcd cluster | [production] | 
            
  | 08:37 | <paravoid> | ms-be1002: echo b > /proc/sysrq-trigger, kernel misbehaving and unrecoverable (out of kernel memory/XFS issues) | [production] | 
            
  | 07:38 | <paravoid> | cr2-eqiad: reenable BGP peerings with GTT | [production] | 
            
  | 05:31 | <paravoid> | rm CirrusSearchRequests.log-201510*.gz on fluorine (saving ~200G) | [production] | 
            
  | 04:07 | <paravoid> | cleaning up elastic1006's /var/log from old logs | [production] | 
            
  | 03:59 | <paravoid> | reenabling puppet on sca1001/2; no reason was left | [production] | 
            
  | 02:33 | <l10nupdate@tin> | l10nupdate@tin ResourceLoader cache refresh completed at Tue Jan 12 02:33:00 UTC 2016 (duration 6m 55s) | [production] | 
            
  | 02:26 | <mwdeploy@tin> | mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 10m 47s) | [production] | 
            
  | 00:46 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv 443026e3ad18934dd0017a258673d88104cf6b5e (duration: 00m 29s) | [production] |