| 
      
        2015-04-02
      
      §
     | 
  
    
  | 15:17 | 
  <manybubbles> | 
  Synchronized wmf-config/CommonSettings.php: SWAT enable newarticle campaign on cawiki 1/3 (duration: 00m 13s) | 
  [production] | 
            
  | 15:16 | 
  <manybubbles> | 
  ignore last log - its a noop failure on my part | 
  [production] | 
            
  | 15:16 | 
  <manybubbles> | 
  Synchronized wmf-config/CommonSettings.php: SWAT enable newarticle campaign on cawiki 1/3 (duration: 00m 12s) | 
  [production] | 
            
  | 15:15 | 
  <manybubbles> | 
  Synchronized php-1.25wmf23/includes/User.php: SWAT user preferences load from the master by default (duration: 00m 12s) | 
  [production] | 
            
  | 14:13 | 
  <hashar> | 
  Jenkins: migrated Zuul cloner on Precise labs slaves (100[1-4] to a version provided by a Debian package. Jobs console output should now shows <tt>Zuul version: 2.0.0-304-g685ca22-wmf1precise1</tt> | 
  [production] | 
            
  | 12:51 | 
  <andrewbogott> | 
  restarted opendj, pdns on neptunium, nembus, virt1000, labcontrol2001 | 
  [production] | 
            
  | 12:48 | 
  <paravoid> | 
  repooling esams | 
  [production] | 
            
  | 12:42 | 
  <paravoid> | 
  upgrading junos on mr1-esams | 
  [production] | 
            
  | 12:15 | 
  <mark> | 
  Shutting down cp3014 for 10G upgrade | 
  [production] | 
            
  | 11:02 | 
  <mark> | 
  Shutting down cp3012 for 10G upgrade | 
  [production] | 
            
  | 10:38 | 
  <_joe_> | 
  stopping pybal on lvs2003, running manually to help debugging | 
  [production] | 
            
  | 09:43 | 
  <paravoid> | 
  asw-d-eqiad: routing-engine backup switch FPC 8 -> FPC 4 | 
  [production] | 
            
  | 09:42 | 
  <paravoid> | 
  asw-d-eqiad: routing-engine backup switch FPC 7 -> FPC 5, master switchover FPC 8 -> FPC 5 | 
  [production] | 
            
  | 08:31 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Thu Apr  2 08:30:42 UTC 2015 (duration 30m 41s) | 
  [production] | 
            
  | 07:48 | 
  <springle> | 
  on sync-file from tin: mw2213.codfw.wmnet returned [255]: Host key verification failed | 
  [production] | 
            
  | 07:46 | 
  <springle> | 
  Synchronized wmf-config/db-eqiad.php: repool db1027, warm up (duration: 00m 11s) | 
  [production] | 
            
  | 06:41 | 
  <bblack> | 
  re-depooled esams ... | 
  [production] | 
            
  | 06:34 | 
  <Jamesofur> | 
  manually merged Ximilian global account per request and account confirmation | 
  [production] | 
            
  | 06:13 | 
  <bblack> | 
  re-pooling esams (GTT event never happened AFAICS) | 
  [production] | 
            
  | 05:13 | 
  <springle> | 
  xtrabackup clone db1027 to db1035 | 
  [production] | 
            
  | 03:45 | 
  <springle> | 
  Synchronized wmf-config/db-eqiad.php: depool db1027 (duration: 00m 11s) | 
  [production] | 
            
  | 03:32 | 
  <springle> | 
  reinstall db1035, reclone data | 
  [production] | 
            
  | 03:18 | 
  <LocalisationUpdate> | 
  completed (1.25wmf24) at 2015-04-02 03:08:52+00:00 | 
  [production] | 
            
  | 03:03 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 08m 55s) | 
  [production] | 
            
  | 02:41 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 56s) | 
  [production] | 
            
  | 02:06 | 
  <legoktm> | 
  set email for TheFons@global and attached nlwiki | 
  [production] | 
            
  | 00:03 | 
  <krenair> | 
  Synchronized php-1.25wmf24/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201386 (duration: 00m 12s) | 
  [production] | 
            
  | 00:01 | 
  <krenair> | 
  Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201385 (duration: 00m 13s) | 
  [production] | 
            
  
    | 
      
        2015-04-01
      
      §
     | 
  
    
  | 23:59 | 
  <bblack> | 
  depooling esams ahead of 2h planned GTT link outage coming up in 1h | 
  [production] | 
            
  | 23:41 | 
  <krenair> | 
  Synchronized wmf-config: https://gerrit.wikimedia.org/r/200098 (duration: 00m 12s) | 
  [production] | 
            
  | 23:37 | 
  <krenair> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201374/ (duration: 00m 12s) | 
  [production] | 
            
  | 23:35 | 
  <krenair> | 
  Synchronized php-1.25wmf24/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 13s) | 
  [production] | 
            
  | 23:33 | 
  <krenair> | 
  Synchronized php-1.25wmf24/extensions/Flow/modules/editor/editors/visualeditor/ext.flow.editors.visualeditor.js: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 11s) | 
  [production] | 
            
  | 23:29 | 
  <krenair> | 
  Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/201093/ (duration: 00m 12s) | 
  [production] | 
            
  | 23:23 | 
  <krenair> | 
  Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/#/c/201367/ (duration: 00m 15s) | 
  [production] | 
            
  | 23:02 | 
  <krenair> | 
  Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/201197/ (duration: 00m 11s) | 
  [production] | 
            
  | 22:59 | 
  <awight> | 
  update payments from f617326761887ed9a9100b472ea3b5736e2c10e6 to d37687239fa79842c0d6ea65e9230a3f14cda867 | 
  [production] | 
            
  | 22:57 | 
  <twentyafterfour> | 
  Purged l10n cache for 1.25wmf22 | 
  [production] | 
            
  | 22:57 | 
  <kaldari> | 
  Synchronized wmf-config/InitialiseSettings.php: enabling Gather on enwiki (duration: 00m 13s) | 
  [production] | 
            
  | 22:48 | 
  <twentyafterfour> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf24 | 
  [production] | 
            
  | 22:31 | 
  <twentyafterfour> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf23 | 
  [production] | 
            
  | 22:14 | 
  <twentyafterfour> | 
  Finished scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache (duration: 54m 14s) | 
  [production] | 
            
  | 21:56 | 
  <bblack> | 
  repooled cp107[1234] in pybal (eqiad upload, row D) | 
  [production] | 
            
  | 21:20 | 
  <twentyafterfour> | 
  Started scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache | 
  [production] | 
            
  | 21:18 | 
  <twentyafterfour> | 
  Started scap: testwiki to 1.25wmf24 and rebuild l10n cache | 
  [production] | 
            
  | 20:32 | 
  <gwicke> | 
  finished RESTBase 0.5.0 deployment | 
  [production] | 
            
  | 20:28 | 
  <gwicke> | 
  deploying RESTBase 0.5.0 | 
  [production] | 
            
  | 20:19 | 
  <twentyafterfour> | 
  Started scap: retrying: testwiki to php-1.25wmf24 and rebuild l10n cache | 
  [production] | 
            
  | 20:08 | 
  <YuviPanda> | 
  run chmod -R g+w . on tin with CWD /srv/deployment/scap/scap/.git | 
  [production] | 
            
  | 19:48 | 
  <twentyafterfour> | 
  scap failed: CalledProcessError Command 'cp '/srv/mediawiki-staging/php-1.25wmf24/cache/l10n/'*.cdb '/tmp/scap_l10n_1816369030'' returned non-zero exit status 1 (duration: 14m 20s) | 
  [production] |