| 
      
        2019-03-11
      
      §
     | 
  
    
  | 17:55 | 
  <addshore@deploy1001> | 
  Synchronized wmf-config/interwiki-labs.php: BETA ONLY https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/495723/ (duration: 00m 48s) | 
  [production] | 
            
  | 17:43 | 
  <addshore@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/495721/ (duration: 00m 49s) | 
  [production] | 
            
  | 17:23 | 
  <arturo> | 
  T215605 copy python-oath from jessie-wikimedia/thirdparty to stretch-wikimedia/thirdpary in reprepro | 
  [production] | 
            
  | 17:02 | 
  <jbond@cumin1001> | 
  END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) | 
  [production] | 
            
  | 17:02 | 
  <jbond@cumin1001> | 
  START - Cookbook sre.hosts.ipmi-password-reset | 
  [production] | 
            
  | 16:31 | 
  <jbond@cumin1001> | 
  END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) | 
  [production] | 
            
  | 16:31 | 
  <jbond@cumin1001> | 
  START - Cookbook sre.hosts.ipmi-password-reset | 
  [production] | 
            
  | 15:26 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Fully repool db1097 (duration: 00m 48s) | 
  [production] | 
            
  | 15:16 | 
  <mlitn@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: Fix syntax for MediaInfo depicts config (beta only) (duration: 00m 49s) | 
  [production] | 
            
  | 14:55 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: More traffic to db1097 (duration: 00m 49s) | 
  [production] | 
            
  | 14:43 | 
  <moritzm> | 
  upgrading mw canaries to PHP 7.2.16 | 
  [production] | 
            
  | 14:32 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: More traffic to db1097 (duration: 00m 48s) | 
  [production] | 
            
  | 14:25 | 
  <hashar> | 
  contint1001: stopping zuul-merger (it is cpu or IO starving the server) | 
  [production] | 
            
  | 14:21 | 
  <moritzm> | 
  upgrading mwdebug servers to PHP 7.2.16 | 
  [production] | 
            
  | 14:11 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Slowly repool db1097 (duration: 00m 47s) | 
  [production] | 
            
  | 14:09 | 
  <moritzm> | 
  importing build of PHP 7.2.16 for component/php72 (T216712) | 
  [production] | 
            
  | 13:58 | 
  <marostegui> | 
  Upgrade mysql on db1097 | 
  [production] | 
            
  | 13:28 | 
  <arturo> | 
  disable active checks in icinga for labtestvirt200[12] (T218023) | 
  [production] | 
            
  | 13:04 | 
  <moritzm> | 
  upgrading mwdebug2002 to php 7.2.16 | 
  [production] | 
            
  | 12:23 | 
  <gtirloni> | 
  updated slapd to version 2.4.47 on serpens (T217280) | 
  [production] | 
            
  | 12:05 | 
  <gtirloni> | 
  updating slapd on serpens/codfw to test possible fix for memory leaks | 
  [production] | 
            
  | 10:48 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1097 for upgrade and schema change (duration: 00m 48s) | 
  [production] | 
            
  | 10:44 | 
  <jdrewniak@deploy1001> | 
  Synchronized portals: Wikimedia Portals Update: [[gerrit:495650| Bumping portals to master (T128546)]] (duration: 00m 49s) | 
  [production] | 
            
  | 10:43 | 
  <jdrewniak@deploy1001> | 
  Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:495650| Bumping portals to master (T128546)]] (duration: 00m 49s) | 
  [production] | 
            
  | 09:56 | 
  <moritzm> | 
  installing chromium security updates on remaining proton hosts | 
  [production] | 
            
  | 09:44 | 
  <moritzm> | 
  installing chromium security updates on proton1001 | 
  [production] | 
            
  | 09:44 | 
  <elukey> | 
  roll restart of aqs on aqs100* to pick up new druid settings | 
  [production] | 
            
  | 08:02 | 
  <marostegui> | 
  Upgrade pc1010 (spare) | 
  [production] | 
            
  | 07:33 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Fully repool db1099 after upgrade (duration: 00m 48s) | 
  [production] | 
            
  | 07:32 | 
  <marostegui> | 
  Upgrade MySQL and kernel on pc2010 (spare) | 
  [production] | 
            
  | 07:13 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Give more traffic to db1099 after upgrade (duration: 00m 48s) | 
  [production] | 
            
  | 06:58 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Give more traffic to db1099 after upgrade (duration: 00m 48s) | 
  [production] | 
            
  | 06:49 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Slowly repool db1099 after upgrade (duration: 00m 52s) | 
  [production] | 
            
  | 06:38 | 
  <kart_> | 
  Finished manual run of unpublished ContentTranslation draft purge script (T217818) | 
  [production] | 
            
  | 06:37 | 
  <marostegui> | 
  Power cycle mw1280 - server down | 
  [production] | 
            
  | 06:35 | 
  <marostegui> | 
  Upgrade mysql and kernel on db1099 | 
  [production] | 
            
  | 06:34 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1099 for upgrade (duration: 03m 01s) | 
  [production] | 
            
  | 06:02 | 
  <effie> | 
  Restarting pdfrender on scb1003 | 
  [production] | 
            
  | 06:02 | 
  <marostegui> | 
  Upgrade MySQL on dbstore1004 (s2, s3, s4) | 
  [production] | 
            
  | 04:00 | 
  <kart_> | 
  Started manual run of unpublished ContentTranslation draft purge script (T217818) | 
  [production] | 
            
  | 03:30 | 
  <kartik@deploy1001> | 
  Finished deploy [cxserver/deploy@101bebd]: Update cxserver to 5a26308 (T216044, T217878) (duration: 04m 01s) | 
  [production] | 
            
  | 03:26 | 
  <kartik@deploy1001> | 
  Started deploy [cxserver/deploy@101bebd]: Update cxserver to 5a26308 (T216044, T217878) | 
  [production] |