| 2018-11-22
      
      § | 
    
  | 09:06 | <moritzm> | installing jasper security updates | [production] | 
            
  | 08:21 | <marostegui> | Deploy schema change on s1 codfw master (db2048) with replication - T86339 | [production] | 
            
  | 08:19 | <marostegui> | Deploy schema change on db1062 (s7 master) - T86339 | [production] | 
            
  | 08:19 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1094 - T86339 (duration: 00m 46s) | [production] | 
            
  | 08:17 | <marostegui> | Deploy schema change on db1094 - T86339 | [production] | 
            
  | 08:16 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1094 - T86339 (duration: 00m 49s) | [production] | 
            
  | 06:58 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1086 - T86339 (duration: 00m 46s) | [production] | 
            
  | 06:55 | <marostegui> | Deploy schema change on db1086 - T86339 | [production] | 
            
  | 06:55 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1086 - T86339 (duration: 00m 46s) | [production] | 
            
  | 06:51 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1079 - T86339 (duration: 00m 46s) | [production] | 
            
  | 06:48 | <marostegui> | Deploy schema change on db1079 (sanitarium master) with replication - T86339 | [production] | 
            
  | 06:48 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1079 - T86339 (duration: 00m 51s) | [production] | 
            
  | 02:25 | <mutante> | stat1005 - started nagios-nrpe-server | [production] | 
            
  
    | 2018-11-21
      
      § | 
    
  | 23:34 | <mutante> | rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036 | [production] | 
            
  | 21:16 | <robh> | cp5001 is offline running hardware tests after firmware updates to see if memory error still exists.  ref: T199675 | [production] | 
            
  | 20:55 | <robh> | cp5001 reboot for firmware update | [production] | 
            
  | 19:54 | <milimetric@deploy1001> | Finished deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints (duration: 05m 34s) | [production] | 
            
  | 19:49 | <milimetric@deploy1001> | Started deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints | [production] | 
            
  | 19:43 | <ejegg> | updated fundraising internal dashboard from b01458b260 to 5e9fb9a3ef | [production] | 
            
  | 17:21 | <elukey> | manually started systemd-journald.service on scb1001 after OOM | [production] | 
            
  | 17:20 | <jynus> | stop and upgrade db2081 | [production] | 
            
  | 17:04 | <jynus> | stop and upgrade db2080 | [production] | 
            
  | 16:40 | <jynus> | stop and upgrade db2066 | [production] | 
            
  | 16:37 | <bawolff> | deploy patch T209794 | [production] | 
            
  | 16:25 | <jynus> | stop and upgrade db2063 | [production] | 
            
  | 15:15 | <jynus> | stop and upgrade db2073 | [production] | 
            
  | 15:15 | <banyek@deploy1001> | Synchronized wmf-config/db-eqiad.php: T85757: repool db1085 (duration: 00m 46s) | [production] | 
            
  | 15:03 | <banyek> | repooling db1085 after schema change (T85757) | [production] | 
            
  | 15:00 | <banyek> | restarting replication on db1085 (T85757) | [production] | 
            
  | 14:31 | <banyek> | stopping replication on db1085 (T85757) | [production] | 
            
  | 14:27 | <banyek@deploy1001> | Synchronized wmf-config/db-eqiad.php: T85757: depool db1085 (duration: 00m 46s) | [production] | 
            
  | 14:22 | <banyek> | depooling db1085 due schema change (T85757) | [production] | 
            
  | 13:39 | <jynus> | stop and upgrade db2077 | [production] | 
            
  | 13:05 | <XioNoX> | remove BGP session to 2603 on cr4-ulsfo | [production] | 
            
  | 12:13 | <jynus> | stop and upgrade db2076 | [production] | 
            
  | 12:08 | <banyek> | running schema change on dbstore1001:3316 (T85757) | [production] | 
            
  | 12:08 | <banyek> | running schema change on dbstore1001 (T85757) | [production] | 
            
  | 12:04 | <jynus> | stop and upgrade db2075 | [production] | 
            
  | 11:53 | <banyek> | running schema change on dbstore1002 (T85757) | [production] | 
            
  | 11:00 | <akosiaris> | disable puppet on ores2* ores1* for gradual rollout of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474694/1/modules/ores/manifests/web.pp | [production] | 
            
  | 10:55 | <jynus> | stop and upgrade db2074 | [production] | 
            
  | 10:51 | <_joe_> | uploading prometheus-php-fpm-exporter to stretch-wikimedia main, T209573 | [production] | 
            
  | 10:44 | <banyek@deploy1001> | Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3316 (duration: 00m 46s) | [production] | 
            
  | 10:42 | <banyek> | repooling db1096:3316 after schema change (T85757) | [production] | 
            
  | 10:30 | <jynus> | stop and upgrade db2095 | [production] | 
            
  | 10:26 | <banyek@deploy1001> | Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3316 (duration: 00m 45s) | [production] | 
            
  | 10:26 | <godog> | initial weight for new ms-be2* hosts (all but ms-be2047) - T209395 | [production] | 
            
  | 10:23 | <banyek> | depooling db1096:3316 due schema change (T85757) | [production] | 
            
  | 10:14 | <banyek@deploy1001> | Synchronized wmf-config/db-eqiad.php: T85757: repool db1098:3316 (duration: 00m 46s) | [production] | 
            
  | 10:13 | <banyek@deploy1001> | sync-file aborted: T85757: depool db1098:3316 (duration: 00m 03s) | [production] |