|
2019-06-04
§
|
| 11:02 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:02 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:44 |
<jbond42> |
mw1* restarts will be delayed untill 11:15 |
[production] |
| 10:42 |
<jbond42> |
will start rolling reboots of mw1* servers 1t 10:50 |
[production] |
| 09:27 |
<moritzm> |
draining ganeti1006 for eventual reboot to MDS-enabled Linux kernel |
[production] |
| 09:25 |
<jijiki> |
disable puppet on mc* hosts to merge 511963 and 511973 |
[production] |
| 09:01 |
<moritzm> |
draining ganeti1005 for eventual reboot to MDS-enabled Linux kernel |
[production] |
| 08:57 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:32 |
<elukey> |
remove memcached nutcracker config from mw1* hosts (not used). Changes will be picked up when nutcracker will be restarted (after reboots, etc..) - T214275 |
[production] |
| 08:23 |
<moritzm> |
draining ganeti1004 for eventual reboot to MDS-enabled Linux kernel |
[production] |
| 08:04 |
<marostegui> |
Stop MySQL on db2046 to clone db2058 - T221533 |
[production] |
| 08:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2046 (duration: 00m 47s) |
[production] |
| 08:03 |
<elukey> |
restart hive-server2 on an-coord1001 to pick up new GC/Heap settings |
[production] |
| 07:35 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@abcb534]: Use only Proton for PDF rendering - T210651 (duration: 19m 16s) |
[production] |
| 07:22 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:21 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:21 |
<moritzm> |
draining ganeti1002 for eventual reboot to MDS-enabled Linux kernel |
[production] |
| 07:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Move db2058 from s4 to s6 (duration: 00m 47s) |
[production] |
| 07:16 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@abcb534]: Use only Proton for PDF rendering - T210651 |
[production] |
| 06:57 |
<elukey> |
restart hive metastore on an-coord1001 to apply new GC/heap settings |
[production] |
| 06:42 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1091 after upgrade (duration: 00m 48s) |
[production] |
| 06:21 |
<elukey> |
restart pdfrender on scb1002 (flapping) |
[production] |
| 06:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 after upgrade (duration: 00m 47s) |
[production] |
| 05:54 |
<marostegui> |
Stop MySQL on db2078:m3 - T221533 |
[production] |
| 05:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 after upgrade (duration: 00m 47s) |
[production] |
| 05:40 |
<marostegui> |
Stop MySQL on db1091 for MySQL upgrade T224852 |
[production] |
| 05:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1091 for upgrade (duration: 00m 48s) |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1097 after upgrade (duration: 00m 46s) |
[production] |
| 05:19 |
<marostegui> |
Stop MySQL on db1097 for upgrade |
[production] |
| 05:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1097 for upgrade (duration: 00m 47s) |
[production] |
| 04:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1081 from API (duration: 00m 49s) |
[production] |
| 01:10 |
<bstorm_> |
T223406 depooled/repooled labsdb1009 for view updates |
[production] |
| 00:09 |
<bstorm_> |
T223406 repooled labsdb1011 after completing view updates |
[production] |
|
2019-06-03
§
|
| 22:20 |
<bstorm_> |
T223406 depooled labsdb1011 |
[production] |
| 22:08 |
<bstorm_> |
T223406 repooled labsdb1010 after completing view updates |
[production] |
| 21:29 |
<XioNoX> |
drop all ICMP frag on all routers - T224186 |
[production] |
| 19:57 |
<XioNoX> |
stop sampling from cr2-eqiad |
[production] |
| 18:47 |
<XioNoX> |
Add RPKI validators to all routers - T220669 |
[production] |
| 18:35 |
<hashar> |
switch most Quibble jobs to node 10 T222406 - ttps://gerrit.wikimedia.org/r/#/c/integration/config/+/514034/ T222406 |
[production] |
| 18:35 |
<XioNoX> |
drop all ICMP frag on cr1/2-eqiad - T224186 |
[production] |
| 18:17 |
<XioNoX> |
add routinator 0.4.0 to APT repo - T220669 |
[production] |
| 17:16 |
<onimisionipe@deploy1001> |
Finished deploy [wdqs/wdqs@9e3035c]: Blazegraph version wmf.4 (duration: 11m 29s) |
[production] |
| 17:05 |
<onimisionipe@deploy1001> |
Started deploy [wdqs/wdqs@9e3035c]: Blazegraph version wmf.4 |
[production] |
| 16:40 |
<onimisionipe> |
started osm-import on maps2004 - T224395 |
[production] |
| 16:30 |
<bstorm_> |
T223406 depooled labsdb1010 for view updates |
[production] |
| 15:39 |
<bstorm_> |
T223406 labsdb1012 updated views for actor table changes |
[production] |
| 14:45 |
<akosiaris> |
deploy kask in sessionstore kubernetes namespace in eqiad, codfw T220401 |
[production] |
| 14:33 |
<arturo> |
T221769 reimaging cloudservices1003 to stretch |
[production] |
| 14:20 |
<vgutierrez> |
upgrading acme-chief to version 0.17 in acme-chief production instances - T220518 |
[production] |