|
2020-06-16
§
|
| 15:15 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@c652f62]: Regular analytics weekly train [analytics/refinery@c652f62] |
[production] |
| 15:06 |
<elukey> |
reboot an-coord1001 for kernel upgrades |
[production] |
| 14:49 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 14:49 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:45 |
<moritzm> |
rebooting scandium for kernel security update |
[production] |
| 14:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:43 |
<cdanis> |
repool eqiad T243080 |
[production] |
| 14:40 |
<papaul> |
power off ms-be2018 for BBU replacement |
[production] |
| 14:33 |
<cdanis> |
eqiad router upgrades completed! 🎉 T243080 |
[production] |
| 14:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:31 |
<elukey> |
reboot druid100[7,8] for kernel upgrades |
[production] |
| 14:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:22 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11541 and previous config saved to /var/cache/conftool/dbconfig/20200616-141540-marostegui.json |
[production] |
| 14:14 |
<cdanis> |
T243080 cdanis@re1.cr2-eqiad> request chassis routing-engine master switch |
[production] |
| 14:10 |
<moritzm> |
removing stray nginx packages from mw canaries (mw1261-mw1265 and mw1276-mw1283) T255565 |
[production] |
| 14:06 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 14:03 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:56 |
<cdanis> |
T243080 cdanis@re0.cr2-eqiad> request chassis routing-engine master switch |
[production] |
| 13:50 |
<cdanis> |
cr2-eqiad: rebooting RE1 [backup] with new junos version T243080 |
[production] |
| 13:39 |
<cdanis> |
cr2-eqiad: disable transit/peering BGP & bump fr MED T243080 |
[production] |
| 13:32 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Repool db2092 T254462', diff saved to https://phabricator.wikimedia.org/P11535 and previous config saved to /var/cache/conftool/dbconfig/20200616-133241-marostegui.json |
[production] |
| 13:17 |
<XioNoX> |
pfw3-eqiad rollback MED to cr1 to 0 - T243080 |
[production] |
| 13:12 |
<XioNoX> |
add graceful-switchover to cr1-eqiad |
[production] |
| 13:09 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 13:08 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.37 |
[production] |
| 13:06 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:03 |
<cdanis> |
T243080 cdanis@re1.cr1-eqiad> request chassis routing-engine master switch |
[production] |
| 13:03 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:03 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:03 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:01 |
<moritzm> |
rebooting mw2291-mw2334 |
[production] |
| 12:54 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 12:51 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 12:47 |
<jbond42> |
upload new memcache package with TLS to component/memcached16 in buster-wikimedia |
[production] |
| 12:42 |
<XioNoX> |
pfw3-eqiad set MED to cr1 to 300 - T243080 |
[production] |
| 12:38 |
<hnowlan@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 12:31 |
<cdanis> |
T243080 cr1-eqiad: request chassis routing-engine master switch |
[production] |
| 12:31 |
<cdanis> |
cr1-eqiad: request chassis routing-engine master switch |
[production] |
| 12:25 |
<cdanis> |
cr1-eqiad: rebooting RE1 [backup] with new junos version T243080 |
[production] |
| 12:15 |
<cdanis> |
cdanis@re0.cr1-eqiad# commit confirmed 2 comment "force VRRP failover T243080" |
[production] |
| 12:14 |
<cdanis> |
disable transit/peering & increase frack MED on cr1-eqiad T243080 |
[production] |
| 12:09 |
<hnowlan@cumin2001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 11:48 |
<cdanis> |
depooling eqiad for router upgrade T243080 |
[production] |