|
2024-04-09
ยง
|
| 13:23 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60074 and previous config saved to /var/cache/conftool/dbconfig/20240409-132320-arnaudb.json |
[production] |
| 13:21 |
<jforrester@deploy1002> |
sdhehua and jforrester: Continuing with sync |
[production] |
| 13:21 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm |
[production] |
| 13:20 |
<jforrester@deploy1002> |
sdhehua and jforrester: Backport for [[gerrit:1015187|zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:18 |
<sukhe@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet |
[production] |
| 13:17 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade |
[production] |
| 13:17 |
<sukhe@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade |
[production] |
| 13:17 |
<jforrester@deploy1002> |
Started scap: Backport for [[gerrit:1015187|zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184)]] |
[production] |
| 13:16 |
<sukhe@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet |
[production] |
| 13:16 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60073 and previous config saved to /var/cache/conftool/dbconfig/20240409-131605-arnaudb.json |
[production] |
| 13:16 |
<sukhe> |
depool cp4052 for firmware upgrade |
[production] |
| 13:16 |
<jforrester@deploy1002> |
Finished scap: Backport for [[gerrit:1017952|ExtensionDistributor: Add REL1_42 as a beta (T359844)]] (duration: 14m 09s) |
[production] |
| 13:14 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable |
[production] |
| 13:14 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable |
[production] |
| 13:08 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60072 and previous config saved to /var/cache/conftool/dbconfig/20240409-130812-arnaudb.json |
[production] |
| 13:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60071 and previous config saved to /var/cache/conftool/dbconfig/20240409-130543-arnaudb.json |
[production] |
| 13:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:05 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60070 and previous config saved to /var/cache/conftool/dbconfig/20240409-130520-arnaudb.json |
[production] |
| 13:04 |
<jforrester@deploy1002> |
jforrester: Continuing with sync |
[production] |
| 13:04 |
<jforrester@deploy1002> |
jforrester: Backport for [[gerrit:1017952|ExtensionDistributor: Add REL1_42 as a beta (T359844)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:01 |
<jforrester@deploy1002> |
Started scap: Backport for [[gerrit:1017952|ExtensionDistributor: Add REL1_42 as a beta (T359844)]] |
[production] |
| 13:00 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60069 and previous config saved to /var/cache/conftool/dbconfig/20240409-130057-arnaudb.json |
[production] |
| 12:53 |
<vgutierrez> |
uploaded golang-github-gopacket-gopacket_1.2.0-2~wmf1 to apt.wm.o (bookworm) |
[production] |
| 12:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::cinder_backups |
[production] |
| 12:50 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 12:50 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60068 and previous config saved to /var/cache/conftool/dbconfig/20240409-125012-arnaudb.json |
[production] |
| 12:45 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60067 and previous config saved to /var/cache/conftool/dbconfig/20240409-124550-arnaudb.json |
[production] |
| 12:45 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 12:44 |
<elukey@cumin1002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002 |
[production] |
| 12:43 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::cinder_backups |
[production] |
| 12:25 |
<gmodena@deploy1002> |
Finished deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] (duration: 15m 41s) |
[production] |
| 12:21 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-host for host cloudbackup1001-dev.eqiad.wmnet |
[production] |
| 12:20 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60063 and previous config saved to /var/cache/conftool/dbconfig/20240409-122050-arnaudb.json |
[production] |
| 12:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60062 and previous config saved to /var/cache/conftool/dbconfig/20240409-121958-arnaudb.json |
[production] |
| 12:17 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60061 and previous config saved to /var/cache/conftool/dbconfig/20240409-121722-arnaudb.json |
[production] |
| 12:17 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:17 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60060 and previous config saved to /var/cache/conftool/dbconfig/20240409-121622-arnaudb.json |
[production] |
| 12:09 |
<gmodena@deploy1002> |
Started deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] |
[production] |
| 12:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60059 and previous config saved to /var/cache/conftool/dbconfig/20240409-120542-arnaudb.json |
[production] |
| 12:02 |
<btullis@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1004.eqiad.wmnet with OS bullseye |
[production] |
| 12:01 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60058 and previous config saved to /var/cache/conftool/dbconfig/20240409-120115-arnaudb.json |
[production] |
| 11:56 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002" |
[production] |
| 11:55 |
<btullis@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002" |
[production] |
| 11:54 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) matomo1003.eqiad.wmnet on all recursors |
[production] |
| 11:54 |
<btullis@cumin1002> |
START - Cookbook sre.dns.wipe-cache matomo1003.eqiad.wmnet on all recursors |
[production] |
| 11:54 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |