2951-3000 of 10000 results (78ms)
2024-04-09 ยง
13:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60070 and previous config saved to /var/cache/conftool/dbconfig/20240409-130520-arnaudb.json [production]
13:04 <jforrester@deploy1002> jforrester: Continuing with sync [production]
13:04 <jforrester@deploy1002> jforrester: Backport for [[gerrit:1017952|ExtensionDistributor: Add REL1_42 as a beta (T359844)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:01 <jforrester@deploy1002> Started scap: Backport for [[gerrit:1017952|ExtensionDistributor: Add REL1_42 as a beta (T359844)]] [production]
13:00 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60069 and previous config saved to /var/cache/conftool/dbconfig/20240409-130057-arnaudb.json [production]
12:53 <vgutierrez> uploaded golang-github-gopacket-gopacket_1.2.0-2~wmf1 to apt.wm.o (bookworm) [production]
12:52 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::cinder_backups [production]
12:50 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
12:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60068 and previous config saved to /var/cache/conftool/dbconfig/20240409-125012-arnaudb.json [production]
12:45 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60067 and previous config saved to /var/cache/conftool/dbconfig/20240409-124550-arnaudb.json [production]
12:45 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
12:44 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002 [production]
12:43 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::cinder_backups [production]
12:25 <gmodena@deploy1002> Finished deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] (duration: 15m 41s) [production]
12:21 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host cloudbackup1001-dev.eqiad.wmnet [production]
12:20 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60063 and previous config saved to /var/cache/conftool/dbconfig/20240409-122050-arnaudb.json [production]
12:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60062 and previous config saved to /var/cache/conftool/dbconfig/20240409-121958-arnaudb.json [production]
12:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60061 and previous config saved to /var/cache/conftool/dbconfig/20240409-121722-arnaudb.json [production]
12:17 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
12:17 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
12:16 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
12:16 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
12:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60060 and previous config saved to /var/cache/conftool/dbconfig/20240409-121622-arnaudb.json [production]
12:09 <gmodena@deploy1002> Started deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] [production]
12:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60059 and previous config saved to /var/cache/conftool/dbconfig/20240409-120542-arnaudb.json [production]
12:02 <btullis@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1004.eqiad.wmnet with OS bullseye [production]
12:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60058 and previous config saved to /var/cache/conftool/dbconfig/20240409-120115-arnaudb.json [production]
11:56 <btullis@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002" [production]
11:55 <btullis@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002" [production]
11:54 <btullis@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) matomo1003.eqiad.wmnet on all recursors [production]
11:54 <btullis@cumin1002> START - Cookbook sre.dns.wipe-cache matomo1003.eqiad.wmnet on all recursors [production]
11:54 <btullis@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:54 <btullis@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002" [production]
11:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60057 and previous config saved to /var/cache/conftool/dbconfig/20240409-115035-arnaudb.json [production]
11:49 <hnowlan@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
11:48 <hnowlan@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
11:47 <hnowlan@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
11:47 <hnowlan@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
11:46 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60056 and previous config saved to /var/cache/conftool/dbconfig/20240409-114607-arnaudb.json [production]
11:44 <arnaudb@cumin1002> START - Cookbook sre.mysql.clone Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet [production]
11:44 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60055 and previous config saved to /var/cache/conftool/dbconfig/20240409-114411-arnaudb.json [production]
11:44 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
11:43 <hnowlan@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
11:43 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
11:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60054 and previous config saved to /var/cache/conftool/dbconfig/20240409-114349-arnaudb.json [production]
11:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Cloning db2112 in db2212 for T355422', diff saved to https://phabricator.wikimedia.org/P60053 and previous config saved to /var/cache/conftool/dbconfig/20240409-114302-arnaudb.json [production]
11:42 <hnowlan@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
11:42 <hnowlan@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
11:42 <hnowlan@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
11:41 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422 [production]