5051-5100 of 10000 results (107ms)
2024-01-11 §
12:33 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2124.codfw.wmnet with OS bookworm [production]
12:20 <pfischer@deploy2002> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:20 <pfischer@deploy2002> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:11 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2124.codfw.wmnet with reason: host reimage [production]
12:08 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2124.codfw.wmnet with reason: host reimage [production]
12:00 <kamila@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
12:00 <kamila@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
11:59 <moritzm> installing Python 2.7 security updates on Bullseye [production]
11:50 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db2124.codfw.wmnet with OS bookworm [production]
11:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2124 T354506', diff saved to https://phabricator.wikimedia.org/P54606 and previous config saved to /var/cache/conftool/dbconfig/20240111-114930-marostegui.json [production]
11:19 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54605 and previous config saved to /var/cache/conftool/dbconfig/20240111-111958-root.json [production]
11:04 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 75%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54604 and previous config saved to /var/cache/conftool/dbconfig/20240111-110453-root.json [production]
10:54 <moritzm> installing Linux 5.10.205 updates on Bullseye hosts [production]
10:49 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54603 and previous config saved to /var/cache/conftool/dbconfig/20240111-104948-root.json [production]
10:34 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54602 and previous config saved to /var/cache/conftool/dbconfig/20240111-103443-root.json [production]
10:31 <moritzm> installing exim4 security updates [production]
10:31 <kharlan@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
10:30 <kharlan@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
10:28 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: druid::public::worker [production]
10:26 <kharlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/ipoid: apply [production]
10:26 <kharlan@deploy2002> helmfile [codfw] START helmfile.d/services/ipoid: apply [production]
10:19 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54601 and previous config saved to /var/cache/conftool/dbconfig/20240111-101938-root.json [production]
10:13 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: druid::public::worker [production]
10:12 <kharlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/ipoid: apply [production]
10:12 <kharlan@deploy2002> helmfile [codfw] START helmfile.d/services/ipoid: apply [production]
10:04 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54600 and previous config saved to /var/cache/conftool/dbconfig/20240111-100433-root.json [production]
10:04 <sfaci@deploy2002> helmfile [staging] DONE helmfile.d/services/edit-analytics: apply [production]
10:03 <sfaci@deploy2002> helmfile [staging] START helmfile.d/services/edit-analytics: apply [production]
10:03 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
10:00 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
10:00 <kharlan@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
09:58 <kharlan@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
09:53 <kharlan@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
09:49 <marostegui@cumin1002> dbctl commit (dc=all): 'db1201 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P54599 and previous config saved to /var/cache/conftool/dbconfig/20240111-094928-root.json [production]
09:39 <hashar> Gerrit back up and operational, now running version 3.6.8 [production]
09:33 <hashar> Gerrit restarted and its reindexing all changes T309870 [production]
09:23 <hashar@deploy2002> Finished deploy [gerrit/gerrit@e099b0b]: Gerrit to version 3.6.8 # T309870 (duration: 00m 07s) [production]
09:23 <hashar@deploy2002> Started deploy [gerrit/gerrit@e099b0b]: Gerrit to version 3.6.8 # T309870 [production]
09:22 <hashar@deploy2002> Finished deploy [gerrit/gerrit@e099b0b]: Gerrit to version 3.6.8 # T309870 (duration: 00m 27s) [production]
09:21 <hashar@deploy2002> Started deploy [gerrit/gerrit@e099b0b]: Gerrit to version 3.6.8 # T309870 [production]
09:21 <hashar> Stopping Gerrit [production]
09:10 <hashar> gerrit: `ssh -p 29418 gerrit.wikimedia.org gerrit copy-approvals` # T309870 [production]
09:06 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1201.eqiad.wmnet with OS bookworm [production]
08:45 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1201.eqiad.wmnet with reason: host reimage [production]
08:42 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1201.eqiad.wmnet with reason: host reimage [production]
02:50 <urandom> decommissioning cassandra, restbase2014-{a,b,c} — T352469 [production]
2024-01-10 §
22:29 <herron@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad [production]
22:05 <herron@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad [production]
21:54 <herron@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw [production]
21:36 <Dreamy_Jazz> UTC late deploys done [production]