7051-7100 of 10000 results (91ms)
2023-09-22 §
13:39 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host pc1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
12:23 <brouberol@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad [production]
12:17 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet [production]
12:13 <fnegri@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet [production]
11:58 <brouberol@cumin1001> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [production]
11:42 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host apt-staging2001.codfw.wmnet with OS bookworm [production]
11:41 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
11:41 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
11:28 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on apt-staging2001.codfw.wmnet with reason: host reimage [production]
11:25 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on apt-staging2001.codfw.wmnet with reason: host reimage [production]
11:09 <eoghan@cumin1001> START - Cookbook sre.hosts.reimage for host apt-staging2001.codfw.wmnet with OS bookworm [production]
10:26 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1001.eqiad.wmnet [production]
10:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host testreduce1001.eqiad.wmnet [production]
10:00 <fabfur> repool cp1090 (T346874) [production]
09:53 <fnegri@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet [production]
09:50 <fnegri@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet [production]
09:45 <fnegri@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcumin1001.eqiad.wmnet [production]
09:45 <fnegri@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet [production]
09:43 <fnegri@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcumin1001.eqiad.wmnet [production]
09:43 <fnegri@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet [production]
09:22 <Amir1> dbmaint on s2@eqiad (T343198) [production]
09:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on 16 hosts with reason: Schema change [production]
09:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on 16 hosts with reason: Schema change [production]
09:18 <Amir1> dbmaint on s6@eqiad (T343198) [production]
09:18 <Amir1> dbmaint on s6@eqiad (T343198) [production]
09:13 <moritzm> installing perf updates on bookworm hosts [production]
09:12 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on 15 hosts with reason: Schema change [production]
09:12 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on 15 hosts with reason: Schema change [production]
09:06 <moritzm> installing perf updates on buster hosts [production]
08:51 <Amir1> dbmaint on s4@eqiad (T343198) [production]
08:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on 20 hosts with reason: Schema change [production]
08:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on 20 hosts with reason: Schema change [production]
07:45 <hashar> Upgrading CI Jenkins from 2.401.3 to 2.414.2 [production]
07:36 <hashar> Restarting Gerrit to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/953967 "Link account creation to IDM" # T345226 [production]
07:06 <moritzm> installing mutt security updates [production]
06:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1132', diff saved to https://phabricator.wikimedia.org/P52577 and previous config saved to /var/cache/conftool/dbconfig/20230922-063617-root.json [production]
06:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1132', diff saved to https://phabricator.wikimedia.org/P52576 and previous config saved to /var/cache/conftool/dbconfig/20230922-063212-root.json [production]
05:13 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
00:43 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
00:43 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
00:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T343198)', diff saved to https://phabricator.wikimedia.org/P52575 and previous config saved to /var/cache/conftool/dbconfig/20230922-004330-arnaudb.json [production]
00:28 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P52574 and previous config saved to /var/cache/conftool/dbconfig/20230922-002823-arnaudb.json [production]
00:13 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P52573 and previous config saved to /var/cache/conftool/dbconfig/20230922-001316-arnaudb.json [production]
2023-09-21 §
23:58 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T343198)', diff saved to https://phabricator.wikimedia.org/P52572 and previous config saved to /var/cache/conftool/dbconfig/20230921-235810-arnaudb.json [production]
22:02 <ejegg> Standalone (listener) SmashPig upgraded from ca5b6218 to 2412df22 [production]
20:28 <brennen> end of UTC late backport & config window [production]
20:27 <brennen@deploy2002> Finished scap: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]] (duration: 21m 36s) [production]
20:18 <brennen@deploy2002> dani and brennen: Continuing with sync [production]
20:17 <brennen@deploy2002> dani and brennen: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:06 <brennen@deploy2002> Started scap: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]] [production]