6851-6900 of 10000 results (98ms)
2023-09-07 §
08:51 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
08:46 <hashar@deploy1002> rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.41.0-wmf.24 - T343727 [production]
08:38 <moritzm> installing librsvg security updates [production]
08:23 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on mc2040.codfw.wmnet with reason: T345802 - hw troubleshooting [production]
08:22 <cgoubert@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on mc2040.codfw.wmnet with reason: T345802 - hw troubleshooting [production]
08:22 <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.41.0-wmf.25 refs T343727 [production]
07:57 <moritzm> installing grub2 updates from Bullseye point release [production]
07:40 <moritzm> installing file/libmagic security updates [production]
07:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1002.eqiad.wmnet [production]
07:12 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netboxdb1002.eqiad.wmnet [production]
07:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2002.codfw.wmnet [production]
07:04 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netboxdb2002.codfw.wmnet [production]
06:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet [production]
06:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet [production]
06:29 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1213:3315 (T343198)', diff saved to https://phabricator.wikimedia.org/P52296 and previous config saved to /var/cache/conftool/dbconfig/20230907-062900-arnaudb.json [production]
06:28 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance [production]
06:28 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on wdqs[1003-1004].eqiad.wmnet with reason: reboot [production]
06:28 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on wdqs[1003-1004].eqiad.wmnet with reason: reboot [production]
06:28 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance [production]
06:28 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1210 (T343198)', diff saved to https://phabricator.wikimedia.org/P52295 and previous config saved to /var/cache/conftool/dbconfig/20230907-062838-arnaudb.json [production]
06:13 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P52294 and previous config saved to /var/cache/conftool/dbconfig/20230907-061332-arnaudb.json [production]
05:58 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P52293 and previous config saved to /var/cache/conftool/dbconfig/20230907-055826-arnaudb.json [production]
05:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1210 (T343198)', diff saved to https://phabricator.wikimedia.org/P52292 and previous config saved to /var/cache/conftool/dbconfig/20230907-054320-arnaudb.json [production]
05:26 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
05:24 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
03:40 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
03:39 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
03:26 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
03:24 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
03:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1210 (T343198)', diff saved to https://phabricator.wikimedia.org/P52291 and previous config saved to /var/cache/conftool/dbconfig/20230907-032306-arnaudb.json [production]
03:23 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance [production]
03:22 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance [production]
03:22 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T343198)', diff saved to https://phabricator.wikimedia.org/P52290 and previous config saved to /var/cache/conftool/dbconfig/20230907-032245-arnaudb.json [production]
03:07 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P52289 and previous config saved to /var/cache/conftool/dbconfig/20230907-030739-arnaudb.json [production]
02:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P52288 and previous config saved to /var/cache/conftool/dbconfig/20230907-025233-arnaudb.json [production]
02:37 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T343198)', diff saved to https://phabricator.wikimedia.org/P52287 and previous config saved to /var/cache/conftool/dbconfig/20230907-023727-arnaudb.json [production]
01:10 <tstarling@deploy1002> Synchronized php-1.41.0-wmf.25/extensions/Phonos/extension.json: fix breakage of Phonos on parser-cached pages T345414 (duration: 06m 59s) [production]
00:30 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1200 (T343198)', diff saved to https://phabricator.wikimedia.org/P52286 and previous config saved to /var/cache/conftool/dbconfig/20230907-003038-arnaudb.json [production]
00:30 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
00:30 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
00:30 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T343198)', diff saved to https://phabricator.wikimedia.org/P52285 and previous config saved to /var/cache/conftool/dbconfig/20230907-003017-arnaudb.json [production]
00:15 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P52284 and previous config saved to /var/cache/conftool/dbconfig/20230907-001510-arnaudb.json [production]
00:00 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P52283 and previous config saved to /var/cache/conftool/dbconfig/20230907-000004-arnaudb.json [production]
2023-09-06 §
23:44 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T343198)', diff saved to https://phabricator.wikimedia.org/P52282 and previous config saved to /var/cache/conftool/dbconfig/20230906-234458-arnaudb.json [production]
22:10 <bking@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host flink-zk2003.codfw.wmnet [production]
22:10 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host flink-zk2003.codfw.wmnet with OS bookworm [production]
21:56 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on flink-zk2003.codfw.wmnet with reason: host reimage [production]
21:53 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on flink-zk2003.codfw.wmnet with reason: host reimage [production]
21:44 <eevans@cumin1001> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [production]
21:42 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1185 (T343198)', diff saved to https://phabricator.wikimedia.org/P52281 and previous config saved to /var/cache/conftool/dbconfig/20230906-214205-arnaudb.json [production]