8151-8200 of 10000 results (106ms)
2023-11-13 §
09:08 <jnuche@deploy2002> bd808 and jnuche: Continuing with sync [production]
09:08 <jnuche@deploy2002> bd808 and jnuche: Backport for [[gerrit:973247|Fix BlockDisablesLogin recursion (T350836 T350080)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:07 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1130', diff saved to https://phabricator.wikimedia.org/P53302 and previous config saved to /var/cache/conftool/dbconfig/20231113-090750-arnaudb.json [production]
09:06 <jnuche@deploy2002> Started scap: Backport for [[gerrit:973247|Fix BlockDisablesLogin recursion (T350836 T350080)]] [production]
08:58 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host webperf2003.codfw.wmnet [production]
08:55 <godog> bounce prometheus eqiad for k8s / k8s-aux - T343529 [production]
08:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1130 (T348183)', diff saved to https://phabricator.wikimedia.org/P53301 and previous config saved to /var/cache/conftool/dbconfig/20231113-085243-arnaudb.json [production]
08:49 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1130 (T348183)', diff saved to https://phabricator.wikimedia.org/P53300 and previous config saved to /var/cache/conftool/dbconfig/20231113-084945-arnaudb.json [production]
08:49 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
08:49 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
08:45 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host webperf2003.codfw.wmnet [production]
08:39 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host graphite2004.codfw.wmnet [production]
08:34 <hashar@deploy2002> Finished deploy [integration/docroot@bc8aaba]: Add more libraries to doc.wikimedia.org homepage - T327604 (duration: 00m 06s) [production]
08:34 <hashar@deploy2002> Started deploy [integration/docroot@bc8aaba]: Add more libraries to doc.wikimedia.org homepage - T327604 [production]
08:30 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host graphite2004.codfw.wmnet [production]
08:29 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host arclamp2001.codfw.wmnet [production]
08:20 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host arclamp2001.codfw.wmnet [production]
07:54 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: search::loader [production]
07:42 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: search::loader [production]
2023-11-12 §
21:28 <jiji@cumin2002> END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0) [production]
21:27 <jiji@cumin2002> START - Cookbook sre.mediawiki.restart-appservers [production]
21:26 <effie> restart php-fpm on jobrunners [production]
2023-11-11 §
01:47 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1058.eqiad.wmnet with OS bookworm [production]
01:20 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1058.eqiad.wmnet with reason: host reimage [production]
01:17 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1058.eqiad.wmnet with reason: host reimage [production]
01:03 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1058.eqiad.wmnet with OS bookworm [production]
00:14 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1061.eqiad.wmnet with OS bookworm [production]
2023-11-10 §
23:51 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1061.eqiad.wmnet with reason: host reimage [production]
23:48 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1061.eqiad.wmnet with reason: host reimage [production]
23:34 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1061.eqiad.wmnet with OS bookworm [production]
21:00 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1064.eqiad.wmnet with OS bookworm [production]
20:51 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1064.eqiad.wmnet with OS bookworm [production]
20:25 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1064.eqiad.wmnet with reason: host reimage [production]
20:22 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1064.eqiad.wmnet with reason: host reimage [production]
20:04 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bookworm [production]
18:47 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS bookworm [production]
18:04 <bvibber> brion adding more vp9 backfill to the transcode runs on mwmaint2002 (requeueTranscodes -> job queue runners). Should increase load on transcode scaler job runners but not elsewhere [production]
17:54 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1066.eqiad.wmnet with OS bookworm [production]
17:53 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1064.eqiad.wmnet with reason: host reimage [production]
17:52 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1067.eqiad.wmnet with OS bookworm [production]
17:51 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS bookworm [production]
17:50 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage [production]
17:49 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1064.eqiad.wmnet with reason: host reimage [production]
17:48 <topranks> withdrawing IPv6 prefixes announced to AS1299 in esams to troubleshoot connectivity problem report [production]
17:47 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage [production]
17:33 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bookworm [production]
17:33 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bookworm [production]
17:33 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1063.eqiad.wmnet with OS bookworm [production]
17:33 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1064.eqiad.wmnet with OS bookworm [production]
17:01 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [production]