4551-4600 of 10000 results (147ms)
2025-06-13 §
01:55 <eileen> * postinstall [production]
01:17 <ladsgroup@cumin2002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1251 gradually with 4 steps - Firmware update done [production]
01:14 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-ulsfo and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) [production]
00:31 <ladsgroup@cumin2002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1253 gradually with 4 steps - Firmware updated [production]
00:29 <ladsgroup@cumin2002> START - Cookbook sre.mysql.pool db1251 gradually with 4 steps - Firmware update done [production]
00:13 <ladsgroup@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1251.eqiad.wmnet with reason: Firmware update [production]
00:13 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1251.eqiad.wmnet [production]
00:08 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host db1251.eqiad.wmnet [production]
2025-06-12 §
23:54 <ladsgroup@cumin2002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1254 gradually with 4 steps - Firmware update done [production]
23:53 <ladsgroup@cumin2002> START - Cookbook sre.hosts.reboot-single for host db1251.eqiad.wmnet [production]
23:53 <ladsgroup@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1251.eqiad.wmnet [production]
23:48 <ladsgroup@cumin2002> dbctl commit (dc=all): 'Depool db1251 for firmware update (T396648)', diff saved to https://phabricator.wikimedia.org/P77872 and previous config saved to /var/cache/conftool/dbconfig/20250612-234855-ladsgroup.json [production]
23:47 <ladsgroup@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1251.eqiad.wmnet with reason: Firmware update [production]
23:43 <ladsgroup@cumin2002> START - Cookbook sre.mysql.pool db1253 gradually with 4 steps - Firmware updated [production]
23:43 <ladsgroup@cumin1002> END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99) [production]
23:37 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1252.eqiad.wmnet [production]
23:36 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host db1252.eqiad.wmnet [production]
23:36 <ladsgroup@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
23:21 <ladsgroup@cumin2002> START - Cookbook sre.hosts.reboot-single for host db1252.eqiad.wmnet [production]
23:20 <ladsgroup@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1252.eqiad.wmnet [production]
23:14 <bvibber@deploy1003> Finished scap sync-world: Backport for [[gerrit:1156454|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]], [[gerrit:1156455|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]] (duration: 61m 18s) [production]
23:13 <ladsgroup@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1252.eqiad.wmnet with reason: Firmware update [production]
23:06 <ladsgroup@cumin2002> START - Cookbook sre.mysql.pool db1254 gradually with 4 steps - Firmware update done [production]
23:01 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1254.eqiad.wmnet [production]
23:00 <ladsgroup@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host db1254.eqiad.wmnet [production]
23:00 <bvibber@deploy1003> bvibber: Continuing with sync [production]
22:59 <bvibber@deploy1003> bvibber: Backport for [[gerrit:1156454|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]], [[gerrit:1156455|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:58 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bullseye [production]
22:45 <ladsgroup@cumin2002> START - Cookbook sre.hosts.reboot-single for host db1254.eqiad.wmnet [production]
22:44 <ladsgroup@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1254.eqiad.wmnet [production]
22:43 <ladsgroup@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1254.eqiad.wmnet with reason: Firmware upgrade (T396648) [production]
22:43 <ladsgroup@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1254.eqiad.wmnet [production]
22:42 <ladsgroup@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1254.eqiad.wmnet [production]
22:39 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1254.eqiad.wmnet with reason: Firmware upgrade (T396648) [production]
22:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool db1254 (T396648)', diff saved to https://phabricator.wikimedia.org/P77867 and previous config saved to /var/cache/conftool/dbconfig/20250612-223834-ladsgroup.json [production]
22:13 <bvibber@deploy1003> Started scap sync-world: Backport for [[gerrit:1156454|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]], [[gerrit:1156455|Specify Lua transform arguments on {{#chart:}} invocations (T395610)]] [production]
22:07 <maryum> Deploy security fix for T394863 [production]
22:00 <maryum> Deployed security fix for T396413 [production]
21:53 <maryum> Deploy security fix for T396524 [production]
21:51 <cstone> civicrm upgraded from 870eed23 to f2f33db5 [production]
21:43 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [production]
21:39 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [production]
21:24 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bullseye [production]
21:18 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1016.eqiad.wmnet with OS bullseye [production]
21:15 <cstone> SmashPig upgraded from 042d5a5b to 84c0668b [production]
21:01 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: host reimage [production]
20:58 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: host reimage [production]
20:48 <bvibber@deploy1003> Finished scap sync-world: Backport for [[gerrit:1156420|Fix for multiple charts on same page using mix of transforms (T396512)]], [[gerrit:1156421|Fix for multiple charts on same page using mix of transforms (T396512)]] (duration: 09m 50s) [production]
20:43 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudcephosd1016.eqiad.wmnet with OS bullseye [production]
20:41 <andrew@cumin1002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1016.eqiad.wmnet'] [production]