851-900 of 10000 results (86ms)
2023-04-11 ยง
20:05 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
19:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2127 (T333332)', diff saved to https://phabricator.wikimedia.org/P46349 and previous config saved to /var/cache/conftool/dbconfig/20230411-193640-ladsgroup.json [production]
19:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
19:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
19:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T333332)', diff saved to https://phabricator.wikimedia.org/P46348 and previous config saved to /var/cache/conftool/dbconfig/20230411-193628-ladsgroup.json [production]
19:31 <ejegg> payments-wiki upgraded from ad6e5801 to 153bdf64 [production]
19:29 <ejegg> civicrm upgraded from e2fdb4a4 to 0f37f981 [production]
19:22 <andrew@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudvirtlocal1003'] [production]
19:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P46347 and previous config saved to /var/cache/conftool/dbconfig/20230411-192122-ladsgroup.json [production]
19:19 <eileen> civicrm upgraded from b573aee4 to e2fdb4a4 [production]
19:16 <andrew@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1003'] [production]
19:16 <andrew@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudvirtlocal1002'] [production]
19:10 <andrew@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1002'] [production]
19:08 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
19:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P46346 and previous config saved to /var/cache/conftool/dbconfig/20230411-190616-ladsgroup.json [production]
19:05 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
19:05 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
18:59 <ebysans@deploy2002> Finished deploy [airflow-dags/analytics@d2cd28d]: (no justification provided) (duration: 00m 11s) [production]
18:59 <ebysans@deploy2002> Started deploy [airflow-dags/analytics@d2cd28d]: (no justification provided) [production]
18:58 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
18:57 <andrew@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudvirtlocal1001'] [production]
18:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T333332)', diff saved to https://phabricator.wikimedia.org/P46345 and previous config saved to /var/cache/conftool/dbconfig/20230411-185110-ladsgroup.json [production]
18:50 <andrew@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1001'] [production]
18:38 <demon@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.4 refs T330210 [production]
18:32 <zabe@deploy2002> Finished scap: close wowikiquote (T334482) (duration: 06m 46s) [production]
18:25 <zabe@deploy2002> Started scap: close wowikiquote (T334482) [production]
18:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2109 (T333332)', diff saved to https://phabricator.wikimedia.org/P46344 and previous config saved to /var/cache/conftool/dbconfig/20230411-182024-ladsgroup.json [production]
18:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
18:20 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye [production]
18:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
18:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
18:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
18:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T333332)', diff saved to https://phabricator.wikimedia.org/P46343 and previous config saved to /var/cache/conftool/dbconfig/20230411-181123-ladsgroup.json [production]
17:59 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3006.esams.wmnet with OS bullseye [production]
17:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P46342 and previous config saved to /var/cache/conftool/dbconfig/20230411-175617-ladsgroup.json [production]
17:42 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3006.esams.wmnet with reason: host reimage [production]
17:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P46341 and previous config saved to /var/cache/conftool/dbconfig/20230411-174110-ladsgroup.json [production]
17:38 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3006.esams.wmnet with reason: host reimage [production]
17:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T333332)', diff saved to https://phabricator.wikimedia.org/P46340 and previous config saved to /var/cache/conftool/dbconfig/20230411-172604-ladsgroup.json [production]
17:17 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host lvs3006.esams.wmnet with OS bullseye [production]
17:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1198 (T333332)', diff saved to https://phabricator.wikimedia.org/P46339 and previous config saved to /var/cache/conftool/dbconfig/20230411-171600-ladsgroup.json [production]
17:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
17:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
17:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T333332)', diff saved to https://phabricator.wikimedia.org/P46338 and previous config saved to /var/cache/conftool/dbconfig/20230411-171537-ladsgroup.json [production]
17:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P46337 and previous config saved to /var/cache/conftool/dbconfig/20230411-170031-ladsgroup.json [production]
16:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P46336 and previous config saved to /var/cache/conftool/dbconfig/20230411-164524-ladsgroup.json [production]
16:33 <sbassett> Deployed security mitigation update for T333140 [production]
16:33 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
16:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T333332)', diff saved to https://phabricator.wikimedia.org/P46335 and previous config saved to /var/cache/conftool/dbconfig/20230411-163018-ladsgroup.json [production]
16:30 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]