1451-1500 of 10000 results (108ms)
2024-10-10 §
06:24 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
06:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
06:10 <jelto@cumin1002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version [production]
06:10 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version [production]
06:03 <XioNoX> cr2-eqsin> request vmhost snapshot - T375961 [production]
03:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P69541 and previous config saved to /var/cache/conftool/dbconfig/20241010-031553-ladsgroup.json [production]
03:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1198 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P69540 and previous config saved to /var/cache/conftool/dbconfig/20241010-031531-ladsgroup.json [production]
03:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P69539 and previous config saved to /var/cache/conftool/dbconfig/20241010-030048-ladsgroup.json [production]
03:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1198 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P69538 and previous config saved to /var/cache/conftool/dbconfig/20241010-030025-ladsgroup.json [production]
02:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P69537 and previous config saved to /var/cache/conftool/dbconfig/20241010-024543-ladsgroup.json [production]
02:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P69536 and previous config saved to /var/cache/conftool/dbconfig/20241010-024519-ladsgroup.json [production]
02:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1223 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P69535 and previous config saved to /var/cache/conftool/dbconfig/20241010-023037-ladsgroup.json [production]
02:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1198 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P69534 and previous config saved to /var/cache/conftool/dbconfig/20241010-023014-ladsgroup.json [production]
02:02 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: repooling eqsin after cr2-eqsin replaced, T375961] [production]
02:02 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repooling eqsin after cr2-eqsin replaced, T375961] [production]
01:50 <sukhe> restart bird on doh5001 and dns5003 to resolve flapping BFD session after cr2-eqsin junos upgrade [production]
01:01 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1198.eqiad.wmnet onto db1223.eqiad.wmnet [production]
00:46 <denisse@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host prometheus1006.eqiad.wmnet [production]
00:41 <eileen> civicrm upgraded from 3b6a7cbb to 07dee21c [production]
00:27 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet [production]
00:26 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet [production]
00:19 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet [production]
00:19 <denisse@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host prometheus2005.codfw.wmnet [production]
00:02 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet [production]
00:02 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet [production]
2024-10-09 §
23:52 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet [production]
23:51 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet [production]
23:49 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org [production]
23:43 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet [production]
23:41 <denisse@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host prometheus1005.eqiad.wmnet [production]
23:26 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet [production]
23:25 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet [production]
23:18 <denisse@cumin2002> START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet [production]
23:07 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: security release 20241009 [production]
23:02 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release 20241009 [production]
22:51 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1012.eqiad.wmnet with OS bookworm [production]
22:51 <jclark@cumin1002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
22:51 <ladsgroup@cumin1002> START - Cookbook sre.mysql.clone of db1198.eqiad.wmnet onto db1223.eqiad.wmnet [production]
22:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool for reclone (T375652)', diff saved to https://phabricator.wikimedia.org/P69532 and previous config saved to /var/cache/conftool/dbconfig/20241009-225055-ladsgroup.json [production]
22:40 <eileen> civicrm upgraded from cc7c7744 to 3b6a7cbb [production]
22:35 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: security release 20241009-3 [production]
22:30 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 20241009-3 [production]
22:28 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: security release 20241009-3 [production]
22:28 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 20241009-3 [production]
22:01 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: release 20241009-3 [production]
22:00 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: release 20241009-3 [production]
21:57 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: release 20241009-3 [production]
21:57 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: release 20241009-3 [production]
21:55 <dzahn@cumin2002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: security release 20241009-2 [production]
21:54 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release 20241009-2 [production]