1051-1100 of 10000 results (82ms)
2024-04-27 ยง
23:11 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance [production]
23:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61285 and previous config saved to /var/cache/conftool/dbconfig/20240427-231112-ladsgroup.json [production]
22:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P61284 and previous config saved to /var/cache/conftool/dbconfig/20240427-225604-ladsgroup.json [production]
22:44 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp7001.magru.wmnet with OS bullseye [production]
22:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P61283 and previous config saved to /var/cache/conftool/dbconfig/20240427-224057-ladsgroup.json [production]
22:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61282 and previous config saved to /var/cache/conftool/dbconfig/20240427-222548-ladsgroup.json [production]
21:16 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cp7001.magru.wmnet with OS bullseye [production]
21:14 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7001'] [production]
21:06 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7001'] [production]
21:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7001'] [production]
20:42 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7001'] [production]
20:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7001.mgmt.magru.wmnet with reboot policy FORCED [production]
20:29 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host cp7001.mgmt.magru.wmnet with reboot policy FORCED [production]
20:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2167 (T352010)', diff saved to https://phabricator.wikimedia.org/P61281 and previous config saved to /var/cache/conftool/dbconfig/20240427-202602-ladsgroup.json [production]
20:25 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
20:25 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
20:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61280 and previous config saved to /var/cache/conftool/dbconfig/20240427-202539-ladsgroup.json [production]
20:25 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp7001.mgmt.magru.wmnet with reboot policy FORCED [production]
20:25 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host cp7001.mgmt.magru.wmnet with reboot policy FORCED [production]
20:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7001 DNS add - pt1979@cumin2002" [production]
20:22 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7001 DNS add - pt1979@cumin2002" [production]
20:20 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
20:15 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:14 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
20:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P61279 and previous config saved to /var/cache/conftool/dbconfig/20240427-201031-ladsgroup.json [production]
19:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P61278 and previous config saved to /var/cache/conftool/dbconfig/20240427-195524-ladsgroup.json [production]
19:46 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:46 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002" [production]
19:45 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002" [production]
19:43 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
19:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61277 and previous config saved to /var/cache/conftool/dbconfig/20240427-194017-ladsgroup.json [production]
19:39 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:39 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002" [production]
19:38 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002" [production]
19:34 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
17:55 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:54 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru oob - ayounsi@cumin1002" [production]
17:42 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru oob - ayounsi@cumin1002" [production]
17:39 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
17:06 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:06 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru - ayounsi@cumin1002" [production]
17:03 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru - ayounsi@cumin1002" [production]
17:01 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
14:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61275 and previous config saved to /var/cache/conftool/dbconfig/20240427-144642-ladsgroup.json [production]
14:46 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance [production]
14:46 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance [production]
09:51 <volans> manually upgraded wmflib in netbox1002/2002's Netbox's venv [production]
08:58 <volans> restarted uwsgi on netbox1002 to pickup the latest wmflib with magru [production]
07:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61274 and previous config saved to /var/cache/conftool/dbconfig/20240427-075233-ladsgroup.json [production]