2001-2050 of 10000 results (91ms)
2023-05-09 ยง
16:35 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2012.codfw.wmnet with OS bullseye [production]
16:33 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudbackup2001-dev.codfw.wmnet with OS bullseye [production]
16:33 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:33 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for cloudcontrol2001-dev - pt1979@cumin2002" [production]
16:33 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['lvs2012'] [production]
16:32 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for cloudcontrol2001-dev - pt1979@cumin2002" [production]
16:30 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
16:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1223 (T335845)', diff saved to https://phabricator.wikimedia.org/P48042 and previous config saved to /var/cache/conftool/dbconfig/20230509-162904-ladsgroup.json [production]
16:27 <rzl> resumed puppet on appservers - T230382 [production]
16:25 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs2012'] [production]
16:23 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['lvs2012'] [production]
16:23 <rzl> rzl@mwdebug1001:~$ sudo apache2ctl restart [production]
16:22 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs2012'] [production]
16:21 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-coord1002.eqiad.wmnet [production]
16:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P48041 and previous config saved to /var/cache/conftool/dbconfig/20230509-162115-ladsgroup.json [production]
16:19 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
16:15 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
16:15 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-coord1002.eqiad.wmnet [production]
16:14 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
16:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P48039 and previous config saved to /var/cache/conftool/dbconfig/20230509-161358-ladsgroup.json [production]
16:11 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
16:10 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
16:10 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
16:09 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:09 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:08 <jnuche@deploy1002> Installing scap version "4.52.1" for 593 hosts [production]
16:07 <rzl> stopping puppet on appservers - T230382 [production]
16:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs2012.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P48038 and previous config saved to /var/cache/conftool/dbconfig/20230509-160608-ladsgroup.json [production]
16:04 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on vrts2001.codfw.wmnet with reason: host reimage [production]
16:01 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on vrts2001.codfw.wmnet with reason: host reimage [production]
15:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P48037 and previous config saved to /var/cache/conftool/dbconfig/20230509-155852-ladsgroup.json [production]
15:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174 (T335845)', diff saved to https://phabricator.wikimedia.org/P48036 and previous config saved to /var/cache/conftool/dbconfig/20230509-155102-ladsgroup.json [production]
15:50 <aokoth@cumin1001> START - Cookbook sre.ganeti.reimage for host vrts2001.codfw.wmnet with OS bullseye [production]
15:48 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on vrts2001.codfw.wmnet with reason: Re-image w/ Bullseye [production]
15:48 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on vrts2001.codfw.wmnet with reason: Re-image w/ Bullseye [production]
15:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1223 (T335845)', diff saved to https://phabricator.wikimedia.org/P48035 and previous config saved to /var/cache/conftool/dbconfig/20230509-154346-ladsgroup.json [production]
15:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2174 (T335845)', diff saved to https://phabricator.wikimedia.org/P48034 and previous config saved to /var/cache/conftool/dbconfig/20230509-154338-ladsgroup.json [production]
15:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance [production]
15:43 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:43 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for cloudcontrol2001-dev - pt1979@cumin2002" [production]
15:43 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance [production]
15:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173 (T335845)', diff saved to https://phabricator.wikimedia.org/P48033 and previous config saved to /var/cache/conftool/dbconfig/20230509-154313-ladsgroup.json [production]
15:42 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS entries for cloudcontrol2001-dev - pt1979@cumin2002" [production]
15:40 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1223 (T335845)', diff saved to https://phabricator.wikimedia.org/P48032 and previous config saved to /var/cache/conftool/dbconfig/20230509-153715-ladsgroup.json [production]
15:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance [production]
15:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance [production]
15:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T335845)', diff saved to https://phabricator.wikimedia.org/P48031 and previous config saved to /var/cache/conftool/dbconfig/20230509-153651-ladsgroup.json [production]
15:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P48030 and previous config saved to /var/cache/conftool/dbconfig/20230509-152804-ladsgroup.json [production]