3101-3150 of 10000 results (95ms)
2024-03-21 §
16:38 <elukey@cumin1002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM registry1003.eqiad.wmnet [production]
16:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P58880 and previous config saved to /var/cache/conftool/dbconfig/20240321-163708-marostegui.json [production]
16:36 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov2005.codfw.wmnet with OS bullseye [production]
16:36 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS bullseye [production]
16:35 <elukey> edit /etc/network/interfaces on registry1003 (ens5 => ens13) - T360637 [production]
16:34 <sukhe@cumin1002> START - Cookbook sre.dns.roll-reboot rolling reboot on A:dns-rec and not P{dns1004*} and A:dnsbox [production]
16:33 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbprov2005.codfw.wmnet with OS bullseye [production]
16:27 <elukey@cumin1002> START - Cookbook sre.ganeti.reboot-vm for VM registry1003.eqiad.wmnet [production]
16:25 <elukey> expand vram for registry100[3,4] from 4G to 6G - T360637 [production]
16:25 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=registry1003.eqiad.wmnet [production]
16:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P58879 and previous config saved to /var/cache/conftool/dbconfig/20240321-162200-marostegui.json [production]
16:17 <sukhe@cumin1002> END (ERROR) - Cookbook sre.dns.roll-reboot (exit_code=97) rolling reboot on A:dnsbox [production]
16:17 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS bullseye [production]
16:14 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbprov2005.codfw.wmnet with OS bullseye [production]
16:14 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:14 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:07 <urandom> disabling read-repair (Cassandra) for restbase tables — T360548 [production]
16:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229 (T356166)', diff saved to https://phabricator.wikimedia.org/P58878 and previous config saved to /var/cache/conftool/dbconfig/20240321-160653-marostegui.json [production]
16:04 <sukhe@cumin1002> START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox [production]
15:50 <claime> cgoubert@deploy1002:~$ sudo chown imagecatalog:imagecatalog /srv/deployment/imagecatalog/catalog.sqlite [production]
15:33 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudbackup2004'] [production]
15:33 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS bullseye [production]
15:32 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dbprov2005.codfw.wmnet with OS bullseye [production]
15:27 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup2004'] [production]
15:27 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudbackup2004'] [production]
15:26 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup2004'] [production]
15:22 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1229 (T356166)', diff saved to https://phabricator.wikimedia.org/P58875 and previous config saved to /var/cache/conftool/dbconfig/20240321-152134-marostegui.json [production]
15:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
15:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
15:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:20 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:20 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58874 and previous config saved to /var/cache/conftool/dbconfig/20240321-152051-marostegui.json [production]
15:12 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P58873 and previous config saved to /var/cache/conftool/dbconfig/20240321-150544-marostegui.json [production]
15:04 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P58872 and previous config saved to /var/cache/conftool/dbconfig/20240321-145036-marostegui.json [production]
14:48 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 17 hosts with reason: Schema change T356166 [production]
14:47 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 17 hosts with reason: Schema change T356166 [production]
14:36 <moritzm> installing glibc security updates on bullseye [production]
14:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58871 and previous config saved to /var/cache/conftool/dbconfig/20240321-143528-marostegui.json [production]
14:21 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudbackup2004 to codfw - jhancock@cumin2002" [production]
14:18 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudbackup2004 to codfw - jhancock@cumin2002" [production]
14:16 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
14:05 <klausman@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
14:02 <moritzm> installing squid security updates [production]
13:51 <damilare> SmashPig upgraded from 47cd65d9 to 5d275ad6 [production]
13:50 <sukhe> upgrading pdns-rec to 4.8.7-1 on dns* and doh* hosts [production]