7101-7150 of 10000 results (97ms)
2024-03-21 ยง
15:20 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:20 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58874 and previous config saved to /var/cache/conftool/dbconfig/20240321-152051-marostegui.json [production]
15:12 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P58873 and previous config saved to /var/cache/conftool/dbconfig/20240321-150544-marostegui.json [production]
15:04 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P58872 and previous config saved to /var/cache/conftool/dbconfig/20240321-145036-marostegui.json [production]
14:48 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 17 hosts with reason: Schema change T356166 [production]
14:47 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 17 hosts with reason: Schema change T356166 [production]
14:36 <moritzm> installing glibc security updates on bullseye [production]
14:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58871 and previous config saved to /var/cache/conftool/dbconfig/20240321-143528-marostegui.json [production]
14:21 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudbackup2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudbackup2004 to codfw - jhancock@cumin2002" [production]
14:18 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudbackup2004 to codfw - jhancock@cumin2002" [production]
14:16 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
14:05 <klausman@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
14:02 <moritzm> installing squid security updates [production]
13:51 <damilare> SmashPig upgraded from 47cd65d9 to 5d275ad6 [production]
13:50 <sukhe> upgrading pdns-rec to 4.8.7-1 on dns* and doh* hosts [production]
13:16 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . [production]
13:14 <jiji@deploy1002> Finished scap: Check new deployment server (deploy1002) post switchover - March 2024 (duration: 35m 20s) [production]
13:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1222 (T356166)', diff saved to https://phabricator.wikimedia.org/P58869 and previous config saved to /var/cache/conftool/dbconfig/20240321-130213-marostegui.json [production]
13:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
13:02 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
13:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T356166)', diff saved to https://phabricator.wikimedia.org/P58868 and previous config saved to /var/cache/conftool/dbconfig/20240321-130151-marostegui.json [production]
12:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P58867 and previous config saved to /var/cache/conftool/dbconfig/20240321-124644-marostegui.json [production]
12:39 <jiji@deploy1002> Started scap: Check new deployment server (deploy1002) post switchover - March 2024 [production]
12:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage [production]
12:34 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage [production]
12:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P58866 and previous config saved to /var/cache/conftool/dbconfig/20240321-123135-marostegui.json [production]
12:19 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS bullseye [production]
12:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T356166)', diff saved to https://phabricator.wikimedia.org/P58865 and previous config saved to /var/cache/conftool/dbconfig/20240321-121628-marostegui.json [production]
12:15 <akosiaris@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad [production]
12:00 <effie> disable puppet on deployment servers [production]
11:46 <akosiaris@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad [production]
11:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P58860 and previous config saved to /var/cache/conftool/dbconfig/20240321-112108-marostegui.json [production]
11:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P58857 and previous config saved to /var/cache/conftool/dbconfig/20240321-110600-marostegui.json [production]
10:58 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1153.eqiad.wmnet with OS bookworm [production]
10:55 <akosiaris@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad [production]
10:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1188 (T356166)', diff saved to https://phabricator.wikimedia.org/P58856 and previous config saved to /var/cache/conftool/dbconfig/20240321-105052-marostegui.json [production]
10:43 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage [production]
10:41 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: host reimage [production]
10:34 <slyngshede@cumin1002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idp1003.wikimedia.org [production]
10:34 <slyngshede@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host idp1003.wikimedia.org with OS bookworm [production]
10:32 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
10:29 <akosiaris@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [production]
10:28 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1153.eqiad.wmnet with OS bookworm [production]
10:20 <slyngshede@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp1003.wikimedia.org with reason: host reimage [production]
10:17 <slyngshede@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on idp1003.wikimedia.org with reason: host reimage [production]
10:13 <akosiaris@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw [production]