1051-1100 of 10000 results (104ms)
2024-06-05 ยง
13:48 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
13:48 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host ping1004.eqiad.wmnet [production]
13:47 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0) [production]
13:47 <jmm@cumin2002> START - Cookbook sre.ganeti.resource-report [production]
13:46 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host prometheus3003.esams.wmnet [production]
13:46 <elukey> factory reset for sretest1001 to test the new provision cookbook - T365372 [production]
13:46 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host prometheus4002.ulsfo.wmnet [production]
13:46 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2073.codfw.wmnet [production]
13:46 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host prometheus5002.eqsin.wmnet [production]
13:46 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1074.eqiad.wmnet [production]
13:45 <inflatador> bking@an-db1001 install acl pkg T363001 [production]
13:43 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1073.eqiad.wmnet [production]
13:43 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet [production]
13:43 <filippo@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7001.magru.wmnet [production]
13:40 <daniel@deploy1002> daniel: Continuing with sync [production]
13:39 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet [production]
13:37 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet [production]
13:37 <filippo@cumin1002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host graphite1005.eqiad.wmnet [production]
13:37 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host prometheus7001.magru.wmnet [production]
13:37 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2072.codfw.wmnet [production]
13:35 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1073.eqiad.wmnet [production]
13:35 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1072.eqiad.wmnet [production]
13:34 <daniel@deploy1002> daniel: Backport for [[gerrit:1038688|Set LinterParseOnDerivedDataUpdate to false (T361013)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:34 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
13:30 <daniel@deploy1002> Started scap: Backport for [[gerrit:1038688|Set LinterParseOnDerivedDataUpdate to false (T361013)]] [production]
13:29 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2072.codfw.wmnet [production]
13:28 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2071.codfw.wmnet [production]
13:27 <elukey> systemctl reset-failed prometheus-redis-exporter@6380.service redis-instance-tcp_6380.service on netbox[12]002 + apt-get purge of redis-server and prometheus-redis-exporter packages to clean up stale configs (no local redis is used) [production]
13:27 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1072.eqiad.wmnet [production]
13:26 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1071.eqiad.wmnet [production]
13:25 <dreamyjazz@deploy1002> Finished scap: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] (duration: 11m 39s) [production]
13:25 <filippo@cumin1002> START - Cookbook sre.hosts.reboot-single for host graphite1005.eqiad.wmnet [production]
13:21 <fabfur> enable magru DC after applying IPIP encapsulation patches (T366466) [production]
13:20 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2071.codfw.wmnet [production]
13:19 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2070.codfw.wmnet [production]
13:17 <dreamyjazz@deploy1002> dreamyjazz: Continuing with sync [production]
13:17 <dreamyjazz@deploy1002> dreamyjazz: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2179 (T364299)', diff saved to https://phabricator.wikimedia.org/P64104 and previous config saved to /var/cache/conftool/dbconfig/20240605-131647-marostegui.json [production]
13:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:16 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T364299)', diff saved to https://phabricator.wikimedia.org/P64103 and previous config saved to /var/cache/conftool/dbconfig/20240605-131623-marostegui.json [production]
13:14 <dreamyjazz@deploy1002> Started scap: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] [production]
13:13 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2070.codfw.wmnet [production]
13:13 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1071.eqiad.wmnet [production]
13:13 <dreamyjazz@deploy1002> Finished scap: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] (duration: 19m 13s) [production]
13:10 <elukey@cumin1002> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker [production]
13:06 <fabfur> restarting pybal on lvs7001/lvs7003 to appy IPIP conf (T366466) [production]
13:04 <dreamyjazz@deploy1002> dreamyjazz: Continuing with sync [production]
13:03 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1070.eqiad.wmnet [production]
13:02 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2069.codfw.wmnet [production]