1101-1150 of 10000 results (91ms)
2024-06-12 §
16:13 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
16:11 <jhathaway@deploy1002> Finished scap: (no justification provided) (duration: 03m 19s) [production]
16:10 <jhathaway@deploy1002> Started scap: (no justification provided) [production]
16:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P64726 and previous config saved to /var/cache/conftool/dbconfig/20240612-160856-marostegui.json [production]
16:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P64725 and previous config saved to /var/cache/conftool/dbconfig/20240612-160603-ladsgroup.json [production]
16:05 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:sessionstore [production]
16:00 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye [production]
15:55 <otto@deploy1002> Finished scap: Backport for [[gerrit:1041115|Remove EventLoggingLegacyConverter code - it has been moved to EventLogging (T353817)]] (duration: 12m 19s) [production]
15:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P64724 and previous config saved to /var/cache/conftool/dbconfig/20240612-155349-marostegui.json [production]
15:53 <kamila@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye [production]
15:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P64723 and previous config saved to /var/cache/conftool/dbconfig/20240612-155056-ladsgroup.json [production]
15:47 <otto@deploy1002> otto: Continuing with sync [production]
15:46 <otto@deploy1002> otto: Backport for [[gerrit:1041115|Remove EventLoggingLegacyConverter code - it has been moved to EventLogging (T353817)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:43 <otto@deploy1002> Started scap: Backport for [[gerrit:1041115|Remove EventLoggingLegacyConverter code - it has been moved to EventLogging (T353817)]] [production]
15:42 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye [production]
15:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367261)', diff saved to https://phabricator.wikimedia.org/P64722 and previous config saved to /var/cache/conftool/dbconfig/20240612-153842-marostegui.json [production]
15:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T352010)', diff saved to https://phabricator.wikimedia.org/P64721 and previous config saved to /var/cache/conftool/dbconfig/20240612-153549-ladsgroup.json [production]
15:35 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet [production]
15:34 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:34 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2001 to codfw - jhancock@cumin2002" [production]
15:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet [production]
15:33 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2001 to codfw - jhancock@cumin2002" [production]
15:31 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
15:28 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet [production]
15:28 <denisse@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=logstash,name=eqiad [production]
15:27 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync [production]
15:27 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync [production]
15:25 <volans> uploaded spicerack_8.6.0 to apt.wikimedia.org bullseye-wikimedia [production]
15:25 <kamila@cumin1002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wikikube-ctrl1003'] [production]
15:24 <kamila@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wikikube-ctrl1003'] [production]
15:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2177 (T367261)', diff saved to https://phabricator.wikimedia.org/P64720 and previous config saved to /var/cache/conftool/dbconfig/20240612-152403-marostegui.json [production]
15:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
15:23 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
15:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T367261)', diff saved to https://phabricator.wikimedia.org/P64719 and previous config saved to /var/cache/conftool/dbconfig/20240612-152351-marostegui.json [production]
15:23 <kamila@cumin1002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wikikube-ctrl1003'] [production]
15:12 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet [production]
15:12 <kamila@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wikikube-ctrl1003'] [production]
15:12 <kamila@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye [production]
15:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P64718 and previous config saved to /var/cache/conftool/dbconfig/20240612-150844-marostegui.json [production]
15:02 <cdanis> T364907 💙cdanis@apt1002.wikimedia.org ~ 🕚☕ sudo -i reprepro --keepunreferencedfiles includedeb bullseye-wikimedia ~/otelcol-contrib_0.102.0_linux_amd64.deb [production]
15:02 <brett> authdns-update run on dns1004 [production]
15:01 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye [production]
15:00 <brett> Depooling ulsfo in preparation for A:cp-text downtime/poweroff for nvme upgrades (T364891) [production]
15:00 <logmsgbot> lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for [[gerrit:1042283|Revert "Only register EntitySchema namespace when feature is enabled"]], [[gerrit:1042284|Revert "Allow loading EntitySchema on client (only) wikis"]] (duration: 12m 36s) [production]
14:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P64717 and previous config saved to /var/cache/conftool/dbconfig/20240612-145337-marostegui.json [production]
14:53 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
14:53 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
14:51 <logmsgbot> lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde: Continuing with sync [production]
14:50 <logmsgbot> lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde: Backport for [[gerrit:1042283|Revert "Only register EntitySchema namespace when feature is enabled"]], [[gerrit:1042284|Revert "Allow loading EntitySchema on client (only) wikis"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:50 <cgoubert@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad [production]