101-150 of 10000 results (79ms)
2023-12-05 ยง
16:14 <samtar@deploy2002> Finished scap: Backport for [[gerrit:959327|.well-known: Add F-Droid signature to assetlinks.json (T346951)]] (duration: 07m 53s) [production]
16:11 <elukey@deploy2002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: sync [production]
16:09 <elukey@deploy2002> helmfile [codfw] START helmfile.d/services/recommendation-api: sync [production]
16:09 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: sync [production]
16:09 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196 (T348183)', diff saved to https://phabricator.wikimedia.org/P54185 and previous config saved to /var/cache/conftool/dbconfig/20231205-160920-arnaudb.json [production]
16:09 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/recommendation-api: sync [production]
16:08 <samtar@deploy2002> samtar: Continuing with sync [production]
16:08 <samtar@deploy2002> samtar: Backport for [[gerrit:959327|.well-known: Add F-Droid signature to assetlinks.json (T346951)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
16:07 <samtar@deploy2002> Started scap: Backport for [[gerrit:959327|.well-known: Add F-Droid signature to assetlinks.json (T346951)]] [production]
16:01 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host testhost2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
16:00 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:00 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding testhost2001 to codfw - jhancock@cumin2002" [production]
15:59 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding testhost2001 to codfw - jhancock@cumin2002" [production]
15:58 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1196 (T348183)', diff saved to https://phabricator.wikimedia.org/P54184 and previous config saved to /var/cache/conftool/dbconfig/20231205-155858-arnaudb.json [production]
15:58 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
15:58 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
15:58 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
15:58 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
15:58 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T348183)', diff saved to https://phabricator.wikimedia.org/P54183 and previous config saved to /var/cache/conftool/dbconfig/20231205-155814-arnaudb.json [production]
15:57 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
15:56 <hnowlan@deploy2002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
15:56 <hnowlan@deploy2002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
15:56 <hnowlan@deploy2002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
15:56 <hnowlan@deploy2002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
15:53 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cp4040.ulsfo.wmnet [production]
15:49 <claime> sudo confctl select "service=kubesvc,cluster=dse-k8s" set/pooled=inactive - T352639 [production]
15:45 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host cp4040.ulsfo.wmnet [production]
15:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P54182 and previous config saved to /var/cache/conftool/dbconfig/20231205-154308-arnaudb.json [production]
15:42 <moritzm> installing monitoring-plugins bugfix updates from Bookworm point release [production]
15:42 <claime> Manually restarting pybal on lvs1020 - T352639 [production]
15:39 <mvernon@cumin1001> START - Cookbook sre.hosts.reimage for host moss-be1002.eqiad.wmnet with OS bookworm [production]
15:31 <kamila@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1471.eqiad.wmnet with OS bullseye [production]
15:29 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
15:29 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['sessionstore2005'] [production]
15:29 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2005'] [production]
15:29 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:29 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['sessionstore2006'] [production]
15:28 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2006'] [production]
15:28 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host sessionstore2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:28 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P54181 and previous config saved to /var/cache/conftool/dbconfig/20231205-152801-arnaudb.json [production]
15:26 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host aqs2001.codfw.wmnet [production]
15:22 <claime> Manually restarting pybal on lvs1019 - T352639 [production]
15:21 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
15:20 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
15:18 <hnowlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
15:17 <hnowlan@deploy2002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
15:16 <claime> Manually restarting pybal on lvs1020 - T352639 [production]
15:15 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:15 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host aqs2001.codfw.wmnet [production]
15:15 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]