151-200 of 10000 results (89ms)
2023-10-10 §
13:08 <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:960545|growth: Enable section-image recommendations on 10 new wikis (T345940)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:07 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:960545|growth: Enable section-image recommendations on 10 new wikis (T345940)]] [production]
13:02 <fnegri@cumin1001> START - Cookbook sre.dns.netbox [production]
12:19 <jbond@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp1100.eqiad.wmnet'] [production]
12:18 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp1100.eqiad.wmnet'] [production]
12:02 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
12:01 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
11:40 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2138:3312 (T343198)', diff saved to https://phabricator.wikimedia.org/P52885 and previous config saved to /var/cache/conftool/dbconfig/20231010-114024-arnaudb.json [production]
11:40 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
11:40 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
11:40 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T343198)', diff saved to https://phabricator.wikimedia.org/P52884 and previous config saved to /var/cache/conftool/dbconfig/20231010-114002-arnaudb.json [production]
11:33 <volans> installed spicerack 7.4.1 on the cumin hosts [production]
11:33 <cgoubert@cumin1001> END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0) [production]
11:32 <cgoubert@cumin1001> START - Cookbook sre.mediawiki.restart-appservers [production]
11:30 <cgoubert@cumin1001> END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0) [production]
11:29 <cgoubert@cumin1001> START - Cookbook sre.mediawiki.restart-appservers [production]
11:24 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P52883 and previous config saved to /var/cache/conftool/dbconfig/20231010-112456-arnaudb.json [production]
11:09 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P52882 and previous config saved to /var/cache/conftool/dbconfig/20231010-110950-arnaudb.json [production]
10:54 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T343198)', diff saved to https://phabricator.wikimedia.org/P52880 and previous config saved to /var/cache/conftool/dbconfig/20231010-105443-arnaudb.json [production]
10:52 <jbond@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp1100.eqiad.wmnet'] [production]
10:52 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp1100.eqiad.wmnet'] [production]
09:56 <ladsgroup@deploy2002> Finished scap: Backport for [[gerrit:964870|Set pagelinks migration stage of cebwiki to write both (T345732)]] (duration: 09m 10s) [production]
09:50 <ladsgroup@deploy2002> ladsgroup: Continuing with sync [production]
09:48 <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:964870|Set pagelinks migration stage of cebwiki to write both (T345732)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:47 <ladsgroup@deploy2002> Started scap: Backport for [[gerrit:964870|Set pagelinks migration stage of cebwiki to write both (T345732)]] [production]
09:33 <volans> uploaded spicerack_7.4.1 to apt.wikimedia.org bullseye-wikimedia [production]
08:35 <hashar@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.30 refs T347081 [production]
08:24 <taavi> wikitech-static: cleanup image archive directory: T348503 [production]
08:09 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2126 (T343198)', diff saved to https://phabricator.wikimedia.org/P52879 and previous config saved to /var/cache/conftool/dbconfig/20231010-080924-arnaudb.json [production]
08:09 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
08:09 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
08:09 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: Maintenance [production]
08:08 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: Maintenance [production]
08:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2125 (T343198)', diff saved to https://phabricator.wikimedia.org/P52878 and previous config saved to /var/cache/conftool/dbconfig/20231010-080847-arnaudb.json [production]
08:00 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
07:53 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P52877 and previous config saved to /var/cache/conftool/dbconfig/20231010-075340-arnaudb.json [production]
07:38 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P52876 and previous config saved to /var/cache/conftool/dbconfig/20231010-073834-arnaudb.json [production]
07:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2125 (T343198)', diff saved to https://phabricator.wikimedia.org/P52875 and previous config saved to /var/cache/conftool/dbconfig/20231010-072327-arnaudb.json [production]
07:19 <kostajh> UTC morning deploys done [production]
07:18 <kharlan@deploy2002> Finished scap: Backport for [[gerrit:964575|ReportIncident: Set developer mode to false]] (duration: 10m 17s) [production]
07:12 <kharlan@deploy2002> kharlan: Continuing with sync [production]
07:09 <kharlan@deploy2002> kharlan: Backport for [[gerrit:964575|ReportIncident: Set developer mode to false]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:08 <kharlan@deploy2002> Started scap: Backport for [[gerrit:964575|ReportIncident: Set developer mode to false]] [production]
06:42 <moritzm> installing qemu security updates on bookworm [production]
03:54 <mwpresync@deploy2002> Pruned MediaWiki: 1.41.0-wmf.28 (duration: 02m 08s) [production]
03:52 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.41.0-wmf.30 refs T347081 (duration: 49m 56s) [production]
03:02 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.41.0-wmf.30 refs T347081 [production]
2023-10-09 §
22:54 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2125 (T343198)', diff saved to https://phabricator.wikimedia.org/P52873 and previous config saved to /var/cache/conftool/dbconfig/20231009-225429-arnaudb.json [production]
22:54 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2125.codfw.wmnet with reason: Maintenance [production]
22:54 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2125.codfw.wmnet with reason: Maintenance [production]