51-100 of 10000 results (112ms)
2026-07-13 §
07:54 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org [production]
07:53 <marostegui@cumin1003> conftool action : set/weight=50; selector: name=clouddb1028.eqiad.wmnet,service=s6 [production]
07:53 <marostegui@cumin1003> conftool action : set/weight=50; selector: name=clouddb1028.eqiad.wmnet,service=s4 [production]
07:52 <Msz2001> UTC morning backport+config window done [production]
07:50 <mvernon@cumin2003> START - Cookbook sre.hosts.reimage for host thanos-be2005.codfw.wmnet with OS trixie [production]
07:50 <mszwarc@deploy2003> Finished scap sync-world: Backport for [[gerrit:1309302|Enable campaignEvents on cowikimedia (T431765)]], [[gerrit:1307836|extwiki: Rename wgSitename to Güiquipedia and remove obsolete namespace alias (T431334)]], [[gerrit:1309901|cowikimedia: Update localtimezone to America/Bogota (T431965)]], [[gerrit:1308652|minwikiquote: set sitename, timezone and projectnamespace & add logo, wordmark (T429943 [production]
07:49 <filippo@cumin1003> START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet [production]
07:46 <marostegui@cumin1003> conftool action : set/weight=10; selector: name=clouddb1028.eqiad.wmnet,service=s4 [production]
07:46 <marostegui@cumin1003> conftool action : set/weight=100; selector: name=clouddb1028.eqiad.wmnet,service=s4 [production]
07:45 <marostegui@cumin1003> conftool action : set/pooled=yes; selector: name=clouddb1028.eqiad.wmnet,service=s6 [production]
07:45 <marostegui@cumin1003> conftool action : set/pooled=yes; selector: name=clouddb1028.eqiad.wmnet,service=s4 [production]
07:44 <marostegui@cumin1003> conftool action : set/weight=10; selector: name=clouddb1028.eqiad.wmnet,service=s6 [production]
07:44 <marostegui@cumin1003> conftool action : set/weight=10; selector: name=clouddb1028.eqiad.wmnet,service=s4 [production]
07:43 <mszwarc@deploy2003> mszwarc, danielyepezgarces, anzx: Continuing with deployment [production]
07:39 <mszwarc@deploy2003> mszwarc, danielyepezgarces, anzx: Backport for [[gerrit:1309302|Enable campaignEvents on cowikimedia (T431765)]], [[gerrit:1307836|extwiki: Rename wgSitename to Güiquipedia and remove obsolete namespace alias (T431334)]], [[gerrit:1309901|cowikimedia: Update localtimezone to America/Bogota (T431965)]], [[gerrit:1308652|minwikiquote: set sitename, timezone and projectnamespace & add logo, wordmark [production]
07:39 <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool es1039: Repooling after testing [production]
07:36 <mszwarc@deploy2003> Started scap sync-world: Backport for [[gerrit:1309302|Enable campaignEvents on cowikimedia (T431765)]], [[gerrit:1307836|extwiki: Rename wgSitename to Güiquipedia and remove obsolete namespace alias (T431334)]], [[gerrit:1309901|cowikimedia: Update localtimezone to America/Bogota (T431965)]], [[gerrit:1308652|minwikiquote: set sitename, timezone and projectnamespace & add logo, wordmark (T429943) [production]
07:35 <mszwarc@deploy2003> Finished scap sync-world: Backport for [[gerrit:1309559|plwiki: Switch back to normal tagline (T430512)]] (duration: 30m 03s) [production]
07:25 <jelto@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org [production]
07:22 <mszwarc@deploy2003> mszwarc: Continuing with deployment [production]
07:21 <mszwarc@deploy2003> mszwarc: Backport for [[gerrit:1309559|plwiki: Switch back to normal tagline (T430512)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
07:19 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org [production]
07:15 <jelto@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet [production]
07:11 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet [production]
07:08 <jelto@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org [production]
07:05 <mszwarc@deploy2003> Started scap sync-world: Backport for [[gerrit:1309559|plwiki: Switch back to normal tagline (T430512)]] [production]
07:02 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org [production]
07:02 <jelto@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org [production]
06:55 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org [production]
06:55 <jelto@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org [production]
06:49 <jelto@cumin1003> START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org [production]
06:34 <marostegui> Drop m5 ipoid database T431007 [production]
06:29 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: reboot [production]
06:24 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1028.eqiad.wmnet with reason: reboot [production]
06:21 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1025.eqiad.wmnet with reason: reboot [production]
06:17 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: reboot [production]
06:03 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbproxy[2005-2008].codfw.wmnet with reason: reboot [production]
05:37 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1217,1228].eqiad.wmnet with reason: cloning [production]
05:10 <marostegui> Drop users_to_rename table T431842 [production]
02:06 <mwpresync@deploy2003> Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) [production]
02:00 <mwpresync@deploy2003> Started scap build-images: Publishing wmf/next image [production]
2026-07-12 §
16:01 <marostegui@cumin1003> dbctl commit (dc=all): 'Depool db2209 T431950', diff saved to https://phabricator.wikimedia.org/P94792 and previous config saved to /var/cache/conftool/dbconfig/20260712-160124-marostegui.json [production]
15:58 <marostegui@cumin1003> dbctl commit (dc=all): 'Promote db2205 to s3 primary T431950', diff saved to https://phabricator.wikimedia.org/P94791 and previous config saved to /var/cache/conftool/dbconfig/20260712-155853-marostegui.json [production]
15:58 <marostegui> Starting s3 codfw emergency failover from db2209 to db2205 - T431950 [production]
15:51 <marostegui@cumin1003> dbctl commit (dc=all): 'Set db2205 with weight 0 T431950', diff saved to https://phabricator.wikimedia.org/P94790 and previous config saved to /var/cache/conftool/dbconfig/20260712-155135-marostegui.json [production]
15:51 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T431950 [production]
02:01 <mwpresync@deploy2003> Finished scap build-images: Publishing wmf/next image (duration: 01m 17s) [production]
02:00 <mwpresync@deploy2003> Started scap build-images: Publishing wmf/next image [production]
2026-07-11 §
02:06 <mwpresync@deploy2003> Finished scap build-images: Publishing wmf/next image (duration: 06m 26s) [production]
02:00 <mwpresync@deploy2003> Started scap build-images: Publishing wmf/next image [production]