351-400 of 10000 results (30ms)
2025-03-20 §
06:59 <marostegui@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P74272 and previous config saved to /var/cache/conftool/dbconfig/20250320-065942-root.json [production]
06:51 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
06:50 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2165.codfw.wmnet [production]
06:44 <marostegui@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P74271 and previous config saved to /var/cache/conftool/dbconfig/20250320-064437-root.json [production]
06:43 <marostegui@cumin1002> START - Cookbook sre.mysql.upgrade for db2165.codfw.wmnet [production]
06:41 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2165 T389367', diff saved to https://phabricator.wikimedia.org/P74270 and previous config saved to /var/cache/conftool/dbconfig/20250320-064131-marostegui.json [production]
06:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2161 to s8 primary T389367', diff saved to https://phabricator.wikimedia.org/P74269 and previous config saved to /var/cache/conftool/dbconfig/20250320-064012-marostegui.json [production]
06:39 <marostegui> Starting s8 codfw failover from db2165 to db2161 - T389367 [production]
06:37 <marostegui@deploy2002> Finished scap sync-world: Backport for [[gerrit:1129611|Revert "db-production.php: Disable writes on es7"]] (duration: 12m 48s) [production]
06:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2161 with weight 0 T389367', diff saved to https://phabricator.wikimedia.org/P74268 and previous config saved to /var/cache/conftool/dbconfig/20250320-063509-marostegui.json [production]
06:34 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T389367 [production]
06:31 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s1 [production]
06:31 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s1 [production]
06:30 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s2 [production]
06:30 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s2 [production]
06:30 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s3 [production]
06:30 <marostegui@deploy2002> marostegui: Continuing with sync [production]
06:29 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s3 [production]
06:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P74267 and previous config saved to /var/cache/conftool/dbconfig/20250320-062931-root.json [production]
06:29 <marostegui@deploy2002> marostegui: Backport for [[gerrit:1129611|Revert "db-production.php: Disable writes on es7"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s4 [production]
06:28 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s4 [production]
06:28 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s5 [production]
06:26 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s5 [production]
06:26 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s6 [production]
06:26 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s6 [production]
06:25 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s7 [production]
06:25 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s7 [production]
06:24 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section s8 [production]
06:24 <marostegui@deploy2002> Started scap sync-world: Backport for [[gerrit:1129611|Revert "db-production.php: Disable writes on es7"]] [production]
06:24 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section s8 [production]
06:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
06:23 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section x1 [production]
06:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section es6 [production]
06:23 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section es6 [production]
06:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section es7 [production]
06:21 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section es7 [production]
06:20 <marostegui@deploy2002> Finished scap sync-world: Backport for [[gerrit:1129609|db-production.php: Disable writes on es7 (T388627)]] (duration: 11m 07s) [production]
06:14 <marostegui@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P74266 and previous config saved to /var/cache/conftool/dbconfig/20250320-061426-root.json [production]
06:13 <marostegui@deploy2002> marostegui: Continuing with sync [production]
06:13 <marostegui@deploy2002> marostegui: Backport for [[gerrit:1129609|db-production.php: Disable writes on es7 (T388627)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:09 <marostegui@deploy2002> Started scap sync-world: Backport for [[gerrit:1129609|db-production.php: Disable writes on es7 (T388627)]] [production]
02:09 <ejegg> disabled dlocal at payments-wiki [production]
2025-03-19 §
22:23 <bd808> Added qrank.wmcloud.org after usurping both the proxy and the domain from the qrank Cloud VPS project. https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/EUIWWEG354NMU57QEBQVA5AIH7ORAF55/ [redirects]
22:14 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.21 refs T386216 [production]
21:26 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2249 [production]
21:21 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2249 [production]
21:19 <eevans@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:18 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:18 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding wikikube-worker2249 to codfw - jhancock@cumin2002" [production]