5801-5850 of 10000 results (85ms)
2024-01-30 ยง
09:00 <brouberol@cumin1002> START - Cookbook sre.hosts.reimage for host an-tool1008.eqiad.wmnet with OS bullseye [production]
08:57 <Emperor> restart swift-object-replicator on ms-be1068 [production]
08:52 <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Switchover', diff saved to https://phabricator.wikimedia.org/P55890 and previous config saved to /var/cache/conftool/dbconfig/20240130-085159-root.json [production]
08:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P55889 and previous config saved to /var/cache/conftool/dbconfig/20240130-085055-root.json [production]
08:38 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P55888 and previous config saved to /var/cache/conftool/dbconfig/20240130-083829-root.json [production]
08:36 <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 50%: Switchover', diff saved to https://phabricator.wikimedia.org/P55887 and previous config saved to /var/cache/conftool/dbconfig/20240130-083654-root.json [production]
08:35 <marostegui@cumin1002> dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P55886 and previous config saved to /var/cache/conftool/dbconfig/20240130-083550-root.json [production]
08:29 <moritzm> upgrading python-pymysql on remaining DB hosts to 1.0.2-2~wmf11u1 T355531 [production]
08:28 <ladsgroup@deploy2002> Finished scap: Backport for [[gerrit:993824|Enable PageNotice extension on testwiki (T61245)]] (duration: 10m 24s) [production]
08:23 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P55885 and previous config saved to /var/cache/conftool/dbconfig/20240130-082324-root.json [production]
08:22 <ladsgroup@deploy2002> ladsgroup and tto: Continuing with sync [production]
08:21 <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Switchover', diff saved to https://phabricator.wikimedia.org/P55884 and previous config saved to /var/cache/conftool/dbconfig/20240130-082149-root.json [production]
08:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P55883 and previous config saved to /var/cache/conftool/dbconfig/20240130-082045-root.json [production]
08:19 <ladsgroup@deploy2002> ladsgroup and tto: Backport for [[gerrit:993824|Enable PageNotice extension on testwiki (T61245)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:18 <ladsgroup@deploy2002> Started scap: Backport for [[gerrit:993824|Enable PageNotice extension on testwiki (T61245)]] [production]
08:08 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P55882 and previous config saved to /var/cache/conftool/dbconfig/20240130-080819-root.json [production]
08:06 <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 10%: Switchover', diff saved to https://phabricator.wikimedia.org/P55881 and previous config saved to /var/cache/conftool/dbconfig/20240130-080644-root.json [production]
08:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P55880 and previous config saved to /var/cache/conftool/dbconfig/20240130-080540-root.json [production]
07:55 <ayounsi@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2034.codfw.wmnet to cluster codfw02 and group AB [production]
07:53 <ayounsi@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2034.codfw.wmnet to cluster codfw02 and group AB [production]
07:53 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P55879 and previous config saved to /var/cache/conftool/dbconfig/20240130-075314-root.json [production]
07:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P55878 and previous config saved to /var/cache/conftool/dbconfig/20240130-075035-root.json [production]
07:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2105 T356069', diff saved to https://phabricator.wikimedia.org/P55877 and previous config saved to /var/cache/conftool/dbconfig/20240130-074746-root.json [production]
07:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2127 to s3 primary and set section read-write T356069', diff saved to https://phabricator.wikimedia.org/P55876 and previous config saved to /var/cache/conftool/dbconfig/20240130-074656-marostegui.json [production]
07:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Set s3 codfw as read-only for maintenance - T356069', diff saved to https://phabricator.wikimedia.org/P55875 and previous config saved to /var/cache/conftool/dbconfig/20240130-074634-marostegui.json [production]
07:46 <marostegui> Starting s3 codfw failover from db2105 to db2127 - T356069 [production]
07:38 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P55874 and previous config saved to /var/cache/conftool/dbconfig/20240130-073807-root.json [production]
07:33 <root@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s3 T356069 [production]
07:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2127 with weight 0 T356069', diff saved to https://phabricator.wikimedia.org/P55873 and previous config saved to /var/cache/conftool/dbconfig/20240130-073257-marostegui.json [production]
07:32 <root@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 23 hosts with reason: Primary switchover s3 T356069 [production]
07:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P55872 and previous config saved to /var/cache/conftool/dbconfig/20240130-072734-root.json [production]
07:23 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P55871 and previous config saved to /var/cache/conftool/dbconfig/20240130-072302-root.json [production]
07:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P55870 and previous config saved to /var/cache/conftool/dbconfig/20240130-071612-root.json [production]
07:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P55869 and previous config saved to /var/cache/conftool/dbconfig/20240130-071229-root.json [production]
07:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2144 to x2 master T356060', diff saved to https://phabricator.wikimedia.org/P55868 and previous config saved to /var/cache/conftool/dbconfig/20240130-071202-root.json [production]
07:07 <marostegui@cumin1002> dbctl commit (dc=all): 'es2020 (re)pooling @ 1%: After switchover', diff saved to https://phabricator.wikimedia.org/P55867 and previous config saved to /var/cache/conftool/dbconfig/20240130-070757-root.json [production]
07:02 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:993775|Revert "db-production.php: Disable writes on es4"]] (duration: 07m 48s) [production]
07:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P55866 and previous config saved to /var/cache/conftool/dbconfig/20240130-070107-root.json [production]
07:00 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover x2 T356060 [production]
07:00 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover x2 T356060 [production]
06:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P55865 and previous config saved to /var/cache/conftool/dbconfig/20240130-065724-root.json [production]
06:55 <marostegui@deploy2002> marostegui: Continuing with sync [production]
06:55 <marostegui@deploy2002> marostegui: Backport for [[gerrit:993775|Revert "db-production.php: Disable writes on es4"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:54 <marostegui@deploy2002> Started scap: Backport for [[gerrit:993775|Revert "db-production.php: Disable writes on es4"]] [production]
06:48 <marostegui@deploy2002> backport Cancelled [production]
06:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P55864 and previous config saved to /var/cache/conftool/dbconfig/20240130-064602-root.json [production]
06:45 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2020 T356064', diff saved to https://phabricator.wikimedia.org/P55863 and previous config saved to /var/cache/conftool/dbconfig/20240130-064526-root.json [production]
06:45 <marostegui@cumin1002> dbctl commit (dc=all): 'Reduce es2021 weight T356064', diff saved to https://phabricator.wikimedia.org/P55862 and previous config saved to /var/cache/conftool/dbconfig/20240130-064512-root.json [production]
06:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P55861 and previous config saved to /var/cache/conftool/dbconfig/20240130-064219-root.json [production]
06:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es2021 to es4 primary T356064', diff saved to https://phabricator.wikimedia.org/P55860 and previous config saved to /var/cache/conftool/dbconfig/20240130-063625-root.json [production]