2851-2900 of 10000 results (94ms)
2023-11-14 ยง
10:22 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
10:21 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
10:21 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T348183)', diff saved to https://phabricator.wikimedia.org/P53386 and previous config saved to /var/cache/conftool/dbconfig/20231114-102145-arnaudb.json [production]
10:15 <klausman@cumin1001> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: etcd::v3::ml_etcd::staging [production]
10:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'Set db1160 with weight 0 T351184', diff saved to https://phabricator.wikimedia.org/P53385 and previous config saved to /var/cache/conftool/dbconfig/20231114-100843-arnaudb.json [production]
10:08 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s4 T351184 [production]
10:07 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s4 T351184 [production]
10:06 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P53384 and previous config saved to /var/cache/conftool/dbconfig/20231114-100638-arnaudb.json [production]
10:05 <klausman@cumin1001> START - Cookbook sre.puppet.migrate-role for role: etcd::v3::ml_etcd::staging [production]
10:03 <jnuche@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.5 refs T350081 [production]
09:54 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
09:53 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: provisionning db1238.eqiad.wmnet - T344036 [production]
09:53 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: provisionning db1238.eqiad.wmnet - T344036 [production]
09:53 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: provisionning db1238.eqiad.wmnet - T344036 [production]
09:53 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: provisionning db1238.eqiad.wmnet - T344036 [production]
09:53 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:974127|Revert "ProductionServices.php: Promote pc2014 to pc3 master"]] (duration: 07m 26s) [production]
09:52 <jayme@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
09:51 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P53383 and previous config saved to /var/cache/conftool/dbconfig/20231114-095132-arnaudb.json [production]
09:47 <marostegui@deploy2002> marostegui: Continuing with sync [production]
09:47 <marostegui@deploy2002> marostegui: Backport for [[gerrit:974127|Revert "ProductionServices.php: Promote pc2014 to pc3 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:45 <marostegui@deploy2002> Started scap: Backport for [[gerrit:974127|Revert "ProductionServices.php: Promote pc2014 to pc3 master"]] [production]
09:45 <klausman@cumin1001> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host ml-staging-etcd2003.codfw.wmnet [production]
09:39 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:974113|ProductionServices.php: Promote pc2014 to pc3 master]] (duration: 07m 11s) [production]
09:38 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
09:38 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
09:36 <jayme> reimaging kubestage2002 to verify with puppet7 [production]
09:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T348183)', diff saved to https://phabricator.wikimedia.org/P53380 and previous config saved to /var/cache/conftool/dbconfig/20231114-093625-arnaudb.json [production]
09:34 <jayme@cumin2002> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bullseye [production]
09:34 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1158 (T348183)', diff saved to https://phabricator.wikimedia.org/P53379 and previous config saved to /var/cache/conftool/dbconfig/20231114-093353-arnaudb.json [production]
09:34 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
09:34 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
09:34 <marostegui@deploy2002> marostegui: Continuing with sync [production]
09:34 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
09:33 <marostegui@deploy2002> marostegui: Backport for [[gerrit:974113|ProductionServices.php: Promote pc2014 to pc3 master]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:33 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
09:32 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
09:32 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
09:32 <marostegui@deploy2002> Started scap: Backport for [[gerrit:974113|ProductionServices.php: Promote pc2014 to pc3 master]] [production]
09:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2013-2014].codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Switch [production]
09:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2013-2014].codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Switch [production]
09:30 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
09:30 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
09:28 <klausman@cumin1001> START - Cookbook sre.puppet.migrate-host for host ml-staging-etcd2003.codfw.wmnet [production]
09:26 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:973805|Revert "ProductionServices.php: Promote pc1014 to pc3 master"]] (duration: 07m 02s) [production]
09:26 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
09:25 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: sync [production]
09:21 <marostegui@deploy2002> marostegui: Continuing with sync [production]
09:20 <marostegui@deploy2002> marostegui: Backport for [[gerrit:973805|Revert "ProductionServices.php: Promote pc1014 to pc3 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:19 <marostegui@deploy2002> Started scap: Backport for [[gerrit:973805|Revert "ProductionServices.php: Promote pc1014 to pc3 master"]] [production]
09:12 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:974106|ProductionServices.php: Promote pc1014 to pc3 master]] (duration: 07m 24s) [production]