2301-2350 of 10000 results (93ms)
2023-12-12 §
10:04 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
10:04 <kharlan@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
10:04 <kharlan@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
09:57 <arnaudb@cumin1001> START - Cookbook sre.mysql.clone of db1128.eqiad.wmnet onto db1228.eqiad.wmnet [production]
09:53 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 clone from db1128 ', diff saved to https://phabricator.wikimedia.org/P54334 and previous config saved to /var/cache/conftool/dbconfig/20231212-095352-arnaudb.json [production]
09:52 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: provisionning db1228.eqiad.wmnet - T344036 [production]
09:51 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: provisionning db1228.eqiad.wmnet - T344036 [production]
09:51 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: provisionning db1228.eqiad.wmnet - T344036 [production]
09:51 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1128.eqiad.wmnet with reason: provisionning db1228.eqiad.wmnet - T344036 [production]
09:43 <moritzm> installing ca-certificates-java updates from Bookworm point release [production]
09:08 <arnaudb@cumin1001> START - Cookbook sre.mysql.clone of db1147.eqiad.wmnet onto db1247.eqiad.wmnet [production]
09:06 <arnaudb@cumin1001> dbctl commit (dc=all): 'Cloning db1147 in db1247 for T344036', diff saved to https://phabricator.wikimedia.org/P54333 and previous config saved to /var/cache/conftool/dbconfig/20231212-090652-arnaudb.json [production]
09:05 <aikochou@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
09:05 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: provisionning db1247.eqiad.wmnet - T344036 [production]
09:05 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: provisionning db1247.eqiad.wmnet - T344036 [production]
09:04 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: provisionning db1247.eqiad.wmnet - T344036 [production]
09:04 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: provisionning db1247.eqiad.wmnet - T344036 [production]
08:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: server BGP in netbox plugin - ayounsi@cumin1001 [production]
08:48 <ayounsi@cumin1001> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: server BGP in netbox plugin - ayounsi@cumin1001 [production]
08:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2135,2160].codfw.wmnet,db[1176,1217].eqiad.wmnet with reason: m5 ipoid maintenance [production]
08:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db[2135,2160].codfw.wmnet,db[1176,1217].eqiad.wmnet with reason: m5 ipoid maintenance [production]
07:52 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
07:52 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync [production]
07:50 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
07:49 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
07:17 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 4800 [production]
07:16 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 4800 [production]
06:46 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:982226|Revert "ProductionServices.php: Promote pc2014 as master of pc1"]] (duration: 09m 00s) [production]
06:38 <marostegui@deploy2002> marostegui: Continuing with sync [production]
06:38 <marostegui@deploy2002> marostegui: Backport for [[gerrit:982226|Revert "ProductionServices.php: Promote pc2014 as master of pc1"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:37 <marostegui@deploy2002> Started scap: Backport for [[gerrit:982226|Revert "ProductionServices.php: Promote pc2014 as master of pc1"]] [production]
06:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2011.codfw.wmnet with OS bookworm [production]
06:21 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2011.codfw.wmnet with reason: host reimage [production]
06:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc2011.codfw.wmnet with reason: host reimage [production]
06:00 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host pc2011.codfw.wmnet with OS bookworm [production]
05:59 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]] (duration: 08m 35s) [production]
05:52 <marostegui@deploy2002> marostegui: Continuing with sync [production]
05:52 <marostegui@deploy2002> marostegui: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
05:50 <marostegui@deploy2002> Started scap: Backport for [[gerrit:982206|ProductionServices.php: Promote pc2014 as master of pc1 (T351787)]] [production]
05:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787 [production]
05:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: pc1 master switch T351787 [production]
04:58 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.5 (duration: 02m 17s) [production]
04:55 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.9 refs T350085 (duration: 53m 03s) [production]
04:02 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.9 refs T350085 [production]
2023-12-11 §
22:39 <jdrewniak@deploy2002> Finished scap: Backport for [[gerrit:982162|[Vector] Deploy the Zebra CSS refactor under feature flag (T353008)]] (duration: 12m 14s) [production]
22:32 <jdrewniak@deploy2002> jdrewniak: Continuing with sync [production]
22:28 <jdrewniak@deploy2002> jdrewniak: Backport for [[gerrit:982162|[Vector] Deploy the Zebra CSS refactor under feature flag (T353008)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:26 <jdrewniak@deploy2002> Started scap: Backport for [[gerrit:982162|[Vector] Deploy the Zebra CSS refactor under feature flag (T353008)]] [production]
22:23 <ladsgroup@deploy2002> Finished scap: Backport for [[gerrit:981737|api: Add support for pagelinks migration in ApiQueryBacklinks::runSecondQuery (T351237)]] (duration: 10m 42s) [production]
22:15 <ladsgroup@deploy2002> jforrester and ladsgroup: Continuing with sync [production]