7901-7950 of 10000 results (103ms)
2023-10-30 §
12:28 <marostegui@cumin1001> dbctl commit (dc=all): 'New host', diff saved to https://phabricator.wikimedia.org/P53065 and previous config saved to /var/cache/conftool/dbconfig/20231030-122855-marostegui.json [production]
12:26 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage [production]
12:24 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1217.eqiad.wmnet with reason: host reimage [production]
12:11 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1217.eqiad.wmnet with OS bookworm [production]
11:52 <arnaudb@cumin1001> START - Cookbook sre.mysql.clone of db1130.eqiad.wmnet onto db1230.eqiad.wmnet [production]
11:34 <arnaudb@cumin1001> dbctl commit (dc=all): 'Adding db1230 depooled, depooling db1130', diff saved to https://phabricator.wikimedia.org/P53064 and previous config saved to /var/cache/conftool/dbconfig/20231030-113401-arnaudb.json [production]
11:28 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: provisionning db1230.eqiad.wmnet - T344036 [production]
11:28 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: provisionning db1230.eqiad.wmnet - T344036 [production]
11:28 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: provisionning db1230.eqiad.wmnet - T344036 [production]
11:28 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: provisionning db1230.eqiad.wmnet - T344036 [production]
09:42 <jnuche@deploy2002> Finished deploy [releng/jenkins-deploy@af33784] (releasing): (no justification provided) (duration: 00m 40s) [production]
09:42 <jnuche@deploy2002> Started deploy [releng/jenkins-deploy@af33784] (releasing): (no justification provided) [production]
08:29 <vgutierrez> switched to digicert-2023 in esams, eqsin and drmrs - T341119 [production]
08:17 <wmde-fisch@deploy2002> Finished scap: Backport for [[gerrit:966520|Cleanup Kartographer Nearby flags (T332785)]] (duration: 07m 35s) [production]
08:12 <wmde-fisch@deploy2002> wmde-fisch: Continuing with sync [production]
08:11 <wmde-fisch@deploy2002> wmde-fisch: Backport for [[gerrit:966520|Cleanup Kartographer Nearby flags (T332785)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:10 <wmde-fisch@deploy2002> Started scap: Backport for [[gerrit:966520|Cleanup Kartographer Nearby flags (T332785)]] [production]
08:10 <vgutierrez> triggering a puppet run on cp hosts in esams, eqsin and drmrs to switch to the new unified digicert certificates - T341119 [production]
08:06 <vgutierrez> repool cp5025 - T341119 [production]
08:06 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:969361|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] (duration: 06m 41s) [production]
08:01 <marostegui@deploy2002> marostegui: Continuing with sync [production]
08:00 <marostegui@deploy2002> marostegui: Backport for [[gerrit:969361|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:59 <marostegui@deploy2002> Started scap: Backport for [[gerrit:969361|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] [production]
07:52 <vgutierrez> depool cp5025 to perform some digicert-2023 related sanity checks - T341119 [production]
07:49 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:969667|ProductionServices.php: Promote pc1014 to pc1 master]] (duration: 06m 36s) [production]
07:48 <marostegui@deploy2002> marostegui: Continuing with sync [production]
07:44 <marostegui@deploy2002> marostegui: Backport for [[gerrit:969667|ProductionServices.php: Promote pc1014 to pc1 master]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:43 <marostegui@deploy2002> Started scap: Backport for [[gerrit:969667|ProductionServices.php: Promote pc1014 to pc1 master]] [production]
07:35 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-airflow1007.eqiad.wmnet with reason: Downtime as we setup the new WMDE Airflow instance [production]
07:34 <stevemunene@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on an-airflow1007.eqiad.wmnet with reason: Downtime as we setup the new WMDE Airflow instance [production]
07:29 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:969360|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] (duration: 06m 33s) [production]
07:24 <marostegui@deploy2002> marostegui: Continuing with sync [production]
07:24 <marostegui@deploy2002> marostegui: Backport for [[gerrit:969360|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:22 <marostegui@deploy2002> Started scap: Backport for [[gerrit:969360|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] [production]
07:22 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:969533|ProductionServices.php: Promote pc1014 to pc1 master]] (duration: 14m 04s) [production]
07:18 <elukey> arm keyholder on acmechief2002 and deploy1002 [production]
07:16 <marostegui@deploy2002> marostegui: Continuing with sync [production]
07:16 <marostegui@deploy2002> marostegui: Backport for [[gerrit:969533|ProductionServices.php: Promote pc1014 to pc1 master]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:08 <marostegui@deploy2002> Started scap: Backport for [[gerrit:969533|ProductionServices.php: Promote pc1014 to pc1 master]] [production]
2023-10-28 §
21:25 <fabfur> re-pooled cp1089 and cp3069 [production]
21:05 <fabfur> depooled cp1089 and cp3069 to restart varnish|haproxy and let purged process incoming messages [production]
20:20 <fabfur> restarted purged on cp1089, cp6005, cp3069 [production]
19:46 <fabfur> restarted purged on cp1078 [production]
2023-10-27 §
22:47 <rzl> reprepro -C main include bullseye-wikimedia k8s-controller-sidecars_1.0.2-1_source.changes [production]
22:05 <ejegg> fundraising civicrm upgraded from 74781efd to 2c79475e [production]
15:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2004.codfw.wmnet with OS bullseye [production]
15:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
15:21 <herron> power cycled titan1001 [production]
14:59 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1001" [production]
14:42 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2004.codfw.wmnet with reason: host reimage [production]