1401-1450 of 10000 results (39ms)
2023-10-31 §
06:43 <marostegui@deploy2002> marostegui: Backport for [[gerrit:969772|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:42 <marostegui@deploy2002> Started scap: Backport for [[gerrit:969772|Revert "ProductionServices.php: Promote pc2014 to pc1 master"]] [production]
06:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'Set db2140 with weight 0 T349820', diff saved to https://phabricator.wikimedia.org/P53068 and previous config saved to /var/cache/conftool/dbconfig/20231031-063647-arnaudb.json [production]
06:33 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
06:33 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 34 hosts with reason: Primary switchover s4 T349820 [production]
06:31 <marostegui@deploy2002> Finished scap: Backport for [[gerrit:970033|ProductionServices.php: Promote pc2014 to pc1 master]] (duration: 06m 50s) [production]
06:26 <marostegui@deploy2002> marostegui: Continuing with sync [production]
06:25 <marostegui@deploy2002> marostegui: Backport for [[gerrit:970033|ProductionServices.php: Promote pc2014 to pc1 master]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
06:24 <marostegui@deploy2002> Started scap: Backport for [[gerrit:970033|ProductionServices.php: Promote pc2014 to pc1 master]] [production]
06:16 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643) [admin]
05:20 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [admin]
03:55 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.1 (duration: 02m 14s) [production]
03:53 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.3 refs T348356 (duration: 50m 44s) [production]
03:02 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.3 refs T348356 [production]
02:27 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643) [admin]
02:27 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node (348643) [admin]
02:27 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643) [admin]
02:26 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (348643) [admin]
01:59 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (348643) [admin]
01:59 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (348643) [admin]
01:07 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [admin]
01:07 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [admin]
01:07 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [admin]
00:46 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1103.eqiad.wmnet with OS bullseye [production]
00:29 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1103.eqiad.wmnet with OS bullseye [production]
00:19 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1103.eqiad.wmnet with OS bullseye [production]
2023-10-30 §
23:56 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1103.eqiad.wmnet with OS bullseye [production]
23:56 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1103.eqiad.wmnet with OS bullseye [production]
23:50 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp1103.eqiad.wmnet with OS bullseye [production]
21:29 <wm-bot> <anticomposite> SULWatcher/manage.sh restart # SULWatcher disconnected [tools.stewardbots]
21:22 <sbassett> Deployed updated security mitigation for T348828 [production]
21:19 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for search-loader[2001-2002].codfw.wmnet,search-loader[1001-1002].eqiad.wmnet [production]
21:19 <bking@cumin1001> START - Cookbook sre.hosts.remove-downtime for search-loader[2001-2002].codfw.wmnet,search-loader[1001-1002].eqiad.wmnet [production]
20:58 <ejegg> re-enabled fundraising scheduled jobs after deployment [production]
20:45 <otto@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [production]
20:45 <otto@deploy2002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [production]
20:44 <otto@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [production]
20:44 <otto@deploy2002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [production]
20:43 <otto@deploy2002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply [production]
20:43 <otto@deploy2002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
20:41 <ejegg> fundraising civicrm upgraded from 2c79475e to 71d26d3b [production]
20:40 <ejegg> disable fundraising scheduled jobs for deployment [production]
20:29 <otto@deploy2002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply [production]
20:29 <otto@deploy2002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
20:29 <James_F> Ran `sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwgate-node16-docker/` on integration-castor05 too for T349986 [releng]
20:28 <otto@deploy2002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
20:27 <James_F> Ran jforrester@integration-castor05:~$ sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node16-rundoc-docker to clean out corrupted cache [releng]
20:21 <otto@deploy2002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
20:20 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3004.wikimedia.org with OS bookworm [production]
20:17 <dancy@deploy2002> Finished scap: Backport for [[gerrit:969353|namespaces:mediawiki: add Extensions/Skins as alias of Extension/Skin (+ tallk) (T349970)]] (duration: 10m 09s) [production]