2251-2300 of 10000 results (44ms)
2024-05-14 §
00:19 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
00:19 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
00:19 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
00:18 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
00:15 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
2024-05-13 §
23:21 <wmbot~bd808@tools-bastion-12> Restarted gitlab job to pick up fix for T364719 [tools.wikibugs]
23:20 <wmbot~bd808@tools-bastion-12> Built new image from git hash efd174cd (T364719) [tools.wikibugs]
23:15 <wmbot~bd808@tools-bastion-12> Build new container based on MR!42 [tools.wikibugs-testing]
22:55 <bking@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=elastic110[5|7]\.eqiad\.wmnet [production]
22:43 <ryankemper@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T363975 eqiad cluster restart - ryankemper@cumin2002 - T363975 [production]
22:30 <zabe> zabe@mwmaint1002:~$ mwscript cleanupTitles.php itwikivoyage # T298315 [production]
22:27 <bking@cumin2002> conftool action : set/weight=10:pooled=no; selector: name=elastic110[5|7]\.eqiad\.wmnet [production]
21:47 <ryankemper@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T363975 eqiad cluster restart - ryankemper@cumin2002 - T363975 [production]
21:46 <ryankemper@cumin2002> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [production]
21:39 <ryankemper@cumin2002> START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [production]
21:39 <eileen> civicrm upgraded from 447e1472 to c854dd3a [production]
21:32 <ryankemper@cumin2002> END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:datahubsearch [production]
21:32 <ebernhardson@deploy1002> Finished scap: Backport for [[gerrit:1030983|Unbreak link buttons (T364062)]] (duration: 22m 00s) [production]
21:22 <ryankemper@cumin2002> START - Cookbook sre.opensearch.roll-restart-reboot rolling restart_daemons on A:datahubsearch [production]
21:22 <wmbot~bd808@tools-bastion-12> Restarted gerrit, gitlab, irc, and phorge tasks to pick up health-check fixes from MR!41 [tools.wikibugs]
21:20 <ebernhardson@deploy1002> jdlrobson and ebernhardson: Continuing with sync [production]
21:14 <wmbot~bd808@tools-bastion-12> Built new image from git hash f727d6c1d8 [tools.wikibugs]
21:12 <ebernhardson@deploy1002> jdlrobson and ebernhardson: Backport for [[gerrit:1030983|Unbreak link buttons (T364062)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:10 <ebernhardson@deploy1002> Started scap: Backport for [[gerrit:1030983|Unbreak link buttons (T364062)]] [production]
20:57 <ebernhardson@deploy1002> Finished scap: Backport for [[gerrit:1017152|IPInfo: Remove $wgIPInfoGeoIP2EnterprisePath (T361884)]] (duration: 17m 22s) [production]
20:45 <ebernhardson@deploy1002> ebernhardson and tchanders: Continuing with sync [production]
20:42 <ebernhardson@deploy1002> ebernhardson and tchanders: Backport for [[gerrit:1017152|IPInfo: Remove $wgIPInfoGeoIP2EnterprisePath (T361884)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:40 <ebernhardson@deploy1002> Started scap: Backport for [[gerrit:1017152|IPInfo: Remove $wgIPInfoGeoIP2EnterprisePath (T361884)]] [production]
20:38 <ebernhardson@deploy1002> Finished scap: Backport for [[gerrit:1014626|Remove old CampaignEvents DB config (prod) (T348281)]] (duration: 21m 14s) [production]
20:29 <eileen> civicrm upgraded from 4f55a7cf to 447e1472 [production]
20:25 <ebernhardson@deploy1002> ebernhardson and daimona: Continuing with sync [production]
20:24 <wmbot~bd808@tools-bastion-12> Restarted gitlab job to pick up MR owner reporting [tools.wikibugs]
20:23 <wmbot~bd808@tools-bastion-12> Built new image from git hash f7d9ef07. [tools.wikibugs]
20:19 <ebernhardson@deploy1002> ebernhardson and daimona: Backport for [[gerrit:1014626|Remove old CampaignEvents DB config (prod) (T348281)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:17 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/opentelemetry-collector: apply [production]
20:16 <ebernhardson@deploy1002> Started scap: Backport for [[gerrit:1014626|Remove old CampaignEvents DB config (prod) (T348281)]] [production]
20:10 <wmbot~bd808@tools-bastion-12> Build new container based on MR!40 [tools.wikibugs-testing]
20:00 <wmbot~bsadowski1@tools-bastion-13> Restarted StewardBot/SULWatcher because of a connection loss [tools.stewardbots]
19:57 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/opentelemetry-collector: apply [production]
19:47 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/opentelemetry-collector: apply [production]
19:27 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/opentelemetry-collector: apply [production]
19:26 <cdanis@deploy1002> helmfile [staging] DONE helmfile.d/services/opentelemetry-collector: apply [production]
19:26 <cdanis@deploy1002> helmfile [staging] START helmfile.d/services/opentelemetry-collector: apply [production]
18:49 <herron@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad [production]
18:49 <ryankemper@cumin2002> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
18:30 <ryankemper@cumin2002> START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
18:24 <herron@cumin1002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad [production]
18:24 <ryankemper@cumin2002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-jumbo-eqiad [production]
18:20 <swfrench@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply [production]
18:19 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 1c3d80a5e6 (l10n updates: eu, zh-hans) [tools.lexeme-forms]