4601-4650 of 10000 results (89ms)
2023-05-04 §
04:36 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reboot - ryankemper@cumin1001 - T335835 [production]
04:30 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 50 hosts with reason: Rolling reboot of eqiad for T335835 [production]
04:29 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 50 hosts with reason: Rolling reboot of eqiad for T335835 [production]
02:42 <eileen> config revision changed from 5ac52d82 to 7ac11236 reduce batch size, avoid failmail [production]
02:35 <eileen> config revision changed from 121a864a to 5ac52d82 [production]
02:33 <eileen> civicrm upgraded from b97aaa08 to 05523a9d [production]
01:29 <eileen> config revision changed from 26147e89 to 121a864a - disabling populate as it keeps rolling back so prob another overlong row [production]
2023-05-03 §
23:55 <eileen> config revision changed from 2995f558 to 26147e89 [production]
23:15 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot - bking@cumin1001 - T335835 [production]
23:10 <tzatziki> removing 1 file for legal compliance [production]
23:01 <eileen> config revision changed from 69f60bb9 to 2995f558 [production]
22:42 <zabe@deploy1002> Finished scap: Backport for [[gerrit:914903|Start writing to af_actor/afh_actor in group1 wikis (T334295)]] (duration: 07m 13s) [production]
22:37 <zabe@deploy1002> zabe: Backport for [[gerrit:914903|Start writing to af_actor/afh_actor in group1 wikis (T334295)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
22:35 <zabe@deploy1002> Started scap: Backport for [[gerrit:914903|Start writing to af_actor/afh_actor in group1 wikis (T334295)]] [production]
22:34 <tzatziki> removing 12 files for legal compliance [production]
22:19 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs1010.eqiad.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
22:11 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs1010.eqiad.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
22:08 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs2001.codfw.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
22:04 <eileen> civicrm upgraded from c6149ad2 to b97aaa08 [production]
22:00 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs2001.codfw.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
21:55 <brett> Disable puppet on lvs4008 for new pybal deployment (just in case immediate config rollback is required) - T263797 [production]
21:43 <milimetric@deploy1002> Finished deploy [analytics/refinery@c53c095] (thin): Deploy THIN [analytics/refinery@c53c095] (duration: 00m 06s) [production]
21:43 <milimetric@deploy1002> Started deploy [analytics/refinery@c53c095] (thin): Deploy THIN [analytics/refinery@c53c095] [production]
21:31 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase10[17-33].eqiad.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
21:31 <brett> Uploaded pybal_1.15.11 to apt1001 via reprepro [production]
21:31 <milimetric@deploy1002> Finished deploy [analytics/refinery@c53c095]: Refinery deploy [analytics/refinery@c53c095] (duration: 08m 22s) [production]
21:22 <milimetric@deploy1002> Started deploy [analytics/refinery@c53c095]: Refinery deploy [analytics/refinery@c53c095] [production]
21:11 <brett> Upgrading pybal to 1.15.11 on lvs4010 [production]
20:54 <cjming> end of UTC late backport window [production]
20:53 <cjming@deploy1002> Finished scap: Backport for [[gerrit:914301|Router handling code should be centralized into mmv.bootstrap (T236591)]] (duration: 10m 08s) [production]
20:44 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:914301|Router handling code should be centralized into mmv.bootstrap (T236591)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
20:43 <cjming@deploy1002> Started scap: Backport for [[gerrit:914301|Router handling code should be centralized into mmv.bootstrap (T236591)]] [production]
20:42 <cjming@deploy1002> Finished scap: Backport for [[gerrit:913241|Explicitly enable MFCustomSiteModules (T270603)]] (duration: 10m 23s) [production]
20:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T335838)', diff saved to https://phabricator.wikimedia.org/P47480 and previous config saved to /var/cache/conftool/dbconfig/20230503-203424-ladsgroup.json [production]
20:33 <cjming@deploy1002> jdlrobson and cjming: Backport for [[gerrit:913241|Explicitly enable MFCustomSiteModules (T270603)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
20:32 <cjming@deploy1002> Started scap: Backport for [[gerrit:913241|Explicitly enable MFCustomSiteModules (T270603)]] [production]
20:30 <cjming@deploy1002> Finished scap: Backport for [[gerrit:914863|Enable graphs on test wikipedia and mediawiki.org (T334940)]] (duration: 08m 19s) [production]
20:23 <cjming@deploy1002> cjming and jdlrobson: Backport for [[gerrit:914863|Enable graphs on test wikipedia and mediawiki.org (T334940)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
20:21 <cjming@deploy1002> Started scap: Backport for [[gerrit:914863|Enable graphs on test wikipedia and mediawiki.org (T334940)]] [production]
20:19 <cjming@deploy1002> Finished scap: Backport for [[gerrit:914428|Create autopatroller and patroller groups on bn.wikiquote (T335829)]] (duration: 08m 36s) [production]
20:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P47479 and previous config saved to /var/cache/conftool/dbconfig/20230503-201918-ladsgroup.json [production]
20:14 <fab@deploy1002> Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 19s) [production]
20:13 <fab@deploy1002> Started deploy [airflow-dags/research@f8dad05]: (no justification provided) [production]
20:12 <cjming@deploy1002> cjming and mdsshakil: Backport for [[gerrit:914428|Create autopatroller and patroller groups on bn.wikiquote (T335829)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
20:11 <cjming@deploy1002> Started scap: Backport for [[gerrit:914428|Create autopatroller and patroller groups on bn.wikiquote (T335829)]] [production]
20:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P47478 and previous config saved to /var/cache/conftool/dbconfig/20230503-200411-ladsgroup.json [production]
19:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T335838)', diff saved to https://phabricator.wikimedia.org/P47477 and previous config saved to /var/cache/conftool/dbconfig/20230503-194905-ladsgroup.json [production]
19:43 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot - bking@cumin1001 - T335835 [production]
19:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2177 (T335838)', diff saved to https://phabricator.wikimedia.org/P47476 and previous config saved to /var/cache/conftool/dbconfig/20230503-194238-ladsgroup.json [production]
19:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]