3451-3500 of 10000 results (37ms)
2023-05-18 §
21:31 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.9 refs T330215 [production]
18:50 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.8 refs T330215 [production]
18:18 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.9 refs T330215 [production]
12:59 <otto@deploy1002> Synchronized wmf-config/ext-EventStreamConfig.php: Revert Enable First Input Delay events. This is causing validation errors as well as breakages in the hadoop ingestion pipepine - T332012 (duration: 06m 19s) [production]
12:46 <otto@deploy1002> Synchronized wmf-config/ext-EventLogging.php: Revert Enable First Input Delay events. This is causing validation errors as well as breakages in the hadoop ingestion pipepine - T332012 (duration: 07m 00s) [production]
10:30 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1110.eqiad.wmnet with reason: Troubleshooting failed disk [production]
10:29 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on an-worker1110.eqiad.wmnet with reason: Troubleshooting failed disk [production]
2023-05-17 §
22:15 <krinkle@deploy1002> Synchronized wmf-config/: T332012 (duration: 06m 51s) [production]
18:34 <brennen@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.9 refs T330215 (duration: 06m 22s) [production]
18:27 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.9 refs T330215 [production]
18:03 <brennen> train 1.41.0-wmf.9 (T330215): no current blockers, rolling to group1 as backup-backup conductor [production]
14:39 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams - EventBus: produce to mediawiki.page_change.v1 stream - T336817 (duration: 06m 20s) [production]
14:14 <otto@deploy1002> Synchronized wmf-config/ext-EventStreamConfig.php: wgEventStreams - Declare mediawiki.page_change.v1 stream - T336817 (duration: 07m 30s) [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 4%: Repooling after a crash', diff saved to https://phabricator.wikimedia.org/P48277 and previous config saved to /var/cache/conftool/dbconfig/20230517-072827-root.json [production]
00:15 <krinkle@deploy1002> Synchronized wmf-config/: I4cfa4a2474b4e (duration: 06m 14s) [production]
2023-05-16 §
18:34 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.9 refs T330215 [production]
03:54 <mwpresync@deploy1002> Pruned MediaWiki: 1.41.0-wmf.6, 1.41.0-wmf.7 (duration: 02m 26s) [production]
03:51 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.9 refs T330215 (duration: 48m 47s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.9 refs T330215 [production]
2023-05-12 §
16:26 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on lists1003.wikimedia.org with reason: maintenance [production]
16:26 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on lists1003.wikimedia.org with reason: maintenance [production]
09:18 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.8 refs T330214 [production]
2023-05-11 §
23:24 <rzl@cumin2002> conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw14(3[789]|4[056]|57)\.eqiad\.wmnet [production]
23:22 <rzl@cumin2002> conftool action : set/pooled=no; selector: cluster=videoscaler,name=mw14(3[789]|4[056]57)\.eqiad\.wmnet [production]
17:12 <brennen@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.8 refs T330214 (duration: 06m 14s) [production]
17:06 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.8 refs T330214 [production]
16:37 <brennen> train 1.41.0-wmf.8 (T330214): rolling back to group1 to test for T336504 presence/absence on enwiki [production]
13:49 <moritzm> uploaded wmf-laptop 0.5.7 to component/wmf-sre-laptop [production]
08:08 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.8 refs T330214 [production]
2023-05-10 §
08:21 <hashar@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.8 refs T330214 (duration: 05m 55s) [production]
08:15 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.8 refs T330214 [production]
2023-05-09 §
13:28 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1088.eqiad.wmnet with reason: Replacing RAID controller battery [production]
13:28 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on an-worker1088.eqiad.wmnet with reason: Replacing RAID controller battery [production]
09:17 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.8 refs T330214 [production]
03:50 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.8 refs T330214 (duration: 47m 55s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.8 refs T330214 [production]
2023-05-08 §
19:20 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs2006.codfw.wmnet with reason: rebooting to help with lag [production]
19:20 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs2006.codfw.wmnet with reason: rebooting to help with lag [production]
19:18 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs1004.eqiad.wmnet with reason: rebooting to help with lag [production]
19:18 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs1004.eqiad.wmnet with reason: rebooting to help with lag [production]
19:12 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 19 hosts with reason: rebooting to help with lag [production]
19:12 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 19 hosts with reason: rebooting to help with lag [production]
14:51 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.7 refs T330214 [production]
14:40 <brennen> train 1.41.0-wmf.7 (T330213): proceeding to all wikis [production]
14:08 <otto@deploy1002> Synchronized wmf-config/ext-EventStreamConfig.php: wgEventStreams - Add mediawiki.page_outlink_topic_prediction_change stream - T328899 (duration: 06m 54s) [production]
2023-05-05 §
18:25 <brennen> train 1.41.0-wmf.7 (T330213): trying revert for T336008, T336022 [production]
2023-05-04 §
20:57 <brennen@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.7 refs T330213 (duration: 06m 02s) [production]
20:51 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.7 refs T330213 [production]
18:16 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.7 refs T330213 [production]
18:08 <brennen> train 1.41.0-wmf.7 (T330213): logs fairly quiet and no current blockers, rolling to group2 [production]