1201-1250 of 10000 results (54ms)
2022-06-04 §
04:53 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
04:53 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host clouddumps1001.wikimedia.org with OS bullseye [production]
04:28 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
04:24 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
03:53 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
2022-06-03 §
22:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
22:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
22:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298560)', diff saved to https://phabricator.wikimedia.org/P29387 and previous config saved to /var/cache/conftool/dbconfig/20220603-221938-ladsgroup.json [production]
22:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P29386 and previous config saved to /var/cache/conftool/dbconfig/20220603-220433-ladsgroup.json [production]
21:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P29385 and previous config saved to /var/cache/conftool/dbconfig/20220603-214928-ladsgroup.json [production]
21:36 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: restart to enable S3 plugin - bking@cumin1001 - T309720 [production]
21:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298560)', diff saved to https://phabricator.wikimedia.org/P29384 and previous config saved to /var/cache/conftool/dbconfig/20220603-213423-ladsgroup.json [production]
20:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T298560)', diff saved to https://phabricator.wikimedia.org/P29383 and previous config saved to /var/cache/conftool/dbconfig/20220603-200606-ladsgroup.json [production]
20:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
20:06 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
20:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T298560)', diff saved to https://phabricator.wikimedia.org/P29382 and previous config saved to /var/cache/conftool/dbconfig/20220603-200557-ladsgroup.json [production]
19:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P29381 and previous config saved to /var/cache/conftool/dbconfig/20220603-195052-ladsgroup.json [production]
19:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P29380 and previous config saved to /var/cache/conftool/dbconfig/20220603-193547-ladsgroup.json [production]
19:29 <mutante> gitlab2002 - stop rsync service, apt-get remove --purge rsync, delete /etc/rsync.d/ and /etc/rsyncd.conf - after gerrit:802847 T274463 [production]
19:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T298560)', diff saved to https://phabricator.wikimedia.org/P29379 and previous config saved to /var/cache/conftool/dbconfig/20220603-192042-ladsgroup.json [production]
18:51 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
18:51 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
18:47 <mutante> testreduce - re-enabling Icinga notifications that were disabled for unknown reasons [production]
18:45 <mutante> testreduce1001 - systemctl reset-failed after gerrit:800245 removed failed auto_restart services for non-existing apache and php services [production]
18:34 <mutante> deleting expired digicert TLS certs https://gerrit.wikimedia.org/r/c/operations/puppet/+/791678 [production]
18:09 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2019.codfw.wmnet [production]
18:01 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2019.codfw.wmnet [production]
17:18 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:20 <dancy@deploy1002> sync-wikiversions aborted: testing mediawiki container image build and deploy (duration: 07m 07s) [production]
16:20 <dancy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
16:20 <dancy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
16:20 <dancy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
16:19 <dancy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:17 <dancy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
16:15 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
16:15 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
16:13 <dancy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:12 <dancy@deploy1002> sync-wikiversions aborted: testing mediawiki container image build and deploy (duration: 00m 11s) [production]
16:11 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: restart to enable S3 plugin - bking@cumin1001 - T309720 [production]
16:06 <herron@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. [production]
14:58 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
14:58 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on mx1001.wikimedia.org with reason: BDAT [production]
14:25 <herron@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. [production]
14:14 <inflatador> patching and restarting a few eqiad elastic hosts T309868 [production]
12:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T298560)', diff saved to https://phabricator.wikimedia.org/P29370 and previous config saved to /var/cache/conftool/dbconfig/20220603-120758-ladsgroup.json [production]
12:07 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
12:07 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]