6251-6300 of 10000 results (87ms)
2023-01-19 §
06:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
06:06 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
06:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db2118 T327372', diff saved to https://phabricator.wikimedia.org/P43190 and previous config saved to /var/cache/conftool/dbconfig/20230119-060449-ladsgroup.json [production]
06:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db2121 to s7 primary T327372', diff saved to https://phabricator.wikimedia.org/P43189 and previous config saved to /var/cache/conftool/dbconfig/20230119-060316-ladsgroup.json [production]
06:02 <Amir1> Starting s7 codfw failover from db2118 to db2121 - T327372 [production]
05:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db2121 with weight 0 T327372', diff saved to https://phabricator.wikimedia.org/P43188 and previous config saved to /var/cache/conftool/dbconfig/20230119-054243-ladsgroup.json [production]
05:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s7 T327372 [production]
05:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 30 hosts with reason: Primary switchover s7 T327372 [production]
2023-01-18 §
23:47 <zabe> run populateCulComment.php on all group0 and group1 wikis # T327290 [production]
23:42 <cstone> civicrm upgraded from 164270b0 to f6093fb2 [production]
22:35 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: raise heap memory to 12G - bking@cumin1001 - T323646 [production]
22:03 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: raise heap memory to 12G - bking@cumin1001 - T323646 [production]
21:50 <kindrobot> close UTC late backport window [production]
21:50 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] (duration: 10m 45s) [production]
21:41 <kindrobot@deploy1002> essexigyan and kindrobot: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:39 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:881462|[config]: Undeploy GDI Safety Survey Wave 4 (T327296)]] [production]
21:36 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] (duration: 13m 01s) [production]
21:25 <kindrobot@deploy1002> kindrobot and jdlrobson: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:23 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:881451|Bump English Wikipedia event logging from 0.5 to 1% (T326892)]], [[gerrit:881431|Legacy Vector is not a responsive skin (T327256)]] [production]
21:08 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1037.eqiad.wmnet with OS bullseye [production]
21:05 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1036.eqiad.wmnet with OS bullseye [production]
21:03 <kindrobot> start UTC late backport window [production]
20:54 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
20:51 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
20:49 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
20:48 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
20:36 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1037.eqiad.wmnet with OS bullseye [production]
20:35 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1036.eqiad.wmnet with OS bullseye [production]
20:34 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
20:34 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
19:54 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1037.eqiad.wmnet with OS buster [production]
19:54 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:52 <bblack> db1129 and lvs1017: removed misconfigured IP address in wrong vlan from eno1 and /e/n/i [production]
19:51 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:47 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logstash1036.eqiad.wmnet with OS buster [production]
19:47 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:40 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:35 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
19:32 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1037.eqiad.wmnet with reason: host reimage [production]
19:26 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
19:23 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1036.eqiad.wmnet with reason: host reimage [production]
19:19 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1037.eqiad.wmnet with OS buster [production]
18:59 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host logstash1036.eqiad.wmnet with OS buster [production]
18:21 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:878927|Enable the REST API on test-wikidata (T324999)]] (duration: 09m 38s) [production]
18:14 <lucaswerkmeister-wmde@deploy1002> migr and lucaswerkmeister-wmde: Backport for [[gerrit:878927|Enable the REST API on test-wikidata (T324999)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
18:12 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:878927|Enable the REST API on test-wikidata (T324999)]] [production]
17:55 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
17:55 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
17:44 <jnuche@deploy1002> Installation of scap version "4.33.0" completed for 560 hosts [production]
17:44 <jnuche@deploy1002> Installing scap version "4.33.0" for 560 hosts [production]