651-700 of 10000 results (79ms)
2023-01-23 ยง
21:35 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1028.eqiad.wmnet [production]
21:34 <zabe@deploy1002> zabe: Backport for [[gerrit:882746|throttle: Remove expired rule]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:32 <zabe@deploy1002> Started scap: Backport for [[gerrit:882746|throttle: Remove expired rule]] [production]
21:29 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1021.eqiad.wmnet [production]
21:22 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1021.eqiad.wmnet [production]
21:12 <kindrobot> close UTC late backport window [production]
21:12 <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:882715|Enable Page Tools for logged-in users on enwiki (T327686)]] (duration: 09m 00s) [production]
21:04 <kindrobot@deploy1002> jdrewniak and kindrobot: Backport for [[gerrit:882715|Enable Page Tools for logged-in users on enwiki (T327686)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:03 <kindrobot@deploy1002> Started scap: Backport for [[gerrit:882715|Enable Page Tools for logged-in users on enwiki (T327686)]] [production]
21:01 <kindrobot> start UTC late backport window [production]
20:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
20:56 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
20:45 <taavi> restart T315510 on group1 after mwmaint restart, currently running on wikidatawiki [production]
19:48 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1020.eqiad.wmnet [production]
19:41 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1020.eqiad.wmnet [production]
19:37 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1019.eqiad.wmnet [production]
19:30 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1019.eqiad.wmnet [production]
19:24 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1016.eqiad.wmnet [production]
19:18 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1016.eqiad.wmnet [production]
19:17 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host restbase1016.eqiad.wmnet [production]
19:17 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1016.eqiad.wmnet [production]
19:16 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host restbase1016.eqiad.wmnet [production]
19:16 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1016.eqiad.wmnet [production]
18:48 <mutante> miscweb1002 - unload CAS apache module and config; apt-get remove libapache2-mod-auth-cas [production]
18:19 <mutante> miscweb2002 - unlink /etc/apache2/mods-enabled/auth_cas.conf - unlink /etc/apache2/mods-enabled/auth_cas.load - apt-get remove libapache2-mod-auth-cas - T327405 [production]
18:08 <mutante> miscweb2002 - unlink /etc/apache2/mods-enabled/auth_cas.conf - unlink /etc/apache2/mods-enabled/auth_cas.load [production]
18:05 <mutante> miscweb1002 - disabling puppet because latest merge would break apache if it runs, debugging in progress on inactive miscweb2002 [production]
18:02 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on miscweb2002.codfw.wmnet with reason: debugging on iactive server [production]
18:02 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on miscweb2002.codfw.wmnet with reason: debugging on iactive server [production]
17:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P43265 and previous config saved to /var/cache/conftool/dbconfig/20230123-175241-ladsgroup.json [production]
17:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P43264 and previous config saved to /var/cache/conftool/dbconfig/20230123-173736-ladsgroup.json [production]
17:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P43263 and previous config saved to /var/cache/conftool/dbconfig/20230123-172231-ladsgroup.json [production]
17:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2114 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P43262 and previous config saved to /var/cache/conftool/dbconfig/20230123-170726-ladsgroup.json [production]
17:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
17:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:58 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:48 <jdrewniak@deploy1002> Synchronized portals: Wikimedia Portals Update: [[gerrit:882682| Bumping portals to master (T128546)]] (duration: 06m 48s) [production]
16:42 <jdrewniak@deploy1002> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:882682| Bumping portals to master (T128546)]] (duration: 06m 48s) [production]
16:41 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
16:41 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
16:40 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:40 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:35 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:35 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
16:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 100%: After adding a column', diff saved to https://phabricator.wikimedia.org/P43261 and previous config saved to /var/cache/conftool/dbconfig/20230123-163207-root.json [production]