4001-4050 of 10000 results (94ms)
2023-08-29 ยง
16:25 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:25 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
16:24 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
16:23 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
16:23 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
16:20 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
16:19 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
16:19 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
16:18 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
16:18 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1022.eqiad.wmnet [production]
16:17 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
16:17 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply [production]
16:17 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/geo-analytics: apply [production]
16:17 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:16 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply [production]
16:16 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/geo-analytics: apply [production]
16:14 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/geo-analytics: apply [production]
16:14 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/geo-analytics: apply [production]
16:13 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:953259|ores-extension: replace first batch of wikis model thresholds with numeric values (T343308)]] (duration: 09m 31s) [production]
16:12 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
16:11 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
16:11 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
16:10 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
16:10 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
16:09 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
16:09 <akosiaris> deploy cxserver mariadb egress functionality. T341117 [production]
16:09 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1022.eqiad.wmnet [production]
16:09 <akosiaris@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
16:09 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
16:09 <akosiaris@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
16:08 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:08 <ladsgroup@deploy1002> ladsgroup and isaranto: Continuing with sync [production]
16:07 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
16:07 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:05 <ladsgroup@deploy1002> ladsgroup and isaranto: Backport for [[gerrit:953259|ores-extension: replace first batch of wikis model thresholds with numeric values (T343308)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
16:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet [production]
16:04 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
16:04 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
16:04 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:953259|ores-extension: replace first batch of wikis model thresholds with numeric values (T343308)]] [production]
16:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T343718)', diff saved to https://phabricator.wikimedia.org/P51939 and previous config saved to /var/cache/conftool/dbconfig/20230829-160415-ladsgroup.json [production]
16:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T343718)', diff saved to https://phabricator.wikimedia.org/P51938 and previous config saved to /var/cache/conftool/dbconfig/20230829-160144-ladsgroup.json [production]
15:58 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet [production]
15:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet [production]
15:56 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2025.codfw.wmnet with OS bullseye [production]
15:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
15:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
15:51 <mvernon@cumin1001> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling reboot on A:swift-fe [production]
15:50 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet [production]
15:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P51937 and previous config saved to /var/cache/conftool/dbconfig/20230829-154909-ladsgroup.json [production]
15:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P51936 and previous config saved to /var/cache/conftool/dbconfig/20230829-154638-ladsgroup.json [production]