5951-6000 of 10000 results (65ms)
2022-06-08 ยง
21:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:13 <mutante> mw1415 - scap pull, restart apache, /usr/local/sbin/restart-php7.2-fpm (INFO: The server is depooled from all services. Restarting the service directly) [production]
21:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:58 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1038.eqiad.wmnet [production]
20:52 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1038.eqiad.wmnet [production]
20:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:42 <Krinkle> krinkle@mw1415: Run `scap pull` manually ref T310225 [production]
20:35 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: Revert "group0 wikis to 1.39.0-wmf.15" [production]
20:33 <dduvall> rolling back group0 as well due to T310214 [production]
19:58 <urandom> restarting Cassandra, aqs1010-{a,b}, to apply logback work-around -- T309896 [production]
19:51 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1037.eqiad.wmnet [production]
19:46 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1037.eqiad.wmnet [production]
19:32 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2055.codfw.wmnet [production]
19:28 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2055.codfw.wmnet [production]
19:27 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2054.codfw.wmnet [production]
19:23 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2054.codfw.wmnet [production]
19:23 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2053.codfw.wmnet [production]
19:20 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2053.codfw.wmnet [production]
19:19 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2052.codfw.wmnet [production]
19:14 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2052.codfw.wmnet [production]
19:14 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2051.codfw.wmnet [production]
19:08 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2051.codfw.wmnet [production]
18:41 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:40 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:38 <urandom> uprading aqs1010.eqiad.wmnet to Cassandra 3.11.13 (canary) -- T309896 [production]
18:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
18:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
18:28 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.39.0-wmf.15" [production]
18:24 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:23 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:21 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
18:20 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
18:20 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.15 refs T308068 (duration: 03m 25s) [production]
18:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T310011)', diff saved to https://phabricator.wikimedia.org/P29561 and previous config saved to /var/cache/conftool/dbconfig/20220608-182051-marostegui.json [production]
18:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:17 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.15 refs T308068 [production]
18:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P29560 and previous config saved to /var/cache/conftool/dbconfig/20220608-180546-marostegui.json [production]
18:02 <joal@deploy1002> Finished deploy [airflow-dags/analytics@6b368f4]: Update more jobs to spark3 (duration: 00m 13s) [production]
18:02 <joal@deploy1002> Started deploy [airflow-dags/analytics@6b368f4]: Update more jobs to spark3 [production]
17:54 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti3003.esams.wmnet with OS bullseye [production]
17:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P29559 and previous config saved to /var/cache/conftool/dbconfig/20220608-175041-marostegui.json [production]
17:45 <rzl@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
17:44 <rzl@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]