751-800 of 10000 results (53ms)
2022-03-09 ยง
11:13 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host cumin1001.eqiad.wmnet with OS bullseye [production]
11:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
11:11 <awight@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:767510|Template search improvements to all wikis except enwiki (T286990)]] (duration: 00m 51s) [production]
11:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
11:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
10:58 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudvirt1016.eqiad.wmnet [production]
10:55 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
10:55 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
10:55 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
10:51 <btullis@cumin2002> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
10:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
10:42 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2001.wikimedia.org [production]
10:40 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host idp-test2001.wikimedia.org [production]
10:39 <btullis@cumin2002> START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet [production]
10:34 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet [production]
10:32 <marostegui@cumin2002> dbctl commit (dc=all): 'Depooling db1175 (T298294)', diff saved to https://phabricator.wikimedia.org/P22194 and previous config saved to /var/cache/conftool/dbconfig/20220309-103226-marostegui.json [production]
10:31 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
10:31 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
10:31 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298294)', diff saved to https://phabricator.wikimedia.org/P22193 and previous config saved to /var/cache/conftool/dbconfig/20220309-103146-marostegui.json [production]
10:29 <marostegui> dbmaint on s6@eqiad T272512 [production]
10:29 <marostegui> dbmaint on s3@eqiad T298295 [production]
10:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
10:21 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
10:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
10:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
10:17 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
10:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet [production]
10:16 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P22192 and previous config saved to /var/cache/conftool/dbconfig/20220309-101610-marostegui.json [production]
10:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
10:11 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
10:08 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:769400|reenable DPL on nowikimedia]] (duration: 00m 51s) [production]
10:00 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P22191 and previous config saved to /var/cache/conftool/dbconfig/20220309-100036-marostegui.json [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2147', diff saved to https://phabricator.wikimedia.org/P22190 and previous config saved to /var/cache/conftool/dbconfig/20220309-094704-marostegui.json [production]
09:45 <marostegui> dbmaint on s7@eqiad T298295 [production]
09:45 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298294)', diff saved to https://phabricator.wikimedia.org/P22189 and previous config saved to /var/cache/conftool/dbconfig/20220309-094501-marostegui.json [production]
09:31 <marostegui@cumin2002> dbctl commit (dc=all): 'Depooling db1098:3317 (T300775)', diff saved to https://phabricator.wikimedia.org/P22188 and previous config saved to /var/cache/conftool/dbconfig/20220309-093119-marostegui.json [production]
09:30 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
09:30 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
09:27 <marostegui@cumin2002> dbctl commit (dc=all): 'Depooling db1166 (T298294)', diff saved to https://phabricator.wikimedia.org/P22187 and previous config saved to /var/cache/conftool/dbconfig/20220309-092731-marostegui.json [production]
09:26 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
09:26 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
09:23 <marostegui> dbmaint on s2@eqiad T298295 [production]
09:18 <marostegui> dbmaint on s1@eqiad T298295 [production]
09:16 <marostegui> dbmaint on s4@eqiad T298295 [production]
09:07 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
09:07 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
09:07 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298294)', diff saved to https://phabricator.wikimedia.org/P22186 and previous config saved to /var/cache/conftool/dbconfig/20220309-090737-marostegui.json [production]