301-350 of 10000 results (34ms)
2022-05-12 ยง
13:26 <tgr@deploy1002> helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [production]
13:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 5%: After optimizing recentchanges', diff saved to https://phabricator.wikimedia.org/P27808 and previous config saved to /var/cache/conftool/dbconfig/20220512-132434-root.json [production]
13:23 <tgr@deploy1002> helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [production]
13:21 <tgr@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [production]
13:19 <tgr@deploy1002> helmfile [staging] START helmfile.d/services/linkrecommendation: apply [production]
13:17 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ores1007.eqiad.wmnet with OS buster [production]
13:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ores1004.eqiad.wmnet with OS buster [production]
12:45 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ores1007.eqiad.wmnet with reason: host reimage [production]
12:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127 for optimizing recentchanges', diff saved to https://phabricator.wikimedia.org/P27807 and previous config saved to /var/cache/conftool/dbconfig/20220512-124406-marostegui.json [production]
12:43 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1003.eqiad.wmnet [production]
12:42 <klausman@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ores1007.eqiad.wmnet with reason: host reimage [production]
12:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ores1004.eqiad.wmnet with reason: host reimage [production]
12:38 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1003.eqiad.wmnet [production]
12:37 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ores1004.eqiad.wmnet with reason: host reimage [production]
12:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
12:34 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
12:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
12:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
12:30 <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.10/includes/api/ApiQueryInfo.php: Backport: [[gerrit:791252|ApiQueryInfo: Force PRIMARY index on templatelinks (T308207)]] (duration: 00m 50s) [production]
12:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
12:28 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1002.eqiad.wmnet [production]
12:27 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
12:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
12:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase traffic on db1127 to test 10.6 T308126', diff saved to https://phabricator.wikimedia.org/P27806 and previous config saved to /var/cache/conftool/dbconfig/20220512-122707-marostegui.json [production]
12:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
12:24 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1002.eqiad.wmnet [production]
12:20 <klausman@cumin1001> START - Cookbook sre.hosts.reimage for host ores1007.eqiad.wmnet with OS buster [production]
12:17 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1001.eqiad.wmnet [production]
12:14 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ores1005.eqiad.wmnet with OS buster [production]
12:12 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host ores1004.eqiad.wmnet with OS buster [production]
12:12 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet [production]
12:04 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2003.codfw.wmnet [production]
12:00 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2003.codfw.wmnet [production]
11:57 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2002.codfw.wmnet [production]
11:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase traffic on db1127 to test 10.6 T308126', diff saved to https://phabricator.wikimedia.org/P27805 and previous config saved to /var/cache/conftool/dbconfig/20220512-115445-marostegui.json [production]
11:51 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2002.codfw.wmnet [production]
11:50 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2001.codfw.wmnet [production]
11:46 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2001.codfw.wmnet [production]
11:43 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ores1005.eqiad.wmnet with reason: host reimage [production]
11:40 <klausman@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ores1005.eqiad.wmnet with reason: host reimage [production]
11:21 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs1020.eqiad.wmnet with OS bullseye [production]
11:17 <klausman@cumin1001> START - Cookbook sre.hosts.reimage for host ores1005.eqiad.wmnet with OS buster [production]
11:14 <jmm@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idp-test1002.wikimedia.org [production]
10:55 <jmm@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase traffic on db1127 to test 10.6 T308126', diff saved to https://phabricator.wikimedia.org/P27804 and previous config saved to /var/cache/conftool/dbconfig/20220512-105432-marostegui.json [production]
10:50 <jmm@cumin1001> START - Cookbook sre.dns.netbox [production]
10:49 <jmm@cumin1001> START - Cookbook sre.ganeti.makevm for new host idp-test1002.wikimedia.org [production]
10:46 <jmm@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idp-test2002.wikimedia.org [production]
10:45 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host aqs1020.eqiad.wmnet with OS bullseye [production]
10:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase traffic on db1127 to test 10.6 T308126', diff saved to https://phabricator.wikimedia.org/P27803 and previous config saved to /var/cache/conftool/dbconfig/20220512-103333-marostegui.json [production]