1801-1850 of 10000 results (51ms)
2022-03-14 §
06:12 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
06:12 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
06:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
06:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
2022-03-11 §
15:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2014.codfw.wmnet with OS bullseye [production]
15:44 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2014.codfw.wmnet with reason: host reimage [production]
15:42 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2014.codfw.wmnet with reason: host reimage [production]
15:39 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
15:38 <jgiannelos@deploy1002> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
15:37 <jgiannelos@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
15:36 <jgiannelos@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
15:36 <jgiannelos@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:35 <jgiannelos@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:33 <jgiannelos@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:33 <jgiannelos@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:27 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2014.codfw.wmnet with OS bullseye [production]
15:07 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host kubernetes2013.codfw.wmnet with OS bullseye [production]
15:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 100%: After reboot', diff saved to https://phabricator.wikimedia.org/P22374 and previous config saved to /var/cache/conftool/dbconfig/20220311-150702-root.json [production]
15:02 <XioNoX> cr1/2-eqiad AVOID-PATHS as-path TI "6762 .*" [production]
15:02 <XioNoX> cr2-esams AVOID-PATHS as-path TI "6762 .*" <- rolled back [production]
14:57 <XioNoX> cr2-esams AVOID-PATHS as-path TI "6762 .*" [production]
14:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2013.codfw.wmnet with reason: host reimage [production]
14:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 75%: After reboot', diff saved to https://phabricator.wikimedia.org/P22373 and previous config saved to /var/cache/conftool/dbconfig/20220311-145159-root.json [production]
14:51 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2013.codfw.wmnet with reason: host reimage [production]
14:36 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 50%: After reboot', diff saved to https://phabricator.wikimedia.org/P22372 and previous config saved to /var/cache/conftool/dbconfig/20220311-143652-root.json [production]
14:35 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2013.codfw.wmnet with OS bullseye [production]
14:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 25%: After reboot', diff saved to https://phabricator.wikimedia.org/P22371 and previous config saved to /var/cache/conftool/dbconfig/20220311-142147-root.json [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 10%: After reboot', diff saved to https://phabricator.wikimedia.org/P22370 and previous config saved to /var/cache/conftool/dbconfig/20220311-140641-root.json [production]
14:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1170:3317', diff saved to https://phabricator.wikimedia.org/P22369 and previous config saved to /var/cache/conftool/dbconfig/20220311-140549-marostegui.json [production]
13:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 5%: After reboot', diff saved to https://phabricator.wikimedia.org/P22368 and previous config saved to /var/cache/conftool/dbconfig/20220311-135137-root.json [production]
13:49 <marostegui> dbmaint on s8@eqiad T300775 [production]
13:49 <marostegui> dbmaint on s1@eqiad T298294 [production]
13:43 <jelto> update pcc facts [production]
13:36 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 1%: After reboot', diff saved to https://phabricator.wikimedia.org/P22367 and previous config saved to /var/cache/conftool/dbconfig/20220311-133633-root.json [production]
13:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1123', diff saved to https://phabricator.wikimedia.org/P22366 and previous config saved to /var/cache/conftool/dbconfig/20220311-133407-marostegui.json [production]
12:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2001.codfw.wmnet [production]
12:00 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:55 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
11:51 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts cumin2001.codfw.wmnet [production]
11:18 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2012.codfw.wmnet with OS bullseye [production]
11:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
11:11 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet [production]
11:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2012.codfw.wmnet with reason: host reimage [production]
11:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2012.codfw.wmnet with reason: host reimage [production]
10:59 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. [production]
10:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2110.codfw.wmnet with OS bullseye [production]
10:46 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2012.codfw.wmnet with OS bullseye [production]
10:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2011.codfw.wmnet with OS bullseye [production]
10:39 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. [production]
10:35 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons. [production]