551-600 of 10000 results (44ms)
2022-03-23 §
10:28 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage [production]
10:24 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage [production]
10:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1132 some more weight T301879', diff saved to https://phabricator.wikimedia.org/P23002 and previous config saved to /var/cache/conftool/dbconfig/20220323-101816-marostegui.json [production]
10:07 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS buster [production]
09:56 <mmandere> depool cp1081 for reimage - T290005 [production]
09:43 <mmandere> pool cp1079 with HAProxy as TLS termination layer - T290005 [production]
09:36 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS buster [production]
09:24 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:17 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
09:15 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage [production]
09:11 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage [production]
09:06 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
08:59 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
08:54 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS buster [production]
08:54 <moritzm> restarting spamassassin/clamav on otrs1001/ticket.wikimedia.org [production]
08:51 <mmandere@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1079.eqiad.wmnet with OS buster [production]
08:47 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS buster [production]
08:43 <moritzm> installing openssl security updates [production]
08:36 <mmandere> depool cp1079 for reimage - T290005 [production]
08:24 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1009.eqiad.wmnet with OS bullseye [production]
08:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes1009.eqiad.wmnet with reason: host reimage [production]
08:10 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1009.eqiad.wmnet with reason: host reimage [production]
07:54 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1009.eqiad.wmnet with OS bullseye [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P23001 and previous config saved to /var/cache/conftool/dbconfig/20220323-074408-root.json [production]
07:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P23000 and previous config saved to /var/cache/conftool/dbconfig/20220323-072904-root.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P22999 and previous config saved to /var/cache/conftool/dbconfig/20220323-071400-root.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P22998 and previous config saved to /var/cache/conftool/dbconfig/20220323-065856-root.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P22997 and previous config saved to /var/cache/conftool/dbconfig/20220323-064353-root.json [production]
06:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1112.eqiad.wmnet with OS bullseye [production]
06:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1112.eqiad.wmnet with reason: host reimage [production]
06:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1112.eqiad.wmnet with reason: host reimage [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1112.eqiad.wmnet with OS bullseye [production]
06:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 for reimage', diff saved to https://phabricator.wikimedia.org/P22996 and previous config saved to /var/cache/conftool/dbconfig/20220323-060533-marostegui.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1132 with low weight T301879', diff saved to https://phabricator.wikimedia.org/P22995 and previous config saved to /var/cache/conftool/dbconfig/20220323-060351-marostegui.json [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:20 <ejegg> updated payments-wiki from 3048f0aa to 28e24856 [production]
00:11 <cjming> end running skin preference update script T299104 [production]
2022-03-22 §
23:56 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
23:39 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1024.eqiad.wmnet with reason: host reimage [production]
23:35 <pt1979@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1024.eqiad.wmnet with reason: host reimage [production]
23:23 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
23:11 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
22:46 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]