251-300 of 10000 results (40ms)
2022-03-23 ยง
13:35 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1027.eqiad.wmnet with OS bullseye [production]
13:34 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1010.eqiad.wmnet with reason: host reimage [production]
13:33 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage [production]
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P23008 and previous config saved to /var/cache/conftool/dbconfig/20220323-132635-marostegui.json [production]
13:19 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1010.eqiad.wmnet with OS bullseye [production]
13:16 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS buster [production]
13:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P23005 and previous config saved to /var/cache/conftool/dbconfig/20220323-131130-marostegui.json [production]
13:07 <mmandere> depool cp1082 for reimage - T290005 [production]
12:58 <moritzm> installing bind security updates [production]
12:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T300775)', diff saved to https://phabricator.wikimedia.org/P23004 and previous config saved to /var/cache/conftool/dbconfig/20220323-125625-marostegui.json [production]
12:29 <moritzm> restarting Turnilo for OpenSSL update [production]
12:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1132 after testing', diff saved to https://phabricator.wikimedia.org/P23003 and previous config saved to /var/cache/conftool/dbconfig/20220323-120749-marostegui.json [production]
11:34 <jbond> upload new puppetboard_3.1.0-1+deb11u1_all.deb [production]
11:33 <moritzm> installing apache security updates on stretch [production]
11:00 <mmandere> pool cp1081 with HAProxy as TLS termination layer - T290005 [production]
10:58 <moritzm> restarting apache on matomo1002/piwik.wikimedia.org [production]
10:52 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS buster [production]
10:30 <moritzm> restarting ntpd [production]
10:28 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage [production]
10:24 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage [production]
10:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1132 some more weight T301879', diff saved to https://phabricator.wikimedia.org/P23002 and previous config saved to /var/cache/conftool/dbconfig/20220323-101816-marostegui.json [production]
10:07 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS buster [production]
09:56 <mmandere> depool cp1081 for reimage - T290005 [production]
09:43 <mmandere> pool cp1079 with HAProxy as TLS termination layer - T290005 [production]
09:36 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS buster [production]
09:24 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:17 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
09:15 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage [production]
09:11 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage [production]
09:06 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
08:59 <jayme@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
08:54 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS buster [production]
08:54 <moritzm> restarting spamassassin/clamav on otrs1001/ticket.wikimedia.org [production]
08:51 <mmandere@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1079.eqiad.wmnet with OS buster [production]
08:47 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS buster [production]
08:43 <moritzm> installing openssl security updates [production]
08:36 <mmandere> depool cp1079 for reimage - T290005 [production]
08:24 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1009.eqiad.wmnet with OS bullseye [production]
08:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes1009.eqiad.wmnet with reason: host reimage [production]
08:10 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1009.eqiad.wmnet with reason: host reimage [production]
07:54 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1009.eqiad.wmnet with OS bullseye [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P23001 and previous config saved to /var/cache/conftool/dbconfig/20220323-074408-root.json [production]
07:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P23000 and previous config saved to /var/cache/conftool/dbconfig/20220323-072904-root.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P22999 and previous config saved to /var/cache/conftool/dbconfig/20220323-071400-root.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P22998 and previous config saved to /var/cache/conftool/dbconfig/20220323-065856-root.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P22997 and previous config saved to /var/cache/conftool/dbconfig/20220323-064353-root.json [production]
06:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1112.eqiad.wmnet with OS bullseye [production]
06:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1112.eqiad.wmnet with reason: host reimage [production]
06:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1112.eqiad.wmnet with reason: host reimage [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1112.eqiad.wmnet with OS bullseye [production]