2801-2850 of 10000 results (42ms)
2021-02-25 ยง
13:57 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: REIMAGE [production]
13:55 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2001.codfw.wmnet with reason: REIMAGE [production]
13:55 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: REIMAGE [production]
13:53 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2001.codfw.wmnet with reason: REIMAGE [production]
13:15 <akosiaris> reinitialize all of staging-codfw. kubestage2* and kubestagemaster* have been scheduled downtime in icinga. [production]
12:32 <moritzm> installing openssl security updates on Buster [production]
12:20 <Lucas_WMDE> EU backport&config window done [production]
12:16 <phuedx@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:666425|[stage 1] Enable WVUI search by default to logged-in modern Vector users except on pilot wikis (T249297)]] (duration: 01m 31s) [production]
11:56 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1134.eqiad.wmnet with reason: REIMAGE [production]
11:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1134.eqiad.wmnet with reason: REIMAGE [production]
11:47 <jbond42> upload new wmf-laptop package [production]
11:40 <marostegui> Stop MySQL on db1134 to reimage it to buster T275343 [production]
11:29 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on dborch1001.wikimedia.org with reason: Restart for new kernel [production]
11:29 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on dborch1001.wikimedia.org with reason: Restart for new kernel [production]
11:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host otrs1001.eqiad.wmnet [production]
11:22 <moritzm> reset-failed ifup@ens5.service on otrs1001 T273026 [production]
11:15 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host otrs1001.eqiad.wmnet [production]
11:15 <moritzm> rebooting otrs1001 (ticket.wikimedia.org) for a kernel update [production]
10:59 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker[1117-1118].eqiad.wmnet [production]
10:57 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker[1117-1118].eqiad.wmnet [production]
10:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE [production]
10:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1117.eqiad.wmnet with reason: REIMAGE [production]
10:40 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE [production]
10:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1117.eqiad.wmnet with reason: REIMAGE [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 100%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14481 and previous config saved to /var/cache/conftool/dbconfig/20210225-103719-root.json [production]
10:34 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE [production]
10:32 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 75%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14480 and previous config saved to /var/cache/conftool/dbconfig/20210225-102215-root.json [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 50%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14479 and previous config saved to /var/cache/conftool/dbconfig/20210225-100712-root.json [production]
10:05 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2003.codfw.wmnet with reason: REIMAGE [production]
10:03 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: REIMAGE [production]
10:01 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2003.codfw.wmnet with reason: REIMAGE [production]
10:01 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: REIMAGE [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 25%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14477 and previous config saved to /var/cache/conftool/dbconfig/20210225-095208-root.json [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 10%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14476 and previous config saved to /var/cache/conftool/dbconfig/20210225-093705-root.json [production]
09:32 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2001.codfw.wmnet with reason: REIMAGE [production]
09:32 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2001.codfw.wmnet with reason: REIMAGE [production]
09:21 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1032.eqiad.wmnet [production]
09:14 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1032.eqiad.wmnet [production]
09:10 <effie> upgrade memcached on mc1032, mc2032, mc2036 [production]
08:32 <volans@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:29 <volans@cumin2001> START - Cookbook sre.dns.netbox [production]
08:15 <vgutierrez> restart ats-tls on cp5006 to enable parent proxies support - T274888 [production]
08:15 <XioNoX> un-drain lumen eqiad-codfw link for BW testing [production]
08:07 <XioNoX> drain lumen eqiad-codfw link for BW testing [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1088 to clone db1168 T258361', diff saved to https://phabricator.wikimedia.org/P14474 and previous config saved to /var/cache/conftool/dbconfig/20210225-065018-marostegui.json [production]
06:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 T275019', diff saved to https://phabricator.wikimedia.org/P14473 and previous config saved to /var/cache/conftool/dbconfig/20210225-063243-marostegui.json [production]
00:29 <ryankemper> T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch` [production]
00:05 <ryankemper> T274204 `Ctrl+C`'d out of the current rolling-upgrade; the 3 hosts that have their elasticsearch systemd units in a failing state are running the latest plugin version, meaning the new version is likely the cause of the failures [production]
00:01 <mutante> mwlog1001 - temp disabling puppet to deploy gerrit::661200 - because this is a jessie [production]