3601-3650 of 10000 results (33ms)
2021-02-25 ยง
15:00 <moritzm> installing libmaxminddb updates from buster 10.8 point release [production]
14:59 <vgutierrez> pool cp4032 [production]
14:42 <vgutierrez> depool cp4032 for ats-tls/NUMA tests [production]
14:35 <klausman@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:27 <moritzm> installing postgresql security updates on buster [production]
14:24 <kormat@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ml-serve-ctrl1001.eqiad.wmnet [production]
14:22 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:20 <klausman@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:17 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:16 <moritzm> installing cairo security updates on buster [production]
14:14 <klausman@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:10 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm for new host ml-serve-ctrl1002.eqiad.wmnet [production]
14:09 <kormat@cumin1001> START - Cookbook sre.ganeti.makevm for new host ml-serve-ctrl1001.eqiad.wmnet [production]
13:57 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: REIMAGE [production]
13:55 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2001.codfw.wmnet with reason: REIMAGE [production]
13:55 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: REIMAGE [production]
13:53 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2001.codfw.wmnet with reason: REIMAGE [production]
13:15 <akosiaris> reinitialize all of staging-codfw. kubestage2* and kubestagemaster* have been scheduled downtime in icinga. [production]
12:32 <moritzm> installing openssl security updates on Buster [production]
12:20 <Lucas_WMDE> EU backport&config window done [production]
12:16 <phuedx@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:666425|[stage 1] Enable WVUI search by default to logged-in modern Vector users except on pilot wikis (T249297)]] (duration: 01m 31s) [production]
11:56 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1134.eqiad.wmnet with reason: REIMAGE [production]
11:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1134.eqiad.wmnet with reason: REIMAGE [production]
11:47 <jbond42> upload new wmf-laptop package [production]
11:40 <marostegui> Stop MySQL on db1134 to reimage it to buster T275343 [production]
11:29 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on dborch1001.wikimedia.org with reason: Restart for new kernel [production]
11:29 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on dborch1001.wikimedia.org with reason: Restart for new kernel [production]
11:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host otrs1001.eqiad.wmnet [production]
11:22 <moritzm> reset-failed ifup@ens5.service on otrs1001 T273026 [production]
11:15 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host otrs1001.eqiad.wmnet [production]
11:15 <moritzm> rebooting otrs1001 (ticket.wikimedia.org) for a kernel update [production]
10:59 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker[1117-1118].eqiad.wmnet [production]
10:57 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker[1117-1118].eqiad.wmnet [production]
10:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE [production]
10:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1117.eqiad.wmnet with reason: REIMAGE [production]
10:40 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE [production]
10:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1117.eqiad.wmnet with reason: REIMAGE [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 100%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14481 and previous config saved to /var/cache/conftool/dbconfig/20210225-103719-root.json [production]
10:34 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE [production]
10:32 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 75%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14480 and previous config saved to /var/cache/conftool/dbconfig/20210225-102215-root.json [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 50%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14479 and previous config saved to /var/cache/conftool/dbconfig/20210225-100712-root.json [production]
10:05 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2003.codfw.wmnet with reason: REIMAGE [production]
10:03 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: REIMAGE [production]
10:01 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2003.codfw.wmnet with reason: REIMAGE [production]
10:01 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: REIMAGE [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 25%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14477 and previous config saved to /var/cache/conftool/dbconfig/20210225-095208-root.json [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1088 (re)pooling @ 10%: After cloning db1168', diff saved to https://phabricator.wikimedia.org/P14476 and previous config saved to /var/cache/conftool/dbconfig/20210225-093705-root.json [production]
09:32 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2001.codfw.wmnet with reason: REIMAGE [production]
09:32 <klausman@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2001.codfw.wmnet with reason: REIMAGE [production]