2801-2850 of 10000 results (106ms)
2024-04-09 ยง
17:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
17:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60111 and previous config saved to /var/cache/conftool/dbconfig/20240409-170155-arnaudb.json [production]
16:56 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60110 and previous config saved to /var/cache/conftool/dbconfig/20240409-165438-arnaudb.json [production]
16:54 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage [production]
16:46 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60109 and previous config saved to /var/cache/conftool/dbconfig/20240409-164647-arnaudb.json [production]
16:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60108 and previous config saved to /var/cache/conftool/dbconfig/20240409-163931-arnaudb.json [production]
16:38 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye [production]
16:37 <sukhe@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1113.eqiad.wmnet with OS bullseye [production]
16:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60107 and previous config saved to /var/cache/conftool/dbconfig/20240409-163140-arnaudb.json [production]
16:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60106 and previous config saved to /var/cache/conftool/dbconfig/20240409-162423-arnaudb.json [production]
16:24 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye [production]
16:21 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60105 and previous config saved to /var/cache/conftool/dbconfig/20240409-162142-arnaudb.json [production]
16:21 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:21 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:21 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60104 and previous config saved to /var/cache/conftool/dbconfig/20240409-162119-arnaudb.json [production]
16:21 <sukhe@cumin1002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1113.eqiad.wmnet [production]
16:20 <sukhe@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1113.eqiad.wmnet [production]
16:20 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage [production]
16:19 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage [production]
16:19 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be) [production]
16:16 <sukhe> depool cp1113 for PXE boot issue related testing T350179 [production]
16:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60103 and previous config saved to /var/cache/conftool/dbconfig/20240409-161632-arnaudb.json [production]
16:13 <denisse> Deleting unused webperf TLS certificates - T360414 [production]
16:13 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1005.eqiad.wmnet with OS bullseye [production]
16:06 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be) [production]
16:06 <sukhe> pool cp4052 after reimaging and new NIC firmware [production]
16:06 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60102 and previous config saved to /var/cache/conftool/dbconfig/20240409-160612-arnaudb.json [production]
16:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60101 and previous config saved to /var/cache/conftool/dbconfig/20240409-160225-arnaudb.json [production]
16:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
16:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
16:01 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm [production]
15:56 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage [production]
15:54 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye [production]
15:54 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage [production]
15:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60100 and previous config saved to /var/cache/conftool/dbconfig/20240409-155104-arnaudb.json [production]
15:48 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
15:48 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
15:45 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage [production]
15:42 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage [production]
15:40 <btullis@cumin1002> START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye [production]
15:39 <moritzm> installing python2.7 security updates [production]
15:35 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60099 and previous config saved to /var/cache/conftool/dbconfig/20240409-153557-arnaudb.json [production]
15:35 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
15:35 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
15:35 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60098 and previous config saved to /var/cache/conftool/dbconfig/20240409-153512-arnaudb.json [production]
15:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60097 and previous config saved to /var/cache/conftool/dbconfig/20240409-153315-arnaudb.json [production]
15:33 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
15:33 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
15:33 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance [production]