1251-1300 of 10000 results (91ms)
2023-09-12 ยง
08:24 <oblivian@deploy1002> Finished scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] (duration: 09m 16s) [production]
08:22 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet [production]
08:18 <oblivian@deploy1002> oblivian: Continuing with sync [production]
08:17 <oblivian@deploy1002> oblivian: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
08:15 <oblivian@deploy1002> Started scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] [production]
08:13 <oblivian@deploy1002> Finished scap: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] (duration: 45m 23s) [production]
08:12 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2014.codfw.wmnet to cluster codfw and group C [production]
08:10 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2014.codfw.wmnet to cluster codfw and group C [production]
08:06 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet [production]
07:58 <brouberol@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1156.eqiad.wmnet [production]
07:58 <oblivian@deploy1002> tto and oblivian: Continuing with sync [production]
07:58 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet [production]
07:56 <brouberol@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1156.eqiad.wmnet [production]
07:56 <oblivian@deploy1002> tto and oblivian: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:53 <brouberol@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1155.eqiad.wmnet [production]
07:51 <brouberol@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1155.eqiad.wmnet [production]
07:51 <brouberol@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1154.eqiad.wmnet [production]
07:49 <brouberol@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1154.eqiad.wmnet [production]
07:45 <brouberol@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1153.eqiad.wmnet [production]
07:43 <brouberol@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1153.eqiad.wmnet [production]
07:36 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host netmon1003.wikimedia.org [production]
07:28 <oblivian@deploy1002> Started scap: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] [production]
07:25 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org [production]
07:23 <oblivian@deploy1002> Finished scap: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] (duration: 13m 46s) [production]
07:17 <oblivian@deploy1002> oblivian: Continuing with sync [production]
07:11 <oblivian@deploy1002> oblivian: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:09 <oblivian@deploy1002> Started scap: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] [production]
07:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1001.eqiad.wmnet [production]
06:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2177 (T343198)', diff saved to https://phabricator.wikimedia.org/P52456 and previous config saved to /var/cache/conftool/dbconfig/20230912-062353-arnaudb.json [production]
06:23 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
06:23 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
06:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T343198)', diff saved to https://phabricator.wikimedia.org/P52455 and previous config saved to /var/cache/conftool/dbconfig/20230912-062332-arnaudb.json [production]
06:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P52454 and previous config saved to /var/cache/conftool/dbconfig/20230912-060825-arnaudb.json [production]
05:53 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P52453 and previous config saved to /var/cache/conftool/dbconfig/20230912-055319-arnaudb.json [production]
05:50 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2014.codfw.wmnet with OS bullseye [production]
05:38 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T343198)', diff saved to https://phabricator.wikimedia.org/P52452 and previous config saved to /var/cache/conftool/dbconfig/20230912-053813-arnaudb.json [production]
05:32 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2014.codfw.wmnet with reason: host reimage [production]
05:27 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2014.codfw.wmnet with reason: host reimage [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1119 with Debian Bookworm in s1 with just 10% T339185', diff saved to https://phabricator.wikimedia.org/P52450 and previous config saved to /var/cache/conftool/dbconfig/20230912-051753-marostegui.json [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2158', diff saved to https://phabricator.wikimedia.org/P52449 and previous config saved to /var/cache/conftool/dbconfig/20230912-051725-root.json [production]
05:11 <moritzm> installing aom security updates [production]
05:08 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2014.codfw.wmnet with OS bullseye [production]
05:00 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2014.codfw.wmnet to cluster codfw and group C [production]
05:00 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1158 (T343198)', diff saved to https://phabricator.wikimedia.org/P52448 and previous config saved to /var/cache/conftool/dbconfig/20230912-050033-arnaudb.json [production]
05:00 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
05:00 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
05:00 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
04:59 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
04:59 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136 (T343198)', diff saved to https://phabricator.wikimedia.org/P52447 and previous config saved to /var/cache/conftool/dbconfig/20230912-045944-arnaudb.json [production]
04:56 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2014.codfw.wmnet to cluster codfw and group C [production]