|
2023-09-12
ยง
|
| 08:41 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T343198)', diff saved to https://phabricator.wikimedia.org/P52459 and previous config saved to /var/cache/conftool/dbconfig/20230912-084059-arnaudb.json |
[production] |
| 08:39 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.26 refs T343728 |
[production] |
| 08:38 |
<moritzm> |
rebalance Ganeti cluster in codfw/C following node replacement |
[production] |
| 08:24 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] (duration: 09m 16s) |
[production] |
| 08:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet |
[production] |
| 08:18 |
<oblivian@deploy1002> |
oblivian: Continuing with sync |
[production] |
| 08:17 |
<oblivian@deploy1002> |
oblivian: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 08:15 |
<oblivian@deploy1002> |
Started scap: Backport for [[gerrit:951048|Replace calls to wfHostname with clusterconfig ones]] |
[production] |
| 08:13 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] (duration: 45m 23s) |
[production] |
| 08:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2014.codfw.wmnet to cluster codfw and group C |
[production] |
| 08:10 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti2014.codfw.wmnet to cluster codfw and group C |
[production] |
| 08:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet |
[production] |
| 07:58 |
<brouberol@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1156.eqiad.wmnet |
[production] |
| 07:58 |
<oblivian@deploy1002> |
tto and oblivian: Continuing with sync |
[production] |
| 07:58 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet |
[production] |
| 07:56 |
<brouberol@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1156.eqiad.wmnet |
[production] |
| 07:56 |
<oblivian@deploy1002> |
tto and oblivian: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:53 |
<brouberol@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1155.eqiad.wmnet |
[production] |
| 07:51 |
<brouberol@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1155.eqiad.wmnet |
[production] |
| 07:51 |
<brouberol@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1154.eqiad.wmnet |
[production] |
| 07:49 |
<brouberol@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1154.eqiad.wmnet |
[production] |
| 07:45 |
<brouberol@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1153.eqiad.wmnet |
[production] |
| 07:43 |
<brouberol@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1153.eqiad.wmnet |
[production] |
| 07:36 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host netmon1003.wikimedia.org |
[production] |
| 07:28 |
<oblivian@deploy1002> |
Started scap: Backport for [[gerrit:951047|ClusterConfig: also allow to return hostname]], [[gerrit:668156|Enable PageNotice on enwiktionary beta (T61245)]] |
[production] |
| 07:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org |
[production] |
| 07:23 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] (duration: 13m 46s) |
[production] |
| 07:17 |
<oblivian@deploy1002> |
oblivian: Continuing with sync |
[production] |
| 07:11 |
<oblivian@deploy1002> |
oblivian: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:09 |
<oblivian@deploy1002> |
Started scap: Backport for [[gerrit:943036|update noc README]], [[gerrit:951046|Use ClusterConfig]] |
[production] |
| 07:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1001.eqiad.wmnet |
[production] |
| 06:23 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2177 (T343198)', diff saved to https://phabricator.wikimedia.org/P52456 and previous config saved to /var/cache/conftool/dbconfig/20230912-062353-arnaudb.json |
[production] |
| 06:23 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 06:23 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 06:23 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T343198)', diff saved to https://phabricator.wikimedia.org/P52455 and previous config saved to /var/cache/conftool/dbconfig/20230912-062332-arnaudb.json |
[production] |
| 06:08 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P52454 and previous config saved to /var/cache/conftool/dbconfig/20230912-060825-arnaudb.json |
[production] |
| 05:53 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P52453 and previous config saved to /var/cache/conftool/dbconfig/20230912-055319-arnaudb.json |
[production] |
| 05:50 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2014.codfw.wmnet with OS bullseye |
[production] |
| 05:38 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T343198)', diff saved to https://phabricator.wikimedia.org/P52452 and previous config saved to /var/cache/conftool/dbconfig/20230912-053813-arnaudb.json |
[production] |
| 05:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2014.codfw.wmnet with reason: host reimage |
[production] |
| 05:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2014.codfw.wmnet with reason: host reimage |
[production] |
| 05:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1119 with Debian Bookworm in s1 with just 10% T339185', diff saved to https://phabricator.wikimedia.org/P52450 and previous config saved to /var/cache/conftool/dbconfig/20230912-051753-marostegui.json |
[production] |
| 05:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2158', diff saved to https://phabricator.wikimedia.org/P52449 and previous config saved to /var/cache/conftool/dbconfig/20230912-051725-root.json |
[production] |
| 05:11 |
<moritzm> |
installing aom security updates |
[production] |
| 05:08 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2014.codfw.wmnet with OS bullseye |
[production] |
| 05:00 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2014.codfw.wmnet to cluster codfw and group C |
[production] |
| 05:00 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db1158 (T343198)', diff saved to https://phabricator.wikimedia.org/P52448 and previous config saved to /var/cache/conftool/dbconfig/20230912-050033-arnaudb.json |
[production] |
| 05:00 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:00 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:00 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |