|
2020-10-02
ยง
|
| 10:44 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:43 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12899 and previous config saved to /var/cache/conftool/dbconfig/20201002-104320-kormat.json |
[production] |
| 10:40 |
<jmm@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 10:36 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 10:28 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2106 (re)pooling @ 67%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12898 and previous config saved to /var/cache/conftool/dbconfig/20201002-102817-kormat.json |
[production] |
| 10:13 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2106 (re)pooling @ 33%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12897 and previous config saved to /var/cache/conftool/dbconfig/20201002-101313-kormat.json |
[production] |
| 10:06 |
<jmm@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 09:58 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) |
[production] |
| 09:56 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 09:48 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:28 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:27 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2106 depooling: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12896 and previous config saved to /var/cache/conftool/dbconfig/20201002-092715-kormat.json |
[production] |
| 09:27 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:27 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:19 |
<jayme> |
running ipvsadm -D -t 10.2.1.20:10042; ipvsadm -D -t 10.2.1.16:1969 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet - T255875 T255869 |
[production] |
| 09:18 |
<jayme> |
running ipvsadm -D -t 10.2.2.20:10042; ipvsadm -D -t 10.2.2.16:1969 on lvs1016.eqiad.wmnet,lvs1015.eqiad.wmnet - T255875 T255869 |
[production] |
| 09:17 |
<jayme> |
restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255875 T255869 |
[production] |
| 09:14 |
<jayme> |
restarting pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255875 T255869 |
[production] |
| 09:12 |
<jayme> |
running puppet on lvs servers - T255875 T255869 |
[production] |
| 09:11 |
<arturo> |
added helm3 package to buster-wikimedia/thirdparty/kubeadm-k8s-1-17 (T264221) |
[production] |
| 09:09 |
<jmm@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 09:08 |
<jmm@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 09:08 |
<jmm@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 09:07 |
<hnowlan> |
bootstrapping restbase1029-b cassandra |
[production] |
| 09:05 |
<hashar> |
gerrit: running garbage collector |
[production] |
| 09:00 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:00 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:59 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:59 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:54 |
<dcausse@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy (duration: 00m 03s) |
[production] |
| 08:54 |
<dcausse@deploy1001> |
Started deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy |
[production] |
| 08:42 |
<dcausse@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy (duration: 00m 34s) |
[production] |
| 08:41 |
<dcausse@deploy1001> |
Started deploy [wikimedia/discovery/analytics@5713fb0]: Test stat1007 deploy |
[production] |
| 08:30 |
<dcausse@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date (duration: 00m 33s) |
[production] |
| 08:30 |
<dcausse@deploy1001> |
Started deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date |
[production] |
| 08:29 |
<moritzm> |
installing pyzmq bugfix update from buster point release |
[production] |
| 08:24 |
<moritzm> |
installing nginx security updates on puppetdb* |
[production] |
| 08:17 |
<dcausse@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date (duration: 01m 35s) |
[production] |
| 08:16 |
<dcausse@deploy1001> |
Started deploy [wikimedia/discovery/analytics@5713fb0]: Fix lexeme dumps expected date |
[production] |
| 07:42 |
<moritzm> |
installing libcommons-compress-java security updates |
[production] |
| 07:35 |
<godog> |
swift codfw-prod bump weight for ms-be2057 - T261633 |
[production] |
| 07:29 |
<godog> |
prometheus codfw/k8s, add 50G to the LV |
[production] |
| 07:23 |
<moritzm> |
installing libx11 security updates on buster |
[production] |
| 06:51 |
<_joe_> |
restarting php-fpm on all appservers in eqiad, in batches of 10%, for testing the procedure suggested at T264362 |
[production] |
| 05:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 05:43 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove es2011 from dbctl T264261', diff saved to https://phabricator.wikimedia.org/P12893 and previous config saved to /var/cache/conftool/dbconfig/20201002-053020-marostegui.json |
[production] |