|
2019-11-08
ยง
|
| 14:40 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:40 |
<jynus> |
stop and upgrade percona-server on test host db1114 |
[production] |
| 13:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 13:12 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Pool db1118 at 20%', diff saved to https://phabricator.wikimedia.org/P9580 and previous config saved to /var/cache/conftool/dbconfig/20191108-131257-jynus.json |
[production] |
| 13:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: ee2027c: Change the language of Votewiki back to English (en) (T230614) (duration: 00m 54s) |
[production] |
| 12:34 |
<elukey@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 12:14 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Pool db1118 at 10%', diff saved to https://phabricator.wikimedia.org/P9578 and previous config saved to /var/cache/conftool/dbconfig/20191108-121444-jynus.json |
[production] |
| 12:02 |
<jynus> |
update and restart db1118 |
[production] |
| 12:01 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1118 fully', diff saved to https://phabricator.wikimedia.org/P9577 and previous config saved to /var/cache/conftool/dbconfig/20191108-120138-jynus.json |
[production] |
| 11:55 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Pool db1118 at 20%', diff saved to https://phabricator.wikimedia.org/P9576 and previous config saved to /var/cache/conftool/dbconfig/20191108-115553-jynus.json |
[production] |
| 11:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Pool db1118 at 50%', diff saved to https://phabricator.wikimedia.org/P9575 and previous config saved to /var/cache/conftool/dbconfig/20191108-112733-jynus.json |
[production] |
| 11:25 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'repool db2130', diff saved to https://phabricator.wikimedia.org/P9574 and previous config saved to /var/cache/conftool/dbconfig/20191108-112503-jynus.json |
[production] |
| 11:12 |
<jynus> |
update and restart db2130 |
[production] |
| 11:11 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2116, depool db2130', diff saved to https://phabricator.wikimedia.org/P9573 and previous config saved to /var/cache/conftool/dbconfig/20191108-111125-jynus.json |
[production] |
| 10:58 |
<Amir1> |
running rebuildItemTerms on 8028 items (T234329) |
[production] |
| 10:51 |
<jynus> |
update and restart db2116 |
[production] |
| 10:50 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2103, depool db2116', diff saved to https://phabricator.wikimedia.org/P9572 and previous config saved to /var/cache/conftool/dbconfig/20191108-105013-jynus.json |
[production] |
| 10:38 |
<jynus> |
update and restart db2103 |
[production] |
| 10:34 |
<jeh> |
enable IPMI `racadm set iDRAC.IPMILan.Enable 1` on cloudcephmon[1-3] T228102 |
[production] |
| 10:33 |
<jeh> |
enable IPMI `racadm set iDRAC.IPMILan.Enable 1` on cloudcephosd[1-3] T224188 |
[production] |
| 10:32 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2092, depool db2103', diff saved to https://phabricator.wikimedia.org/P9571 and previous config saved to /var/cache/conftool/dbconfig/20191108-103218-jynus.json |
[production] |
| 10:19 |
<jynus> |
update and restart db2092 |
[production] |
| 10:18 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2071, depool db2092', diff saved to https://phabricator.wikimedia.org/P9570 and previous config saved to /var/cache/conftool/dbconfig/20191108-101759-jynus.json |
[production] |
| 10:09 |
<elukey> |
restart jvm-based hadoop daemons on an-master100[1,2] to pick up the new openjdk version |
[production] |
| 10:06 |
<jynus> |
update and restart db2071 |
[production] |
| 10:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2071', diff saved to https://phabricator.wikimedia.org/P9569 and previous config saved to /var/cache/conftool/dbconfig/20191108-100310-jynus.json |
[production] |
| 10:01 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2072', diff saved to https://phabricator.wikimedia.org/P9568 and previous config saved to /var/cache/conftool/dbconfig/20191108-100128-jynus.json |
[production] |
| 09:50 |
<moritzm> |
uploaded openjdk 8u232-b09-1~deb10u1 to component/jdk8 for apt.wikimedia.org/buster-wikimedia |
[production] |
| 09:41 |
<jynus> |
update and restart db2072 |
[production] |
| 09:41 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2072', diff saved to https://phabricator.wikimedia.org/P9567 and previous config saved to /var/cache/conftool/dbconfig/20191108-094100-jynus.json |
[production] |
| 09:39 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1106 at 50%', diff saved to https://phabricator.wikimedia.org/P9566 and previous config saved to /var/cache/conftool/dbconfig/20191108-093958-jynus.json |
[production] |
| 09:35 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 09:29 |
<jynus> |
update and restart db2094 |
[production] |
| 09:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1106 at 10%', diff saved to https://phabricator.wikimedia.org/P9565 and previous config saved to /var/cache/conftool/dbconfig/20191108-092735-jynus.json |
[production] |
| 09:10 |
<jynus> |
update and restart db1106 |
[production] |
| 09:08 |
<moritzm> |
installing Java security updates on kafka-jumbo |
[production] |
| 09:07 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 fully', diff saved to https://phabricator.wikimedia.org/P9564 and previous config saved to /var/cache/conftool/dbconfig/20191108-090746-jynus.json |
[production] |
| 09:05 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 09:04 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 at 10%', diff saved to https://phabricator.wikimedia.org/P9563 and previous config saved to /var/cache/conftool/dbconfig/20191108-090451-jynus.json |
[production] |
| 09:00 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 at 50%', diff saved to https://phabricator.wikimedia.org/P9562 and previous config saved to /var/cache/conftool/dbconfig/20191108-090012-jynus.json |
[production] |
| 08:57 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:52 |
<jynus> |
stop and upgrade db1124 (may create temporary lag on wikireplicas) |
[production] |
| 08:31 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 08:23 |
<elukey> |
restart kafka on kafka-jumbo1001 to test the new openjdk |
[production] |
| 08:07 |
<moritzm> |
installing fribidi security updates on Buster |
[production] |
| 03:03 |
<vgutierrez> |
Switch from nginx to ats-tls on cp3054 - T231627 |
[production] |
| 02:42 |
<vgutierrez> |
Switch from nginx to ats-tls on cp3052 - T231627 |
[production] |
| 01:23 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/GlobalBlocking/: Prevent some extra db queries (duration: 00m 53s) |
[production] |
| 01:14 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/resources/: Use internationalized semicolon separators (T233649) (duration: 00m 53s) |
[production] |
| 01:09 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@11d4ad8]: deploying one more time, hopefully without killing elastic (duration: 03m 04s) |
[production] |