|
2019-11-08
§
|
| 10:50 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2103, depool db2116', diff saved to https://phabricator.wikimedia.org/P9572 and previous config saved to /var/cache/conftool/dbconfig/20191108-105013-jynus.json |
[production] |
| 10:38 |
<jynus> |
update and restart db2103 |
[production] |
| 10:34 |
<jeh> |
enable IPMI `racadm set iDRAC.IPMILan.Enable 1` on cloudcephmon[1-3] T228102 |
[production] |
| 10:33 |
<jeh> |
enable IPMI `racadm set iDRAC.IPMILan.Enable 1` on cloudcephosd[1-3] T224188 |
[production] |
| 10:32 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2092, depool db2103', diff saved to https://phabricator.wikimedia.org/P9571 and previous config saved to /var/cache/conftool/dbconfig/20191108-103218-jynus.json |
[production] |
| 10:19 |
<jynus> |
update and restart db2092 |
[production] |
| 10:18 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2071, depool db2092', diff saved to https://phabricator.wikimedia.org/P9570 and previous config saved to /var/cache/conftool/dbconfig/20191108-101759-jynus.json |
[production] |
| 10:09 |
<elukey> |
restart jvm-based hadoop daemons on an-master100[1,2] to pick up the new openjdk version |
[production] |
| 10:06 |
<jynus> |
update and restart db2071 |
[production] |
| 10:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2071', diff saved to https://phabricator.wikimedia.org/P9569 and previous config saved to /var/cache/conftool/dbconfig/20191108-100310-jynus.json |
[production] |
| 10:01 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db2072', diff saved to https://phabricator.wikimedia.org/P9568 and previous config saved to /var/cache/conftool/dbconfig/20191108-100128-jynus.json |
[production] |
| 09:50 |
<moritzm> |
uploaded openjdk 8u232-b09-1~deb10u1 to component/jdk8 for apt.wikimedia.org/buster-wikimedia |
[production] |
| 09:41 |
<jynus> |
update and restart db2072 |
[production] |
| 09:41 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2072', diff saved to https://phabricator.wikimedia.org/P9567 and previous config saved to /var/cache/conftool/dbconfig/20191108-094100-jynus.json |
[production] |
| 09:39 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1106 at 50%', diff saved to https://phabricator.wikimedia.org/P9566 and previous config saved to /var/cache/conftool/dbconfig/20191108-093958-jynus.json |
[production] |
| 09:35 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 09:29 |
<jynus> |
update and restart db2094 |
[production] |
| 09:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1106 at 10%', diff saved to https://phabricator.wikimedia.org/P9565 and previous config saved to /var/cache/conftool/dbconfig/20191108-092735-jynus.json |
[production] |
| 09:10 |
<jynus> |
update and restart db1106 |
[production] |
| 09:08 |
<moritzm> |
installing Java security updates on kafka-jumbo |
[production] |
| 09:07 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 fully', diff saved to https://phabricator.wikimedia.org/P9564 and previous config saved to /var/cache/conftool/dbconfig/20191108-090746-jynus.json |
[production] |
| 09:05 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 09:04 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 at 10%', diff saved to https://phabricator.wikimedia.org/P9563 and previous config saved to /var/cache/conftool/dbconfig/20191108-090451-jynus.json |
[production] |
| 09:00 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 at 50%', diff saved to https://phabricator.wikimedia.org/P9562 and previous config saved to /var/cache/conftool/dbconfig/20191108-090012-jynus.json |
[production] |
| 08:57 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:52 |
<jynus> |
stop and upgrade db1124 (may create temporary lag on wikireplicas) |
[production] |
| 08:31 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 08:23 |
<elukey> |
restart kafka on kafka-jumbo1001 to test the new openjdk |
[production] |
| 08:07 |
<moritzm> |
installing fribidi security updates on Buster |
[production] |
| 03:03 |
<vgutierrez> |
Switch from nginx to ats-tls on cp3054 - T231627 |
[production] |
| 02:42 |
<vgutierrez> |
Switch from nginx to ats-tls on cp3052 - T231627 |
[production] |
| 01:23 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/GlobalBlocking/: Prevent some extra db queries (duration: 00m 53s) |
[production] |
| 01:14 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/resources/: Use internationalized semicolon separators (T233649) (duration: 00m 53s) |
[production] |
| 01:09 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@11d4ad8]: deploying one more time, hopefully without killing elastic (duration: 03m 04s) |
[production] |
| 01:06 |
<twentyafterfour@deploy1001> |
Started deploy [releng/phatality@11d4ad8]: deploying one more time, hopefully without killing elastic |
[production] |
| 00:44 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/GrowthExperiments/modules/homepage/ext.growthExperiments.Homepage.Logging.js: Fix homepage instrumentation (T237600) (duration: 00m 52s) |
[production] |
| 00:40 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes: Sync DiffEngine changes that were needed to unbreak CI (duration: 00m 55s) |
[production] |
| 00:34 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/resources/: Semicolon should appear after log entries (T237500) (duration: 00m 53s) |
[production] |
| 00:26 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Fix remote API configs for GrowthExperiments (duration: 00m 51s) |
[production] |
| 00:19 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable suggested edits as hidden preference on arwiki, cswiki, kowiki, viwiki (T236968) (duration: 00m 53s) |
[production] |
|
2019-11-07
§
|
| 23:49 |
<foks> |
removing one file for legal compliance |
[production] |
| 23:47 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@62e2870]: revert phatalaty again (duration: 03m 04s) |
[production] |
| 23:44 |
<shdubsh> |
start elasticsearch on logstash1008 |
[production] |
| 23:44 |
<twentyafterfour@deploy1001> |
Started deploy [releng/phatality@62e2870]: revert phatalaty again |
[production] |
| 23:41 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@11d4ad8]: one more time (duration: 03m 00s) |
[production] |
| 23:38 |
<twentyafterfour@deploy1001> |
Started deploy [releng/phatality@11d4ad8]: one more time |
[production] |
| 23:31 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@11d4ad8]: trying again with a longer scap timeout (duration: 03m 02s) |
[production] |
| 23:28 |
<twentyafterfour@deploy1001> |
Started deploy [releng/phatality@11d4ad8]: trying again with a longer scap timeout |
[production] |
| 23:23 |
<twentyafterfour@deploy1001> |
Finished deploy [releng/phatality@62e2870]: revert to previous phatality plugin version (duration: 02m 55s) |
[production] |
| 23:20 |
<twentyafterfour@deploy1001> |
Started deploy [releng/phatality@62e2870]: revert to previous phatality plugin version |
[production] |