|
2020-11-19
§
|
| 14:45 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:44 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:43 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:41 |
<marostegui> |
Sanitize enwiki on clouddb1013 T267090 |
[production] |
| 14:39 |
<volans@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:36 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:29 |
<moritzm> |
rolling restart of app server canaries to pick up latest sec updates |
[production] |
| 14:21 |
<moritzm> |
installing krb5 security updates on stretch |
[production] |
| 14:02 |
<bblack> |
authdns2001: upgrade gdnsd to 3.4.0 |
[production] |
| 13:45 |
<XioNoX> |
push current state of audited cloud-in4 filter - T264993 |
[production] |
| 13:42 |
<moritzm> |
removing stray wireshark 2.2.6 wireshark libs on Stretch |
[production] |
| 13:32 |
<moritzm> |
installing wireshark security updates |
[production] |
| 13:30 |
<bblack> |
dns4002: upgrade gdnsd to 3.4.0 |
[production] |
| 13:28 |
<bblack> |
reprepro: updated buster-wikimedia gdnsd package to 3.4.0-1~wmf1 |
[production] |
| 12:43 |
<moritzm> |
installing libproxy security updates on stretch |
[production] |
| 12:38 |
<marostegui> |
Stop mysql on db1106 to clone clouddb1013 and clouddb1017 T267090 |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 T267090', diff saved to https://phabricator.wikimedia.org/P13334 and previous config saved to /var/cache/conftool/dbconfig/20201119-122459-marostegui.json |
[production] |
| 12:00 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 11:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:46 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 11:44 |
<moritzm> |
installing Java security updates on Hadoop/Kafka Jumbo hosts |
[production] |
| 11:42 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:40 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 11:33 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:00 |
<Urbanecm> |
Start of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=ruwiki; T246539) |
[production] |
| 10:28 |
<marostegui> |
Restart mysql on db1115, tendril and dbtree will be down for a few minutes |
[production] |
| 09:40 |
<marostegui> |
Stop mysql on db1124:3311 to clone clouddb1013 and clouddb1017, there will be lag on s1 on wikireplicas - T267090 |
[production] |
| 09:29 |
<moritzm> |
upgrading serpens to Buster |
[production] |
| 09:26 |
<XioNoX> |
eqiad row C: move Ganeti/LVS interfaces to individual terms |
[production] |
| 09:07 |
<elukey> |
restart kafka daemons on kafka-jumbo1001 for openjdk upgrades (canary) |
[production] |
| 08:56 |
<effie> |
disable puppet on mw canaries to merge 641816 |
[production] |
| 08:55 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) |
[production] |
| 08:49 |
<elukey> |
restart hadoop daemons on analytics1058 for openjdk upgrades (canary) |
[production] |
| 08:25 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-masters |
[production] |
| 08:19 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) |
[production] |
| 08:19 |
<XioNoX> |
eqiad row C: standardize interfaces config |
[production] |
| 07:55 |
<XioNoX> |
eqiad row D: move Ganeti/LVS interfaces to individual terms |
[production] |
| 07:47 |
<XioNoX> |
eqiad row D: standardize interfaces config |
[production] |
| 07:22 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-masters |
[production] |
| 07:21 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) |
[production] |
| 07:05 |
<elukey> |
roll restart java daemons on Hadoop test for openjdk upgrades |
[production] |
| 07:05 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 06:22 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:21 |
<marostegui> |
Remove es1014 from tendril and zarcillo T268102 |
[production] |
| 06:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:08 |
<marostegui> |
Stop mysql on db1125:3316 to clone clouddb1015 and clouddb1019, there will be lag on s6 on wikireplicas - T267090 |
[production] |
| 02:41 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 01:30 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |