|
2020-09-10
ยง
|
| 18:00 |
<mutante> |
helium (former backup host) is being removed from ferm rules on all hosts, it was replaced by backup1001 (T260717) |
[production] |
| 17:33 |
<bblack> |
dns servers: upgrading remainder of fleet to gdnsd-3.3.0-1~wmf1 |
[production] |
| 16:50 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:48 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<bblack> |
authdns1001 - upgrade gdnsd to 3.3.0-1~wmf1 |
[production] |
| 16:06 |
<bblack> |
dns4001 - upgrade gdnsd to 3.3.0-1~wmf1 |
[production] |
| 16:04 |
<bblack> |
reprepro: uploaded gdnsd-3.3.0-1~wmf1 - T261340 |
[production] |
| 15:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:30 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:04 |
<volans> |
uploaded cumin_4.0.0 to apt.wikimedia.org buster-wikimedia (no code changes) |
[production] |
| 13:58 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 13:52 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 13:48 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 13:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:42 |
<moritzm> |
rebooting etherpad1002 (etherpad.wikimedia.org) for kernel update |
[production] |
| 13:24 |
<moritzm> |
installing rake security updates on stretch |
[production] |
| 13:10 |
<ebernhardson> |
delete lldwiki_{content|general} indices from search.svc.{eqiad|codfw}.wmnet:9643 (psi), they should be on 9443 (omega) |
[production] |
| 12:57 |
<klausman> |
Ran puppet-merge to get my dotfiles from https://gerrit.wikimedia.org/r/c/operations/puppet/+/626367 out |
[production] |
| 12:34 |
<moritzm> |
installing firejail updates on maps/thumbor/restbase |
[production] |
| 12:01 |
<moritzm> |
upgrading deployment servers to git 2.20 T262244 |
[production] |
| 11:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P12557 and previous config saved to /var/cache/conftool/dbconfig/20200910-113758-marostegui.json |
[production] |
| 11:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P12556 and previous config saved to /var/cache/conftool/dbconfig/20200910-113426-marostegui.json |
[production] |
| 11:13 |
<matthiasmullie> |
Euro B&C done |
[production] |
| 11:13 |
<moritzm> |
uploaded git 2.20.1-2+deb10u3~wmf1 to stretch-wikimedia/main T262244 |
[production] |
| 11:11 |
<mlitn@deploy1001> |
Synchronized php-1.36.0-wmf.8//extensions/WikimediaEvents/: WikimediaEvents: Enable MediaSearch A/B test (duration: 01m 06s) |
[production] |
| 10:42 |
<duesen_> |
daniel@mwmaint2001:~$ mwscript maintenance/findBadBlobs.php jvwiki --revisions 214173 --mark T262457 |
[production] |
| 10:34 |
<mvolz@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:32 |
<mvolz@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:28 |
<XioNoX> |
move VRRP master to cr2-esams |
[production] |
| 10:21 |
<mvolz@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 09:58 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 09:45 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 09:43 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:42 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:40 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12555 and previous config saved to /var/cache/conftool/dbconfig/20200910-093106-marostegui.json |
[production] |
| 09:26 |
<dcausse> |
creating missing cirrus indices for jawikivoyage T262518 |
[production] |
| 09:24 |
<dcausse> |
creating missing cirrus indices for jawikivoyage T260228 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12554 and previous config saved to /var/cache/conftool/dbconfig/20200910-091335-marostegui.json |
[production] |
| 08:49 |
<jynus@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:47 |
<jynus@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12551 and previous config saved to /var/cache/conftool/dbconfig/20200910-082304-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12550 and previous config saved to /var/cache/conftool/dbconfig/20200910-073107-marostegui.json |
[production] |
| 07:03 |
<elukey> |
resize search-loader vms (+4 vcores +4GB of ram) on Ganeti - T262385 |
[production] |
| 05:29 |
<marostegui> |
Deploy schema change on s3 master - T260476 |
[production] |
| 00:31 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master (duration: 06m 42s) |
[production] |
| 00:24 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master |
[production] |
| 00:23 |
<twentyafterfour> |
done. Phabricator update complete |
[production] |
| 00:23 |
<twentyafterfour> |
applying database migrations to phabricator db |
[production] |