|
2020-09-10
§
|
| 13:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:42 |
<moritzm> |
rebooting etherpad1002 (etherpad.wikimedia.org) for kernel update |
[production] |
| 13:24 |
<moritzm> |
installing rake security updates on stretch |
[production] |
| 13:10 |
<ebernhardson> |
delete lldwiki_{content|general} indices from search.svc.{eqiad|codfw}.wmnet:9643 (psi), they should be on 9443 (omega) |
[production] |
| 12:57 |
<klausman> |
Ran puppet-merge to get my dotfiles from https://gerrit.wikimedia.org/r/c/operations/puppet/+/626367 out |
[production] |
| 12:34 |
<moritzm> |
installing firejail updates on maps/thumbor/restbase |
[production] |
| 12:01 |
<moritzm> |
upgrading deployment servers to git 2.20 T262244 |
[production] |
| 11:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P12557 and previous config saved to /var/cache/conftool/dbconfig/20200910-113758-marostegui.json |
[production] |
| 11:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P12556 and previous config saved to /var/cache/conftool/dbconfig/20200910-113426-marostegui.json |
[production] |
| 11:13 |
<matthiasmullie> |
Euro B&C done |
[production] |
| 11:13 |
<moritzm> |
uploaded git 2.20.1-2+deb10u3~wmf1 to stretch-wikimedia/main T262244 |
[production] |
| 11:11 |
<mlitn@deploy1001> |
Synchronized php-1.36.0-wmf.8//extensions/WikimediaEvents/: WikimediaEvents: Enable MediaSearch A/B test (duration: 01m 06s) |
[production] |
| 10:42 |
<duesen_> |
daniel@mwmaint2001:~$ mwscript maintenance/findBadBlobs.php jvwiki --revisions 214173 --mark T262457 |
[production] |
| 10:34 |
<mvolz@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:32 |
<mvolz@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:28 |
<XioNoX> |
move VRRP master to cr2-esams |
[production] |
| 10:21 |
<mvolz@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 09:58 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 09:45 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 09:43 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:42 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:40 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12555 and previous config saved to /var/cache/conftool/dbconfig/20200910-093106-marostegui.json |
[production] |
| 09:26 |
<dcausse> |
creating missing cirrus indices for jawikivoyage T262518 |
[production] |
| 09:24 |
<dcausse> |
creating missing cirrus indices for jawikivoyage T260228 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12554 and previous config saved to /var/cache/conftool/dbconfig/20200910-091335-marostegui.json |
[production] |
| 08:49 |
<jynus@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:47 |
<jynus@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12551 and previous config saved to /var/cache/conftool/dbconfig/20200910-082304-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12550 and previous config saved to /var/cache/conftool/dbconfig/20200910-073107-marostegui.json |
[production] |
| 07:03 |
<elukey> |
resize search-loader vms (+4 vcores +4GB of ram) on Ganeti - T262385 |
[production] |
| 05:29 |
<marostegui> |
Deploy schema change on s3 master - T260476 |
[production] |
| 00:31 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master (duration: 06m 42s) |
[production] |
| 00:24 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master |
[production] |
| 00:23 |
<twentyafterfour> |
done. Phabricator update complete |
[production] |
| 00:23 |
<twentyafterfour> |
applying database migrations to phabricator db |
[production] |
| 00:09 |
<twentyafterfour> |
deploying phabricator update 2020-09-10 https://phabricator.wikimedia.org/project/view/4755/ |
[production] |
|
2020-09-09
§
|
| 23:51 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@55fccc6]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 (duration: 00m 05s) |
[production] |
| 23:51 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@55fccc6]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 |
[production] |
| 23:37 |
<ebernhardson@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/CirrusSearch/includes/Search/InterleavedResultSet.php: Repair passing interleaved search metrics from backend to frontend (duration: 01m 04s) |
[production] |
| 20:13 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:625914 (duration: 01m 03s) |
[production] |
| 20:03 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:626190 T261425 (duration: 01m 03s) |
[production] |
| 20:01 |
<ppchelko@deploy1001> |
Synchronized php-1.36.0-wmf.8/skins/WikimediaApiPortal: Backport gerrit:626044, T261425 (duration: 01m 12s) |
[production] |
| 19:11 |
<jhuneidi@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.8 (duration: 01m 03s) |
[production] |
| 19:10 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.8 |
[production] |
| 18:19 |
<_joe_> |
banning urls ^/api/rest_v1/page/mobile-html-offline-resources/ from varnish caches |
[production] |
| 18:19 |
<Urbanecm> |
Morning B&C window done |
[production] |
| 18:17 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:17 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |