|
2020-09-10
§
|
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12554 and previous config saved to /var/cache/conftool/dbconfig/20200910-091335-marostegui.json |
[production] |
| 08:49 |
<jynus@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:47 |
<jynus@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12551 and previous config saved to /var/cache/conftool/dbconfig/20200910-082304-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2014 after cloning es2026', diff saved to https://phabricator.wikimedia.org/P12550 and previous config saved to /var/cache/conftool/dbconfig/20200910-073107-marostegui.json |
[production] |
| 07:03 |
<elukey> |
resize search-loader vms (+4 vcores +4GB of ram) on Ganeti - T262385 |
[production] |
| 05:29 |
<marostegui> |
Deploy schema change on s3 master - T260476 |
[production] |
| 00:31 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master (duration: 06m 42s) |
[production] |
| 00:24 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master |
[production] |
| 00:23 |
<twentyafterfour> |
done. Phabricator update complete |
[production] |
| 00:23 |
<twentyafterfour> |
applying database migrations to phabricator db |
[production] |
| 00:09 |
<twentyafterfour> |
deploying phabricator update 2020-09-10 https://phabricator.wikimedia.org/project/view/4755/ |
[production] |
|
2020-09-09
§
|
| 23:51 |
<dpifke@deploy1001> |
Finished deploy [performance/arc-lamp@55fccc6]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 (duration: 00m 05s) |
[production] |
| 23:51 |
<dpifke@deploy1001> |
Started deploy [performance/arc-lamp@55fccc6]: Deploying https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/622915 |
[production] |
| 23:37 |
<ebernhardson@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/CirrusSearch/includes/Search/InterleavedResultSet.php: Repair passing interleaved search metrics from backend to frontend (duration: 01m 04s) |
[production] |
| 20:13 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:625914 (duration: 01m 03s) |
[production] |
| 20:03 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:626190 T261425 (duration: 01m 03s) |
[production] |
| 20:01 |
<ppchelko@deploy1001> |
Synchronized php-1.36.0-wmf.8/skins/WikimediaApiPortal: Backport gerrit:626044, T261425 (duration: 01m 12s) |
[production] |
| 19:11 |
<jhuneidi@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.8 (duration: 01m 03s) |
[production] |
| 19:10 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.8 |
[production] |
| 18:19 |
<_joe_> |
banning urls ^/api/rest_v1/page/mobile-html-offline-resources/ from varnish caches |
[production] |
| 18:19 |
<Urbanecm> |
Morning B&C window done |
[production] |
| 18:17 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:17 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 18:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: b226330c1b3bd3dae113e375e2afb4d6af774cde: Enable $wgAllowCrossOrigin on all wikis (T262425) (duration: 01m 04s) |
[production] |
| 18:15 |
<urbanecm@deploy1001> |
sync-file aborted: (no justification provided) (duration: 00m 01s) |
[production] |
| 18:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 18:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 85e36ae12e7467a559e3d52c58cc3a71ffd09ded: Enable MediaWiki client errors on commonswiki and metawiki (T255585) (duration: 01m 06s) |
[production] |
| 18:10 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 18:02 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, feed timeout (duration: 02m 55s) |
[production] |
| 17:59 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, feed timeout |
[production] |
| 17:59 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, feed timeout (duration: 06m 47s) |
[production] |
| 17:52 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, feed timeout |
[production] |
| 17:52 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, take 2 (duration: 09m 38s) |
[production] |
| 17:42 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@b90472d]: Require mobile-html 1.2.2 T262437, take 2 |
[production] |
| 17:41 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@dc3b955]: Require mobile-html 1.2.2 T262437 (duration: 06m 00s) |
[production] |
| 17:35 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@dc3b955]: Require mobile-html 1.2.2 T262437 |
[production] |
| 17:29 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 17:29 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 17:28 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:25 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 17:24 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 17:22 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:18 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:15 |
<marostegui> |
Stop mysql on db2125 for on-site maintenance T260670 |
[production] |
| 16:10 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@e120c6c]: Deploying r20200909 tag (T262323, T144111) [take 3] (duration: 00m 11s) |
[production] |
| 16:10 |
<bd808@deploy1001> |
Started deploy [striker/deploy@e120c6c]: Deploying r20200909 tag (T262323, T144111) [take 3] |
[production] |
| 16:10 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 16:10 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |