|
2020-02-27
ยง
|
| 20:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:26 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:14 |
<effie> |
pool mw1262 |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:05 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 19:46 |
<mutante> |
Welcome new deployers Thalia Chan, Moriel Schottlender and Dayllan Maza (Anti-Harrassment-Tools team) |
[production] |
| 19:38 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:26 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:21 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: once more for good measure (duration: 01m 03s) |
[production] |
| 19:20 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:574634|Enable articletopic: search keyword in CirrusSearch (T240559)]] (duration: 01m 05s) |
[production] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<effie> |
depool mw1262 |
[production] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<mutante> |
ganeti2001 - removing VM apt2001 to re-create it after IP change |
[production] |
| 19:13 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 (duration: 00m 07s) |
[production] |
| 19:12 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 |
[production] |
| 19:06 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:51 |
<elukey> |
upgrade prometheus-mcrouter-exporter to 0.1.0+git20200227-1 on hosts |
[production] |
| 18:48 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 (duration: 10m 11s) |
[production] |
| 18:43 |
<mutante> |
adding parse2* machines to puppet |
[production] |
| 18:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 |
[production] |
| 18:31 |
<volans> |
restarting icinga on icinga1001, command file randomly discarding commands |
[production] |
| 18:21 |
<addshore> |
END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) (will do 2 more passes tomorrow) |
[production] |
| 18:20 |
<elukey> |
upload prometheus-mcrouter-exporter 0.1.0+git20200227-1 to stretch-wikimedia |
[production] |
| 17:52 |
<addshore> |
resume item migration script at Q50 million T219123 (batch size of 100, 1s sleep) |
[production] |
| 17:49 |
<ebernhardson> |
delete commonswiki_file_1582685980 from cloudelastic-chi, reindex failed and commonswiki_file_first is still primary |
[production] |
| 17:41 |
<effie> |
enable puppet on thumbor* |
[production] |
| 17:40 |
<effie> |
stop and mask all nginx on thumbor* |
[production] |
| 17:34 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:34 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:33 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:31 |
<addshore> |
START warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) |
[production] |
| 17:31 |
<vgutierrez> |
(from 17:03) reimage lvs5003 with buster - T245984 |
[production] |
| 17:30 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:30 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 at 20% T232446', diff saved to https://phabricator.wikimedia.org/P10547 and previous config saved to /var/cache/conftool/dbconfig/20200227-173017-jynus.json |
[production] |
| 17:20 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q6M for the new term store for clients (was Q4M) + warm db1126 caches (T219123) cache bust (duration: 01m 04s) |
[production] |
| 17:19 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q6M for the new term store for clients (was Q4M) + warm db1126 caches (T219123) (duration: 01m 04s) |
[production] |
| 17:18 |
<addshore> |
(relog FROM 5:11) END warming wikidata term cache on db1126 for Q4-6 million T219123 (pass2) |
[production] |
| 16:59 |
<bd808> |
Disabled new account creation on wikitech via horrible TitleBlacklist hack. |
[production] |
| 16:55 |
<vgutierrez> |
re-enable BGP in lvs4005 - T245984 |
[production] |