|
2020-02-27
ยง
|
| 20:57 |
<bstorm_> |
upgrading toollabs-webservice to stretch-toolsbeta version for jdk8:testing image only |
[tools] |
| 20:32 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:26 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:19 |
<jeh> |
update elasticsearch VPS security group to allow toolsbeta-elastic7-1 access on tcp 80 T236606 |
[tools] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:14 |
<effie> |
pool mw1262 |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:05 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 19:46 |
<mutante> |
Welcome new deployers Thalia Chan, Moriel Schottlender and Dayllan Maza (Anti-Harrassment-Tools team) |
[production] |
| 19:38 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:26 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:21 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: once more for good measure (duration: 01m 03s) |
[production] |
| 19:20 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:574634|Enable articletopic: search keyword in CirrusSearch (T240559)]] (duration: 01m 05s) |
[production] |
| 19:18 |
<bstorm_> |
upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing |
[toolsbeta] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<effie> |
depool mw1262 |
[production] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<mutante> |
ganeti2001 - removing VM apt2001 to re-create it after IP change |
[production] |
| 19:13 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 (duration: 00m 07s) |
[production] |
| 19:12 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 |
[production] |
| 19:06 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:53 |
<bstorm_> |
hard rebooted a rather stuck tools-sgecron-01 |
[tools] |
| 18:51 |
<elukey> |
upgrade prometheus-mcrouter-exporter to 0.1.0+git20200227-1 on hosts |
[production] |
| 18:48 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 (duration: 10m 11s) |
[production] |
| 18:43 |
<mutante> |
adding parse2* machines to puppet |
[production] |
| 18:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 |
[production] |
| 18:31 |
<volans> |
restarting icinga on icinga1001, command file randomly discarding commands |
[production] |
| 18:21 |
<addshore> |
END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) (will do 2 more passes tomorrow) |
[production] |
| 18:20 |
<bd808> |
Building tools-k8s-worker-[36-55] |
[tools] |
| 18:20 |
<elukey> |
upload prometheus-mcrouter-exporter 0.1.0+git20200227-1 to stretch-wikimedia |
[production] |
| 18:11 |
<James_F> |
Reset mediawiki-quibble-composer-mysql-php72-docker to nominal T246358 |
[releng] |
| 17:59 |
<James_F> |
Updating mediawiki-quibble-composer-mysql-php72-docker to test probably-breaking requirement that error logs are empty T246358 |
[releng] |
| 17:56 |
<bd808> |
Deleted instances tools-worker-10[21-40] |
[tools] |
| 17:52 |
<addshore> |
resume item migration script at Q50 million T219123 (batch size of 100, 1s sleep) |
[production] |
| 17:49 |
<ebernhardson> |
delete commonswiki_file_1582685980 from cloudelastic-chi, reindex failed and commonswiki_file_first is still primary |
[production] |
| 17:41 |
<effie> |
enable puppet on thumbor* |
[production] |
| 17:40 |
<effie> |
stop and mask all nginx on thumbor* |
[production] |
| 17:34 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:34 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:33 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |