|
2020-02-27
ยง
|
| 21:40 |
<jforrester@deploy1001> |
Synchronized dblists/: Re-establish dblists everywhere (duration: 00m 33s) |
[production] |
| 21:39 |
<jforrester@deploy1001> |
scap failed: average error rate on 11/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 21:25 |
<jforrester@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: Touch the dblists list (duration: 00m 56s) |
[production] |
| 21:24 |
<jforrester@deploy1001> |
Synchronized dblists/: Again, this time without blanked files (duration: 00m 56s) |
[production] |
| 21:22 |
<jforrester@deploy1001> |
Scap failed!: 8/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 21:19 |
<jforrester@deploy1001> |
Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 21:19 |
<jforrester@deploy1001> |
Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 21:16 |
<milimetric@deploy1001> |
Finished deploy [analytics/aqs/deploy@c70b338]: AQS: Minor fix (duration: 02m 30s) |
[production] |
| 21:14 |
<jforrester@deploy1001> |
Synchronized multiversion/MWWikiversions.php: Drop references to four dblists to canaries too (duration: 00m 55s) |
[production] |
| 21:13 |
<milimetric@deploy1001> |
Started deploy [analytics/aqs/deploy@c70b338]: AQS: Minor fix |
[production] |
| 21:13 |
<jforrester@deploy1001> |
Synchronized dblists/: Add back the deleted dblists to make the canaries quiet (duration: 00m 56s) |
[production] |
| 21:11 |
<jforrester@deploy1001> |
sync-file aborted: Drop references to four dblists (duration: 00m 05s) |
[production] |
| 21:11 |
<jforrester@deploy1001> |
Synchronized multiversion/MWWikiversions.php: Drop references to four dblists (duration: 00m 35s) |
[production] |
| 21:10 |
<jforrester@deploy1001> |
Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 21:07 |
<jforrester@deploy1001> |
Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 21:04 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 00m 56s) |
[production] |
| 21:02 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Merge wgMinervaCustomLogos into wgLogos (duration: 00m 57s) |
[production] |
| 20:32 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:26 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:22 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:21 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:16 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:14 |
<effie> |
pool mw1262 |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:07 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:05 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 19:46 |
<mutante> |
Welcome new deployers Thalia Chan, Moriel Schottlender and Dayllan Maza (Anti-Harrassment-Tools team) |
[production] |
| 19:38 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:26 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:21 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: once more for good measure (duration: 01m 03s) |
[production] |
| 19:20 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:574634|Enable articletopic: search keyword in CirrusSearch (T240559)]] (duration: 01m 05s) |
[production] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<effie> |
depool mw1262 |
[production] |
| 19:17 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 19:17 |
<mutante> |
ganeti2001 - removing VM apt2001 to re-create it after IP change |
[production] |
| 19:13 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 (duration: 00m 07s) |
[production] |
| 19:12 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 |
[production] |
| 19:06 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 18:51 |
<elukey> |
upgrade prometheus-mcrouter-exporter to 0.1.0+git20200227-1 on hosts |
[production] |
| 18:48 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 (duration: 10m 11s) |
[production] |
| 18:43 |
<mutante> |
adding parse2* machines to puppet |
[production] |
| 18:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 |
[production] |
| 18:31 |
<volans> |
restarting icinga on icinga1001, command file randomly discarding commands |
[production] |