201-250 of 10000 results (65ms)
2020-02-27 ยง
21:19 <jforrester@deploy1001> Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
21:16 <milimetric@deploy1001> Finished deploy [analytics/aqs/deploy@c70b338]: AQS: Minor fix (duration: 02m 30s) [production]
21:14 <jforrester@deploy1001> Synchronized multiversion/MWWikiversions.php: Drop references to four dblists to canaries too (duration: 00m 55s) [production]
21:13 <milimetric@deploy1001> Started deploy [analytics/aqs/deploy@c70b338]: AQS: Minor fix [production]
21:13 <jforrester@deploy1001> Synchronized dblists/: Add back the deleted dblists to make the canaries quiet (duration: 00m 56s) [production]
21:11 <jforrester@deploy1001> sync-file aborted: Drop references to four dblists (duration: 00m 05s) [production]
21:11 <jforrester@deploy1001> Synchronized multiversion/MWWikiversions.php: Drop references to four dblists (duration: 00m 35s) [production]
21:10 <jforrester@deploy1001> Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
21:07 <jforrester@deploy1001> Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) [production]
21:04 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 00m 56s) [production]
21:02 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Merge wgMinervaCustomLogos into wgLogos (duration: 00m 57s) [production]
20:32 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:30 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:26 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:26 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:22 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:22 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:21 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:21 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:16 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:16 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:14 <effie> pool mw1262 [production]
20:07 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:07 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:05 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.21 refs T233869 [production]
20:00 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:00 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
19:46 <mutante> Welcome new deployers Thalia Chan, Moriel Schottlender and Dayllan Maza (Anti-Harrassment-Tools team) [production]
19:38 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:26 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:21 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: once more for good measure (duration: 01m 03s) [production]
19:20 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:574634|Enable articletopic: search keyword in CirrusSearch (T240559)]] (duration: 01m 05s) [production]
19:17 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
19:17 <effie> depool mw1262 [production]
19:17 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:17 <mutante> ganeti2001 - removing VM apt2001 to re-create it after IP change [production]
19:13 <milimetric@deploy1001> Finished deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 (duration: 00m 07s) [production]
19:12 <milimetric@deploy1001> Started deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 [production]
19:06 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
18:51 <elukey> upgrade prometheus-mcrouter-exporter to 0.1.0+git20200227-1 on hosts [production]
18:48 <milimetric@deploy1001> Finished deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 (duration: 10m 11s) [production]
18:43 <mutante> adding parse2* machines to puppet [production]
18:37 <milimetric@deploy1001> Started deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 [production]
18:31 <volans> restarting icinga on icinga1001, command file randomly discarding commands [production]
18:21 <addshore> END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) (will do 2 more passes tomorrow) [production]
18:20 <elukey> upload prometheus-mcrouter-exporter 0.1.0+git20200227-1 to stretch-wikimedia [production]
17:52 <addshore> resume item migration script at Q50 million T219123 (batch size of 100, 1s sleep) [production]
17:49 <ebernhardson> delete commonswiki_file_1582685980 from cloudelastic-chi, reindex failed and commonswiki_file_first is still primary [production]
17:41 <effie> enable puppet on thumbor* [production]
17:40 <effie> stop and mask all nginx on thumbor* [production]