4401-4450 of 10000 results (73ms)
2020-02-27 ยง
20:16 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:16 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:14 <effie> pool mw1262 [production]
20:07 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:07 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:05 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.21 refs T233869 [production]
20:00 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:00 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [production]
19:46 <mutante> Welcome new deployers Thalia Chan, Moriel Schottlender and Dayllan Maza (Anti-Harrassment-Tools team) [production]
19:38 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:26 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:21 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: once more for good measure (duration: 01m 03s) [production]
19:20 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:574634|Enable articletopic: search keyword in CirrusSearch (T240559)]] (duration: 01m 05s) [production]
19:17 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
19:17 <effie> depool mw1262 [production]
19:17 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:17 <mutante> ganeti2001 - removing VM apt2001 to re-create it after IP change [production]
19:13 <milimetric@deploy1001> Finished deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 (duration: 00m 07s) [production]
19:12 <milimetric@deploy1001> Started deploy [analytics/refinery@357ff5c] (thin): Refinery using 0.0.115 [production]
19:06 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
18:51 <elukey> upgrade prometheus-mcrouter-exporter to 0.1.0+git20200227-1 on hosts [production]
18:48 <milimetric@deploy1001> Finished deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 (duration: 10m 11s) [production]
18:43 <mutante> adding parse2* machines to puppet [production]
18:37 <milimetric@deploy1001> Started deploy [analytics/refinery@357ff5c]: Refinery using 0.0.115 [production]
18:31 <volans> restarting icinga on icinga1001, command file randomly discarding commands [production]
18:21 <addshore> END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) (will do 2 more passes tomorrow) [production]
18:20 <elukey> upload prometheus-mcrouter-exporter 0.1.0+git20200227-1 to stretch-wikimedia [production]
17:52 <addshore> resume item migration script at Q50 million T219123 (batch size of 100, 1s sleep) [production]
17:49 <ebernhardson> delete commonswiki_file_1582685980 from cloudelastic-chi, reindex failed and commonswiki_file_first is still primary [production]
17:41 <effie> enable puppet on thumbor* [production]
17:40 <effie> stop and mask all nginx on thumbor* [production]
17:34 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:34 <volans@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:33 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:31 <addshore> START warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1) [production]
17:31 <vgutierrez> (from 17:03) reimage lvs5003 with buster - T245984 [production]
17:30 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:30 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1087 at 20% T232446', diff saved to https://phabricator.wikimedia.org/P10547 and previous config saved to /var/cache/conftool/dbconfig/20200227-173017-jynus.json [production]
17:20 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Reading up to Q6M for the new term store for clients (was Q4M) + warm db1126 caches (T219123) cache bust (duration: 01m 04s) [production]
17:19 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Reading up to Q6M for the new term store for clients (was Q4M) + warm db1126 caches (T219123) (duration: 01m 04s) [production]
17:18 <addshore> (relog FROM 5:11) END warming wikidata term cache on db1126 for Q4-6 million T219123 (pass2) [production]
16:59 <bd808> Disabled new account creation on wikitech via horrible TitleBlacklist hack. [production]
16:55 <vgutierrez> re-enable BGP in lvs4005 - T245984 [production]
16:50 <volans> temporarily decommented external check for icinga2001. Restarting Icinga on icinga2001 [production]
16:49 <addshore> START warming wikidata term cache on db1126 for Q4-6 million T219123 (pass2) [production]
16:49 <addshore> END warming wikidata term cache on db1126 for Q4-6 million T219123 (pass1) [production]
16:39 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:36 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:27 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
16:24 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]