3701-3750 of 10000 results (72ms)
2020-03-05 ยง
20:37 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: MachineVision: Update label blacklist (duration: 00m 59s) [production]
20:20 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:18 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:18 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:17 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:46 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: re-sync for bug 236104 (duration: 00m 56s) [production]
19:45 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:577325|Switch GrowthExperiments topic search to ORES (T240517)]] (duration: 00m 58s) [production]
19:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:39 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:10 <ebernhardson@deploy1001> Synchronized wmf-config/SearchSettingsForWikibase.php: (no justification provided) (duration: 00m 57s) [production]
18:33 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:32 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
18:30 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:30 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
18:28 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:27 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
18:27 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
18:26 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
18:11 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:09 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
18:05 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:02 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
17:50 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
17:50 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
17:44 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:41 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
17:40 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:38 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
17:37 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I8f0d82164, Iaac7cbfbb9 (no-op) (duration: 00m 59s) [production]
17:32 <elukey> run homer on cumin1001 to apply https://gerrit.wikimedia.org/r/576873 on cr1/cr2-eqiad [production]
17:27 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
17:27 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
17:24 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
17:24 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
17:19 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:15 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
17:14 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
17:14 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
17:11 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:09 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
16:58 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
16:58 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
16:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1078 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10631 and previous config saved to /var/cache/conftool/dbconfig/20200305-165555-marostegui.json [production]
16:55 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
16:54 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
16:50 <krinkle@deploy1001> Synchronized dblists/: I22a3c2a82f7be4a (duration: 00m 57s) [production]
16:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1078 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10630 and previous config saved to /var/cache/conftool/dbconfig/20200305-164319-marostegui.json [production]
16:22 <marostegui> Restart tendril/dbtree database [production]
16:18 <_joe_> repooling mw1394 [production]
16:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1078 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10629 and previous config saved to /var/cache/conftool/dbconfig/20200305-161222-marostegui.json [production]