|
2020-07-23
§
|
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:51 |
<jhuneidi@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'echostore' for release 'staging' . |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:21 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:18 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:48 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:45 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:29 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:27 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables (duration: 00m 34s) |
[production] |
| 21:20 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables |
[production] |
| 21:02 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:00 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:59 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:13 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 19:11 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 19:09 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 18:51 |
<ryankemper> |
restarted blazegraph on codfw wdqs2001 |
[production] |
| 18:44 |
<ryankemper> |
Restarted blazegraph on following codfw wdqs nodes: 2007, 2003, and 2002 |
[production] |
| 18:39 |
<Amir1> |
BACC is done |
[production] |
| 18:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:613235|Load WikibaseClient from extension.json file instead of php one (T257437 T256228 T88258)]] (duration: 01m 05s) |
[production] |
| 18:21 |
<mutante> |
testreduce1001 - rm -rf /srv/testreduce and run puppet to re-clone testreduce to it from the scandium branch (T257906) |
[production] |
| 18:13 |
<ryankemper> |
restarted blazegraph on 2001 |
[production] |
| 17:59 |
<ryankemper> |
sudo -E cumin -b 10 'A:wdqs-all and not A:wdqs-test and not P{wdqs1003.eqiad.wmnet} and not P{wdqs2001.codfw.wmnet}' 'sudo systemctl restart wdqs-blazegraph.service' |
[production] |
| 17:53 |
<cdanis> |
❌cdanis@cumin1001.eqiad.wmnet ~ 🕑☕ sudo cumin -b10 'wdqs*' "run-puppet-agent --unless-version 1a4ae81" |
[production] |
| 17:52 |
<cdanis@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=wdqs.*,name=codfw |
[production] |
| 17:35 |
<cdanis@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=wdqs.*,name=codfw |
[production] |
| 17:22 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 16:57 |
<ryankemper@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 16:56 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-reload |
[production] |
| 15:36 |
<urbanecm@deploy1001> |
Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 01m 05s) |
[production] |
| 13:49 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=.* |
[production] |
| 12:29 |
<marostegui> |
Decrease labsdb1009 weight a bit, as it is lagging again. |
[production] |