|
2020-07-23
§
|
| 22:52 |
<mutante> |
stashbot quadruple log test |
[production] |
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:51 |
<bstorm> |
deploying via the default 'latest' tag in the new cluster T211096 |
[paws] |
| 22:51 |
<jhuneidi@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'echostore' for release 'staging' . |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:51 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:50 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:48 |
<bstorm> |
tagged the newbuild tags with "latest" to set sane defaults for all images in the helm chart T211096 |
[paws] |
| 22:21 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:18 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:48 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:45 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:29 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:27 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:21 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables (duration: 00m 34s) |
[production] |
| 21:20 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@c99c626]: airflow: centralize installation specific airflow Variables |
[production] |
| 21:14 |
<bstorm> |
pushing quay.io/wikimedia-paws-prod/nbserve:newbuild to main repo T211096 |
[paws] |
| 21:11 |
<bstorm> |
pushing quay.io/wikimedia-paws-prod/deploy-hook:newbuild to main repo T211096 |
[paws] |
| 21:09 |
<bstorm> |
pushing quay.io/wikimedia-paws-prod/singleuser:newbuild to the main repo T211096 |
[paws] |
| 21:08 |
<bstorm> |
pushing quay.io/wikimedia-paws-prod/paws-hub:newbuild to the main repo T211096 |
[paws] |
| 21:06 |
<bstorm> |
pushing dbproxy docker image for new cluster into main quay.io repo T211096 |
[paws] |
| 21:02 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:00 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:59 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:58 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:13 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 19:11 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 19:09 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 18:51 |
<ryankemper> |
restarted blazegraph on codfw wdqs2001 |
[production] |
| 18:44 |
<ryankemper> |
Restarted blazegraph on following codfw wdqs nodes: 2007, 2003, and 2002 |
[production] |
| 18:39 |
<Amir1> |
BACC is done |
[production] |
| 18:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:613235|Load WikibaseClient from extension.json file instead of php one (T257437 T256228 T88258)]] (duration: 01m 05s) |
[production] |
| 18:21 |
<mutante> |
testreduce1001 - rm -rf /srv/testreduce and run puppet to re-clone testreduce to it from the scandium branch (T257906) |
[production] |
| 18:13 |
<ryankemper> |
restarted blazegraph on 2001 |
[production] |
| 17:59 |
<ryankemper> |
sudo -E cumin -b 10 'A:wdqs-all and not A:wdqs-test and not P{wdqs1003.eqiad.wmnet} and not P{wdqs2001.codfw.wmnet}' 'sudo systemctl restart wdqs-blazegraph.service' |
[production] |
| 17:53 |
<cdanis> |
❌cdanis@cumin1001.eqiad.wmnet ~ 🕑☕ sudo cumin -b10 'wdqs*' "run-puppet-agent --unless-version 1a4ae81" |
[production] |