|
2019-04-16
ยง
|
| 16:53 |
<mutante> |
bast2001 - shutdown -h now - decom'ed (T219492) |
[production] |
| 16:48 |
<mutante> |
puppet node clean bast2001.wikimedia.org ; puppet node deactivate bast2001.wikimedia.org ; it showed up in Icinga again despite running decom cookbook (T219492) |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values --set wmfdebug_enabled=true stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 16:43 |
<jynus> |
upgrading and shutting down db1078 T219115 |
[production] |
| 16:41 |
<jynus> |
disabling notifications on db1078 T219115 |
[production] |
| 16:37 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 52s) |
[production] |
| 15:36 |
<arturo> |
reimaging cloudnet2002-dev because role name change |
[production] |
| 15:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:20 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging --version 0.0.28 -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:16 |
<elukey> |
roll restart kafka on kafka-jumbo100[1-6] to pick up openjdk upgrades |
[production] |
| 14:58 |
<gehel> |
manual data transfer from wdqs1008 to wdqs1009 - T220830 |
[production] |
| 14:56 |
<ema> |
swift-fe-eqiad: nginx reload for new TLS certificate T204245 |
[production] |
| 14:53 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 14:52 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 14:51 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ms-fe1005.eqiad.wmnet |
[production] |
| 14:45 |
<ema> |
test https://gerrit.wikimedia.org/r/504340 on ms-fe1005 T204245 |
[production] |
| 14:30 |
<ema> |
swift-fe-codfw: nginx reload for new TLS certificate T204245 |
[production] |
| 14:22 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 14:21 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 14:20 |
<elukey> |
roll restart of all the druid daemons on druid100[1-6] to pick up new openjdk updates |
[production] |
| 14:17 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ms-fe2005.codfw.wmnet |
[production] |
| 14:07 |
<jijiki> |
Pooling thumbor1001 |
[production] |
| 14:04 |
<ema> |
test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504331/ on ms-fe2005 T204245 |
[production] |
| 14:01 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ms-fe2005.codfw.wmnet |
[production] |
| 14:01 |
<jijiki> |
Depooling thumbor1001 |
[production] |
| 13:58 |
<jijiki> |
Disable puppet on thumbor1001 for ~24h to serve traffic via haproxy - T187765 |
[production] |
| 13:54 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 13:53 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 13:52 |
<jijiki> |
Enable puppet on thumbor* |
[production] |
| 13:42 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 13:41 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 13:39 |
<gehel> |
restetting cookbooks repo on cumin1001 (local changes) |
[production] |
| 13:34 |
<jijiki> |
Disabling puppet on thumbor* to merge 504284 |
[production] |
| 13:13 |
<ema> |
cp-ats: upgrade fifo-log-demux to 0.2 and restart services |
[production] |
| 13:10 |
<ema> |
fifo-log-demux 0.2 uploaded to stretch-wikimedia |
[production] |
| 13:03 |
<arturo> |
T220095 renaming/reimaging labtestcontrol2003 as cloudcontrol2003-dev |
[production] |
| 12:58 |
<moritzm> |
installing ghostscript update on thumbor1001 |
[production] |
| 12:54 |
<gehel> |
cleanup redundant prometheus-elasticsearch units on elasticsearch servers |
[production] |