|
2019-04-16
ยง
|
| 17:36 |
<arturo> |
toolforge k8s reallocation (from nova-network to neutron) is causing troubles with IRC bots, expect missing entries in the SAL |
[production] |
| 17:28 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:28 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:27 |
<andrewbogott> |
restarting rabbitmq on cloudcontrol1003 |
[production] |
| 17:26 |
<cdanis@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=eqiad,name=mw1280.eqiad.wmnet,cluster=api_appserver |
[production] |
| 17:25 |
<arturo> |
rebooted cloudnet1003 |
[production] |
| 17:24 |
<gehel> |
force initialization of unassigned shards on elasticsearch eqiad |
[production] |
| 17:16 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:504374|no-op preparatory change (T221108)]] (duration: 00m 52s) |
[production] |
| 16:54 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwmaint1002:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=testwikidatawiki --config-format=wgConf | tee T221108.php |
[production] |
| 16:53 |
<mutante> |
bast2001 - shutdown -h now - decom'ed (T219492) |
[production] |
| 16:48 |
<mutante> |
puppet node clean bast2001.wikimedia.org ; puppet node deactivate bast2001.wikimedia.org ; it showed up in Icinga again despite running decom cookbook (T219492) |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 16:47 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values --set wmfdebug_enabled=true stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 16:44 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 16:43 |
<jynus> |
upgrading and shutting down db1078 T219115 |
[production] |
| 16:41 |
<jynus> |
disabling notifications on db1078 T219115 |
[production] |
| 16:37 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 52s) |
[production] |
| 15:36 |
<arturo> |
reimaging cloudnet2002-dev because role name change |
[production] |
| 15:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:21 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:20 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging --version 0.0.28 -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:19 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 15:18 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 15:16 |
<elukey> |
roll restart kafka on kafka-jumbo100[1-6] to pick up openjdk upgrades |
[production] |
| 14:58 |
<gehel> |
manual data transfer from wdqs1008 to wdqs1009 - T220830 |
[production] |
| 14:56 |
<ema> |
swift-fe-eqiad: nginx reload for new TLS certificate T204245 |
[production] |
| 14:53 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 14:52 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 14:51 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ms-fe1005.eqiad.wmnet |
[production] |
| 14:45 |
<ema> |
test https://gerrit.wikimedia.org/r/504340 on ms-fe1005 T204245 |
[production] |
| 14:30 |
<ema> |
swift-fe-codfw: nginx reload for new TLS certificate T204245 |
[production] |
| 14:22 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 14:21 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 14:20 |
<elukey> |
roll restart of all the druid daemons on druid100[1-6] to pick up new openjdk updates |
[production] |
| 14:17 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ms-fe2005.codfw.wmnet |
[production] |
| 14:07 |
<jijiki> |
Pooling thumbor1001 |
[production] |
| 14:04 |
<ema> |
test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/504331/ on ms-fe2005 T204245 |
[production] |
| 14:01 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ms-fe2005.codfw.wmnet |
[production] |
| 14:01 |
<jijiki> |
Depooling thumbor1001 |
[production] |
| 13:58 |
<jijiki> |
Disable puppet on thumbor1001 for ~24h to serve traffic via haproxy - T187765 |
[production] |
| 13:54 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 13:53 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 13:52 |
<jijiki> |
Enable puppet on thumbor* |
[production] |