|
2020-12-08
ยง
|
| 13:28 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) |
[production] |
| 13:28 |
<hnowlan@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 13:19 |
<moritzm> |
installing xen security updates (client side libs only) |
[production] |
| 12:57 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 12:45 |
<akosiaris@cumin1001> |
conftool action : set/pooled=inactive; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:43 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: service=apertium,name=kubernetes1001* |
[production] |
| 12:35 |
<dcausse> |
EU B&C done |
[production] |
| 12:34 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:32 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:30 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) |
[production] |
| 12:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 12:28 |
<dcausse@deploy1001> |
Synchronized php-1.36.0-wmf.20/extensions/WikimediaEvents/: T266027: [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 00s) |
[production] |
| 12:27 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:23 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:23 |
<akosiaris> |
pooling of apertium kubernetes service, take #2 |
[production] |
| 12:08 |
<akosiaris> |
rollback pooling of apertium kubernetes service. |
[production] |
| 12:08 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:03 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:01 |
<akosiaris@cumin1001> |
conftool action : set/weight=1; selector: service=apertium,name=kubernetes.* |
[production] |
| 12:01 |
<moritzm> |
installing jasper security updates |
[production] |
| 11:51 |
<moritzm> |
uploaded jasper 1.900.1-debian1-2.4+deb8u6+wmf1 to apt.wikimedia.org / jessie-wikimedia |
[production] |
| 11:45 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 11:44 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 11:43 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 11:43 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 11:42 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 11:42 |
<akosiaris> |
add a TLS enabled release for apertium |
[production] |
| 11:42 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 11:41 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 11:41 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 10:45 |
<ema> |
A:cp remove libvmod-tbf -- see https://gerrit.wikimedia.org/r/c/operations/puppet/+/623583/ |
[production] |
| 10:33 |
<ema> |
cp4032: restart ats-{tls,be} |
[production] |
| 10:29 |
<godog> |
reboot ms-be1021 - T268435 |
[production] |
| 10:13 |
<ema> |
cp3054: upgrade varnish back to 6.0.7-1wm1 T264398 |
[production] |
| 09:56 |
<ema> |
cp3054: downgrade varnish to 5.2.1-1wm1 T264398 |
[production] |
| 09:06 |
<effie> |
upgrade scap to 3.16 everywhere - T268634 |
[production] |
| 08:16 |
<godog> |
upgrade hw raid firmware on ms-be1032 and reboot - T269624 |
[production] |
| 08:09 |
<godog> |
power reset ms-be1032 |
[production] |
| 03:25 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 03:16 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 03:05 |
<ryankemper@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 03:03 |
<ryankemper@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 02:30 |
<ejegg> |
deployed queue consumers for new Adyen capture jobs |
[production] |
| 01:57 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 01:56 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 01:55 |
<ryankemper@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 01:54 |
<ryankemper@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 01:53 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 01:52 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 01:51 |
<ryankemper> |
T246345 Brought two new public wdqs nodes `wdqs101[2,3]` into service: `sudo confctl select 'name=wdqs1012.eqiad.wmnet|wdqs1013.eqiad.wmnet' set/pooled=yes:weight=10` |
[production] |