|
2019-09-13
ยง
|
| 23:42 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:06 |
<gehel> |
re-enable puppet on maps - T232817 |
[production] |
| 20:23 |
<chaomodus> |
restarting netbox1001.wikimedia.org |
[production] |
| 20:00 |
<twentyafterfour> |
hotfixing T232600 due to severity of the bug and relative safety of the fix (if this breaks, yell at James_F who twisted my arm and made me do it) |
[production] |
| 19:54 |
<urandom> |
bootstrapping Cassandra, restbase2009-c -- T224553 |
[production] |
| 17:24 |
<urandom> |
bootstrapping Cassandra, restbase2009-b -- T224553 |
[production] |
| 16:10 |
<XioNoX> |
fix bgp group netflow on cr2-codfw |
[production] |
| 15:47 |
<urandom> |
bootstrapping Cassandra, restbase2009-a -- T224553 |
[production] |
| 15:43 |
<effie> |
reverting live hacks on mw1348 |
[production] |
| 15:34 |
<hashar@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Disable adhoc core dump logging - T232613 (duration: 01m 04s) |
[production] |
| 15:14 |
<akosiaris> |
upload apertium-dan_0.6.0-1+wmf3 apertium-nno_1.0.0-1+wmf1 apertium-nob_1.0.0-2+wmf1 apertium-swe_0.8.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia T218184 |
[production] |
| 15:11 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:11 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:08 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:08 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:02 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Add more log and context for T232613 logging - T232613 (duration: 01m 04s) |
[production] |
| 15:02 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:02 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:51 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:51 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:37 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:36 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:30 |
<akosiaris@> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:30 |
<moritzm> |
installing cups security update on buster (only client-side libs installed) |
[production] |
| 14:22 |
<moritzm> |
installing bzip2 update from Buster 10.1 point release |
[production] |
| 14:18 |
<moritzm> |
installing reportbug update from Buster 10.1 point release |
[production] |
| 14:14 |
<akosiaris@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:05 |
<akosiaris@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:57 |
<oblivian@deploy1001> |
Synchronized wmf-config/logging.php: unbreak mediawiki logging on scandium (duration: 01m 04s) |
[production] |
| 13:28 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:27 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:21 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:20 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:19 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 12:56 |
<_joe_> |
banning more urls on maps1003 |
[production] |
| 12:37 |
<_joe_> |
temp ban of class of urls on maps1003 nginx |
[production] |
| 12:14 |
<jbond42> |
add timing information to maps1003 access logs |
[production] |
| 11:39 |
<jbond42> |
enable access logs on maps1003 |
[production] |
| 11:38 |
<_joe_> |
manually raising the worker heap limit to 600 MB on kartotherian on maps1003 |
[production] |
| 11:11 |
<elukey> |
reboot an-conf100* (Analytics Zookeeper nodes - not yet in production) for kernel upgrades |
[production] |
| 11:10 |
<elukey> |
reboot an-tool1007 (runs turnilo) for kernel upgrades |
[production] |
| 11:08 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:08 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:05 |
<godog> |
silence kartotherian pages for 2h, known issue |
[production] |
| 10:47 |
<vgutierrez> |
rebooting acmechief-test servers to catch up latest kernel upgrades |
[production] |
| 10:42 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 10:41 |
<moritzm> |
reimage restbase2009 to stretch T224553 |
[production] |
| 10:38 |
<moritzm> |
repool restbase1018 after reimage to stretch and completed Cassandra bootstrap |
[production] |
| 10:36 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 10:36 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |