|
2019-07-19
ยง
|
| 15:54 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:40 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 15:27 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 15:26 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 15:22 |
<fsero> |
deploy coredns in staging T226516 |
[production] |
| 15:03 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 14:42 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/Collection/Collection.php: 90eed0fad / T87899 (duration: 00m 54s) |
[production] |
| 14:35 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/Collection/Collection.php: 66ce154d7d734209c76a62cf / T87899 (duration: 00m 56s) |
[production] |
| 14:29 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@71e62ee]: better exception handling for misc dumps (duration: 00m 03s) |
[production] |
| 14:29 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@71e62ee]: better exception handling for misc dumps |
[production] |
| 14:28 |
<Krinkle> |
krinkle@deploy1001: Untracked file found in php-1.34-wmf.13 |
[production] |
| 14:28 |
<Krinkle> |
krinkle@deploy1001: extensions/CheckUser is dirty in php-1.34-wmf.13 and php-1.34-wmf.14 |
[production] |
| 13:30 |
<tarrow@> |
helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . |
[production] |
| 13:04 |
<moritzm> |
installing bzip2 security updates on jessie |
[production] |
| 12:28 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) |
[production] |
| 10:56 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 10:55 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 10:53 |
<root@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 10:53 |
<fsero> |
deploying calico from helmfile in staging T227775 |
[production] |
| 10:35 |
<jijiki> |
enable puppet on jobrunners |
[production] |
| 10:26 |
<jijiki> |
disable puppet on jobrunners for 523908 |
[production] |
| 08:37 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@440faa0]: more error reporting for stubs/abstracts/pagelogs; more public table dumps by default (duration: 00m 04s) |
[production] |
| 08:37 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@440faa0]: more error reporting for stubs/abstracts/pagelogs; more public table dumps by default |
[production] |
| 08:36 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 08:24 |
<gehel> |
repooling wdqs2004 - T228122 |
[production] |
| 08:22 |
<gehel> |
repooling wdqs2003 - T228122 |
[production] |
| 08:20 |
<vgutierrez> |
restart pybal on lvs2003 |
[production] |
| 08:16 |
<vgutierrez> |
restart pybal on lvs2006 |
[production] |
| 08:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db1109 into API (duration: 00m 54s) |
[production] |
| 07:57 |
<moritzm> |
installing idp1001 T228403 |
[production] |
| 07:38 |
<moritzm> |
rebooting tungsten for kernel update |
[production] |
| 07:38 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:38 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:03 |
<elukey> |
restart php-fpm on mw1330 - op-cache hit ratio low |
[production] |
| 07:02 |
<jynus> |
reloading dbproxy1004/9 |
[production] |
| 07:01 |
<elukey> |
depool wdqs2004 from all services (waiting for maintenance) |
[production] |
| 06:32 |
<legoktm@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/EventBus/includes/EventBus.php: Add more debugging to figure out which events are invalid: T225199 (duration: 00m 55s) |
[production] |
| 06:30 |
<legoktm@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/EventBus/includes/EventBus.php: Add more debugging to figure out which events are invalid: T225199 (duration: 00m 55s) |
[production] |
| 06:15 |
<elukey> |
clear opcache on mwdebug* |
[production] |
| 05:26 |
<fsero> |
repool ms-fe2005 - T228196 |
[production] |
| 05:11 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2116 (duration: 00m 55s) |
[production] |
| 04:11 |
<eileen> |
I think I didn't push the turn it on commit - tried again process-control config revision is 9f7eba2193 |
[production] |
| 03:03 |
<eileen> |
process-control config revision is 7598dc1bf9 (jobs reenabled) |
[production] |
| 01:52 |
<XioNoX> |
enable outbound sampling on eqiad's router |
[production] |
| 00:52 |
<sbassett@deploy1001> |
Synchronized private/PrivateSettings.php: Add even more severe rate limits for eswikiquote and some other, smaller wikis (T227416) (duration: 00m 58s) |
[production] |
| 00:38 |
<mutante> |
mwmaint2001 - puppet fails - not removing a bunch of log dirs for maintenance crons |
[production] |