|
2019-10-24
§
|
| 11:08 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:08 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:08 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:00 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:58 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:56 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:55 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:55 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:55 |
<ema@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:52 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Adjust s6 weights for db1093 and db1085', diff saved to https://phabricator.wikimedia.org/P9466 and previous config saved to /var/cache/conftool/dbconfig/20191024-101810-marostegui.json |
[production] |
| 09:59 |
<hashar> |
Converting CI jobs to use the new PostBuildScript plugin config | https://gerrit.wikimedia.org/r/#/c/integration/config/+/544907/ | T188398 |
[production] |
| 09:57 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 09:35 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:33 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:14 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:12 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T234853 Re-enable performance perception survey on ruwiki (duration: 01m 04s) |
[production] |
| 08:39 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:37 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:36 |
<godog> |
roll restart rsyslog in codfw/eqiad to pick up new kafka partitions |
[production] |
| 08:18 |
<godog> |
roll restart rsyslog in ulsfo/esams/eqsin to pick up new kafka partitions |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2092 for analyze table', diff saved to https://phabricator.wikimedia.org/P9465 and previous config saved to /var/cache/conftool/dbconfig/20191024-081519-marostegui.json |
[production] |
| 07:57 |
<XioNoX> |
reboot mr1-esams |
[production] |
| 07:42 |
<godog> |
bump rsyslog- topics partitions to 6 and roll-restart logstash frontends |
[production] |
| 07:24 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 07:22 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:22 |
<XioNoX> |
drain Telia link on cr2-esams |
[production] |
| 06:32 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=parsoid-php,name=eqiad |
[production] |
| 05:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9463 and previous config saved to /var/cache/conftool/dbconfig/20191024-052002-marostegui.json |
[production] |
| 05:18 |
<marostegui> |
Run analyze enwiki.revision on db2092 T223151 |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9462 and previous config saved to /var/cache/conftool/dbconfig/20191024-045954-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1089 from special slaves group and leave it with its original pooling options T223151', diff saved to https://phabricator.wikimedia.org/P9461 and previous config saved to /var/cache/conftool/dbconfig/20191024-045924-marostegui.json |
[production] |
| 04:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3315 after compression', diff saved to https://phabricator.wikimedia.org/P9460 and previous config saved to /var/cache/conftool/dbconfig/20191024-045544-marostegui.json |
[production] |
| 04:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 04:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 03:55 |
<shdubsh> |
temporarily turn down accept delay on fermium - T235983 |
[production] |
| 00:03 |
<mutante> |
restarting gerrit to increase heap_size from 20G to 32G (T225166 T222391) |
[production] |