|
2020-02-07
ยง
|
| 10:37 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:36 |
<akosiaris> |
conduct experiments with stopping/starting uwsgi-ores on ores2001 T242705 |
[production] |
| 10:24 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:23 |
<vgutierrez> |
depool and reimage ncredir5001 as buster - T243391 |
[production] |
| 10:14 |
<vgutierrez> |
depool & reimage cp4022 as buster - T242093 |
[production] |
| 10:02 |
<akosiaris> |
increase capacity for wikifeeds by 50% T244535 |
[production] |
| 10:01 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:01 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 09:53 |
<ema> |
A:mw: increase keepalive_requests from 100 to 200 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/570670/ T241145 |
[production] |
| 09:09 |
<godog> |
roll restart cassandra instance on restbase-dev |
[production] |
| 09:03 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 09:03 |
<godog> |
restart cassandra on restbase-dev1004 to test logging pipeline onboard |
[production] |
| 09:01 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 08:59 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1090:3312, db1090:3317', diff saved to https://phabricator.wikimedia.org/P10343 and previous config saved to /var/cache/conftool/dbconfig/20200207-085846-marostegui.json |
[production] |
| 08:54 |
<marostegui> |
Upgrade db1090:3312, db1090:3317 |
[production] |
| 08:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1090:3312, db1090:3317 for upgrade', diff saved to https://phabricator.wikimedia.org/P10342 and previous config saved to /var/cache/conftool/dbconfig/20200207-085432-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1101:3317 T239453', diff saved to https://phabricator.wikimedia.org/P10341 and previous config saved to /var/cache/conftool/dbconfig/20200207-084447-marostegui.json |
[production] |
| 08:44 |
<moritzm> |
installing libexif security updates |
[production] |
| 08:21 |
<akosiaris> |
deploy https://gerrit.wikimedia.org/r/570726 T244535 to avoid CPU throttling of wikifeeds |
[production] |
| 08:21 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 07:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase base weight for db1126', diff saved to https://phabricator.wikimedia.org/P10340 and previous config saved to /var/cache/conftool/dbconfig/20200207-075323-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317 T239453', diff saved to https://phabricator.wikimedia.org/P10339 and previous config saved to /var/cache/conftool/dbconfig/20200207-075234-marostegui.json |
[production] |
| 07:48 |
<marostegui> |
Remove revision partitions from db2085:3318 T239453 |
[production] |
| 07:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fullyy repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10338 and previous config saved to /var/cache/conftool/dbconfig/20200207-074511-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10337 and previous config saved to /var/cache/conftool/dbconfig/20200207-074407-marostegui.json |
[production] |
| 07:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317 T239453', diff saved to https://phabricator.wikimedia.org/P10336 and previous config saved to /var/cache/conftool/dbconfig/20200207-074258-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1101:3317 T239453', diff saved to https://phabricator.wikimedia.org/P10335 and previous config saved to /var/cache/conftool/dbconfig/20200207-073130-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10334 and previous config saved to /var/cache/conftool/dbconfig/20200207-073026-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10333 and previous config saved to /var/cache/conftool/dbconfig/20200207-063831-marostegui.json |
[production] |
| 06:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1105:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10332 and previous config saved to /var/cache/conftool/dbconfig/20200207-063402-marostegui.json |
[production] |
| 06:31 |
<elukey> |
force a puppet run on all ores[12] nodes |
[production] |
| 06:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10331 and previous config saved to /var/cache/conftool/dbconfig/20200207-062731-marostegui.json |
[production] |
| 06:26 |
<marostegui> |
Reboot db1107 for update - T242702 |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10330 and previous config saved to /var/cache/conftool/dbconfig/20200207-062502-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10329 and previous config saved to /var/cache/conftool/dbconfig/20200207-062345-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1105:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10328 and previous config saved to /var/cache/conftool/dbconfig/20200207-062043-marostegui.json |
[production] |
| 04:49 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 04:46 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:16 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 04:14 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:13 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 04:11 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:51 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:49 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:42 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:40 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:27 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 01:25 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:24 |
<robh> |
eqsin pdu work ongoing starting now. ps1-603 swapping per T242250 |
[production] |