|
2020-01-07
ยง
|
| 11:12 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:12 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 11:12 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 11:11 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:10 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 11:10 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:53 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 10:53 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:39 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.11/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermIdsAcquirer.php: [[gerrit:562477|Temporary add metrics of the need to reinsert in the new term store]] (duration: 00m 57s) |
[production] |
| 10:24 |
<arturo> |
created canary1009-01 VM using horizon and the cold-migrate it to cloudvirt1009 from cloudvirt1006 (T242078) |
[testlabs] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:08 |
<arturo> |
delete VM stretch-boot-arturo-02 no longer in use |
[testlabs] |
| 10:08 |
<arturo> |
delete VM buster-boot-arturo-01 no longer in use |
[testlabs] |
| 10:08 |
<arturo> |
delete VM stretch-boot-arturo-01 no longer in use |
[testlabs] |
| 10:07 |
<arturo> |
delete canary1009-01 VM to re-create it (T242078) |
[testlabs] |
| 10:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1092', diff saved to https://phabricator.wikimedia.org/P10062 and previous config saved to /var/cache/conftool/dbconfig/20200107-100743-marostegui.json |
[production] |
| 10:05 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 10:05 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:02 |
<arturo> |
icinga downtime cloudvirt1009 for 30 minutes to re-create canary VM (T242078) |
[admin] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P10061 and previous config saved to /var/cache/conftool/dbconfig/20200107-100157-marostegui.json |
[production] |
| 10:01 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:01 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P10060 and previous config saved to /var/cache/conftool/dbconfig/20200107-095501-marostegui.json |
[production] |
| 09:52 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 09:52 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 09:51 |
<arturo> |
stop/start webservice. It was showing 503 for whatever reason |
[tools.hay] |
| 09:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P10059 and previous config saved to /var/cache/conftool/dbconfig/20200107-094944-marostegui.json |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P10058 and previous config saved to /var/cache/conftool/dbconfig/20200107-094506-marostegui.json |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 for alter and upgrade', diff saved to https://phabricator.wikimedia.org/P10057 and previous config saved to /var/cache/conftool/dbconfig/20200107-092221-marostegui.json |
[production] |
| 09:02 |
<twentyafterfour> |
deploying https://gerrit.wikimedia.org/r/c/integration/config/+/562417 |
[releng] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084 for compression', diff saved to https://phabricator.wikimedia.org/P10056 and previous config saved to /var/cache/conftool/dbconfig/20200107-082236-marostegui.json |
[production] |
| 08:11 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@7a0f7aa]: Upgrade LibreNMS to 1.59 - T241962 (duration: 00m 10s) |
[production] |
| 08:11 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@7a0f7aa]: Upgrade LibreNMS to 1.59 - T241962 |
[production] |
| 07:59 |
<elukey> |
restart hue (again) with correct principal settings) |
[analytics] |
| 07:42 |
<elukey> |
restart Hue after applying a new kerberos setting (hue_principal, was not specified before) |
[analytics] |
| 07:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool es1019', diff saved to https://phabricator.wikimedia.org/P10055 and previous config saved to /var/cache/conftool/dbconfig/20200107-074159-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1019 for upgrade', diff saved to https://phabricator.wikimedia.org/P10054 and previous config saved to /var/cache/conftool/dbconfig/20200107-074035-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool es1013', diff saved to https://phabricator.wikimedia.org/P10053 and previous config saved to /var/cache/conftool/dbconfig/20200107-073922-marostegui.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1013 for upgrade', diff saved to https://phabricator.wikimedia.org/P10052 and previous config saved to /var/cache/conftool/dbconfig/20200107-073543-marostegui.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool es1018', diff saved to https://phabricator.wikimedia.org/P10051 and previous config saved to /var/cache/conftool/dbconfig/20200107-073508-marostegui.json |
[production] |
| 07:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1018 for upgrade', diff saved to https://phabricator.wikimedia.org/P10050 and previous config saved to /var/cache/conftool/dbconfig/20200107-072930-marostegui.json |
[production] |
| 07:15 |
<marostegui> |
Remove partitions from s5: db2084:3315 T239453 |
[production] |
| 07:12 |
<marostegui> |
Remove partitions from revision table on s6: db1098 T239453 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10049 and previous config saved to /var/cache/conftool/dbconfig/20200107-070850-marostegui.json |
[production] |
| 07:05 |
<marostegui> |
Depool labsdb1011 |
[production] |
| 07:03 |
<marostegui> |
Deploy schema change on s8 codfw (this will generate lag on s8 codfw) - T234052 |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2089:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10048 and previous config saved to /var/cache/conftool/dbconfig/20200107-064846-marostegui.json |
[production] |
| 05:28 |
<bd808> |
Creating tools-k8s-worker-[6-14] (again) |
[tools] |
| 05:20 |
<bd808> |
Deleting busted tools-k8s-worker-[6-14] |
[tools] |