|
2021-02-09
ยง
|
| 13:25 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh (exit_code=0) for Hadoop analytics cluster: Change Hadoop distribution - elukey@cumin1001 |
[production] |
| 13:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 30%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14264 and previous config saved to /var/cache/conftool/dbconfig/20210209-132145-root.json |
[production] |
| 13:08 |
<twentyafterfour> |
restart phabricator daemons to free 3.5gb of ram (memory leak?) |
[production] |
| 13:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14263 and previous config saved to /var/cache/conftool/dbconfig/20210209-130641-root.json |
[production] |
| 12:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 20%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14262 and previous config saved to /var/cache/conftool/dbconfig/20210209-125138-root.json |
[production] |
| 12:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 15%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14261 and previous config saved to /var/cache/conftool/dbconfig/20210209-123634-root.json |
[production] |
| 12:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 13%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14260 and previous config saved to /var/cache/conftool/dbconfig/20210209-122131-root.json |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 10%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14259 and previous config saved to /var/cache/conftool/dbconfig/20210209-120627-root.json |
[production] |
| 12:05 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro-from-cdh for Hadoop analytics cluster: Change Hadoop distribution - elukey@cumin1001 |
[production] |
| 12:02 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) for Hadoop analytics cluster: Stop the Hadoop cluster before maintenance. - elukey@cumin1001 |
[production] |
| 11:58 |
<hnowlan@puppetmaster1001> |
conftool action : set/weight=10; selector: name=maps2010.codfw.wmnet |
[production] |
| 11:58 |
<hnowlan@puppetmaster1001> |
conftool action : set/weight=10; selector: name=maps2009.codfw.wmnet |
[production] |
| 11:58 |
<hnowlan@puppetmaster1001> |
conftool action : set/weight=10; selector: name=maps2008.codfw.wmnet |
[production] |
| 11:58 |
<hnowlan@puppetmaster1001> |
conftool action : set/weight=10; selector: name=maps2006.codfw.wmnet |
[production] |
| 11:57 |
<hnowlan@puppetmaster1001> |
conftool action : set/weight=10; selector: name=maps2005.codfw.wmnet |
[production] |
| 11:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1013.eqiad.wmnet |
[production] |
| 11:52 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=maps1010.eqiad.wmnet |
[production] |
| 11:52 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=maps1008.eqiad.wmnet |
[production] |
| 11:51 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=maps1007.eqiad.wmnet |
[production] |
| 11:51 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=maps1006.eqiad.wmnet |
[production] |
| 11:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 8%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14258 and previous config saved to /var/cache/conftool/dbconfig/20210209-115124-root.json |
[production] |
| 11:51 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs1013.eqiad.wmnet |
[production] |
| 11:50 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=maps1001.eqiad.wmnet |
[production] |
| 11:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1014.eqiad.wmnet |
[production] |
| 11:40 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs1014.eqiad.wmnet |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 5%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14257 and previous config saved to /var/cache/conftool/dbconfig/20210209-113620-root.json |
[production] |
| 11:34 |
<elukey> |
start the upgrade process for Hadoop Analytics |
[production] |
| 11:33 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster for Hadoop analytics cluster: Stop the Hadoop cluster before maintenance. - elukey@cumin1001 |
[production] |
| 11:32 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1015.eqiad.wmnet |
[production] |
| 11:27 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs1015.eqiad.wmnet |
[production] |
| 11:23 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1016.eqiad.wmnet |
[production] |
| 11:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 4%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14256 and previous config saved to /var/cache/conftool/dbconfig/20210209-112116-root.json |
[production] |
| 11:18 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs1016.eqiad.wmnet |
[production] |
| 11:17 |
<vgutierrez> |
rolling restart of eqiad LVS instances to catch up on kernel upgrades |
[production] |
| 11:07 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs3005.esams.wmnet |
[production] |
| 11:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 3%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14255 and previous config saved to /var/cache/conftool/dbconfig/20210209-110613-root.json |
[production] |
| 11:02 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs3005.esams.wmnet |
[production] |
| 10:57 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database, still |
[production] |
| 10:57 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database, still |
[production] |
| 10:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs3006.esams.wmnet |
[production] |
| 10:53 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2001.codfw.wmnet |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 2%: Slowly pool db1157 into s3', diff saved to https://phabricator.wikimedia.org/P14254 and previous config saved to /var/cache/conftool/dbconfig/20210209-105109-root.json |
[production] |
| 10:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs3006.esams.wmnet |
[production] |
| 10:48 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs3007.esams.wmnet |
[production] |
| 10:43 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs3007.esams.wmnet |
[production] |
| 10:41 |
<vgutierrez> |
rolling restart of esams LVS instances to catch up on kernel upgrades |
[production] |
| 10:40 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cumin2001.codfw.wmnet |
[production] |
| 10:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3317 (re)pooling @ 100%: Slowly repooling db1090:3317 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14253 and previous config saved to /var/cache/conftool/dbconfig/20210209-103443-root.json |
[production] |
| 10:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1090:3312 (re)pooling @ 100%: Slowly repooling db1090:3312 after cloning db1170', diff saved to https://phabricator.wikimedia.org/P14252 and previous config saved to /var/cache/conftool/dbconfig/20210209-103414-root.json |
[production] |
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1157 for the first time in s3 T258361', diff saved to https://phabricator.wikimedia.org/P14251 and previous config saved to /var/cache/conftool/dbconfig/20210209-102109-marostegui.json |
[production] |