|
2021-01-28
ยง
|
| 09:48 |
<_joe_> |
upgrading pybal to 1.15.9 in codfw, starting from lvs2010 |
[production] |
| 09:47 |
<jbond42> |
upload new cas package to apt |
[production] |
| 09:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 80%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14016 and previous config saved to /var/cache/conftool/dbconfig/20210128-094139-root.json |
[production] |
| 09:30 |
<_joe_> |
upgrading pybal on lvs4006 |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 70%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14015 and previous config saved to /var/cache/conftool/dbconfig/20210128-092635-root.json |
[production] |
| 09:25 |
<_joe_> |
upgrading pybal on lvs4005 |
[production] |
| 09:11 |
<_joe_> |
installing pybal 1.15.9 on lvs4007 |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 60%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14014 and previous config saved to /var/cache/conftool/dbconfig/20210128-091131-root.json |
[production] |
| 09:08 |
<moritzm> |
installing perf updates on Stretch |
[production] |
| 09:06 |
<marostegui> |
Testing wikitech |
[production] |
| 09:00 |
<marostegui> |
m5 master restart, wikitech will be unavailable - T272596 |
[production] |
| 09:00 |
<_joe_> |
uploading pybal 1.15.9 to apt.wikimedia.org |
[production] |
| 08:58 |
<moritzm> |
installing perf updates on Buster |
[production] |
| 08:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14013 and previous config saved to /var/cache/conftool/dbconfig/20210128-085627-root.json |
[production] |
| 08:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 40%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14012 and previous config saved to /var/cache/conftool/dbconfig/20210128-084123-root.json |
[production] |
| 08:34 |
<godog> |
swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837 |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14011 and previous config saved to /var/cache/conftool/dbconfig/20210128-083347-root.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14010 and previous config saved to /var/cache/conftool/dbconfig/20210128-083337-root.json |
[production] |
| 08:32 |
<vgutierrez> |
pool cp1087 - T273153 |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 30%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14009 and previous config saved to /var/cache/conftool/dbconfig/20210128-082620-root.json |
[production] |
| 08:20 |
<vgutierrez> |
restart purged on cp1087 - T273153 |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14008 and previous config saved to /var/cache/conftool/dbconfig/20210128-081843-root.json |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14007 and previous config saved to /var/cache/conftool/dbconfig/20210128-081834-root.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 20%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14006 and previous config saved to /var/cache/conftool/dbconfig/20210128-081116-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14005 and previous config saved to /var/cache/conftool/dbconfig/20210128-080340-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14004 and previous config saved to /var/cache/conftool/dbconfig/20210128-080330-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 15%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14003 and previous config saved to /var/cache/conftool/dbconfig/20210128-075613-root.json |
[production] |
| 07:54 |
<moritzm> |
installing tomcat9 security updates |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14002 and previous config saved to /var/cache/conftool/dbconfig/20210128-074836-root.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14001 and previous config saved to /var/cache/conftool/dbconfig/20210128-074827-root.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1169 some more minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14000 and previous config saved to /var/cache/conftool/dbconfig/20210128-073426-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13999 and previous config saved to /var/cache/conftool/dbconfig/20210128-073333-root.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13998 and previous config saved to /var/cache/conftool/dbconfig/20210128-073323-root.json |
[production] |
| 07:25 |
<elukey> |
powercycle cp1087 (after depooling it) |
[production] |
| 07:24 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3315 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13997 and previous config saved to /var/cache/conftool/dbconfig/20210128-072154-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13996 and previous config saved to /var/cache/conftool/dbconfig/20210128-072120-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1169 some more minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13995 and previous config saved to /var/cache/conftool/dbconfig/20210128-072036-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to s1 for the first time, with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13994 and previous config saved to /var/cache/conftool/dbconfig/20210128-063806-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to dbctl T258361', diff saved to https://phabricator.wikimedia.org/P13993 and previous config saved to /var/cache/conftool/dbconfig/20210128-063655-marostegui.json |
[production] |
| 03:03 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 03:00 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2291.codfw.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2290.codfw.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2288.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2288.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2290.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2291.codfw.wmnet |
[production] |
| 02:02 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |
| 02:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |