|
2021-01-28
ยง
|
| 11:01 |
<_joe_> |
restarting php-fpm on the appserver,api and jobrunner clusters in eqiad, 10% at a time, for simulating scap rolling restarts T266055 |
[production] |
| 10:52 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool es5 on writes T266483 (duration: 01m 05s) |
[production] |
| 10:46 |
<marostegui> |
Restart mysql on es1024 T266483 |
[production] |
| 10:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool es5 from writes T266483 (duration: 01m 09s) |
[production] |
| 10:33 |
<_joe_> |
performing a test-run of the rolling restart of php-fpm in codfw, using the same code scap will use T266055. Starting from the api cluster, then proceeding whith others |
[production] |
| 10:15 |
<_joe_> |
upgrading pybal on lvs2008 |
[production] |
| 10:11 |
<_joe_> |
upgrading pybal on lvs2009 |
[production] |
| 10:10 |
<vgutierrez> |
upgrade pybal to 1.15.9 in eqsin |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14017 and previous config saved to /var/cache/conftool/dbconfig/20210128-095642-root.json |
[production] |
| 09:48 |
<_joe_> |
upgrading pybal to 1.15.9 in codfw, starting from lvs2010 |
[production] |
| 09:47 |
<jbond42> |
upload new cas package to apt |
[production] |
| 09:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 80%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14016 and previous config saved to /var/cache/conftool/dbconfig/20210128-094139-root.json |
[production] |
| 09:30 |
<_joe_> |
upgrading pybal on lvs4006 |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 70%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14015 and previous config saved to /var/cache/conftool/dbconfig/20210128-092635-root.json |
[production] |
| 09:25 |
<_joe_> |
upgrading pybal on lvs4005 |
[production] |
| 09:11 |
<_joe_> |
installing pybal 1.15.9 on lvs4007 |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 60%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14014 and previous config saved to /var/cache/conftool/dbconfig/20210128-091131-root.json |
[production] |
| 09:08 |
<moritzm> |
installing perf updates on Stretch |
[production] |
| 09:06 |
<marostegui> |
Testing wikitech |
[production] |
| 09:00 |
<marostegui> |
m5 master restart, wikitech will be unavailable - T272596 |
[production] |
| 09:00 |
<_joe_> |
uploading pybal 1.15.9 to apt.wikimedia.org |
[production] |
| 08:58 |
<moritzm> |
installing perf updates on Buster |
[production] |
| 08:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14013 and previous config saved to /var/cache/conftool/dbconfig/20210128-085627-root.json |
[production] |
| 08:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 40%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14012 and previous config saved to /var/cache/conftool/dbconfig/20210128-084123-root.json |
[production] |
| 08:34 |
<godog> |
swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837 |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14011 and previous config saved to /var/cache/conftool/dbconfig/20210128-083347-root.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14010 and previous config saved to /var/cache/conftool/dbconfig/20210128-083337-root.json |
[production] |
| 08:32 |
<vgutierrez> |
pool cp1087 - T273153 |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 30%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14009 and previous config saved to /var/cache/conftool/dbconfig/20210128-082620-root.json |
[production] |
| 08:20 |
<vgutierrez> |
restart purged on cp1087 - T273153 |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14008 and previous config saved to /var/cache/conftool/dbconfig/20210128-081843-root.json |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14007 and previous config saved to /var/cache/conftool/dbconfig/20210128-081834-root.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 20%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14006 and previous config saved to /var/cache/conftool/dbconfig/20210128-081116-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14005 and previous config saved to /var/cache/conftool/dbconfig/20210128-080340-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14004 and previous config saved to /var/cache/conftool/dbconfig/20210128-080330-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 15%: Pooling for the first time very slowly', diff saved to https://phabricator.wikimedia.org/P14003 and previous config saved to /var/cache/conftool/dbconfig/20210128-075613-root.json |
[production] |
| 07:54 |
<moritzm> |
installing tomcat9 security updates |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14002 and previous config saved to /var/cache/conftool/dbconfig/20210128-074836-root.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P14001 and previous config saved to /var/cache/conftool/dbconfig/20210128-074827-root.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1169 some more minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14000 and previous config saved to /var/cache/conftool/dbconfig/20210128-073426-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13999 and previous config saved to /var/cache/conftool/dbconfig/20210128-073333-root.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13998 and previous config saved to /var/cache/conftool/dbconfig/20210128-073323-root.json |
[production] |
| 07:25 |
<elukey> |
powercycle cp1087 (after depooling it) |
[production] |
| 07:24 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3315 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13997 and previous config saved to /var/cache/conftool/dbconfig/20210128-072154-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13996 and previous config saved to /var/cache/conftool/dbconfig/20210128-072120-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1169 some more minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13995 and previous config saved to /var/cache/conftool/dbconfig/20210128-072036-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to s1 for the first time, with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13994 and previous config saved to /var/cache/conftool/dbconfig/20210128-063806-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to dbctl T258361', diff saved to https://phabricator.wikimedia.org/P13993 and previous config saved to /var/cache/conftool/dbconfig/20210128-063655-marostegui.json |
[production] |
| 03:03 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |