|
2021-01-28
ยง
|
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13999 and previous config saved to /var/cache/conftool/dbconfig/20210128-073333-root.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 10%: After upgrading the kernel', diff saved to https://phabricator.wikimedia.org/P13998 and previous config saved to /var/cache/conftool/dbconfig/20210128-073323-root.json |
[production] |
| 07:25 |
<elukey> |
powercycle cp1087 (after depooling it) |
[production] |
| 07:24 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3315 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13997 and previous config saved to /var/cache/conftool/dbconfig/20210128-072154-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for kernel upgrade and enablement of report_host', diff saved to https://phabricator.wikimedia.org/P13996 and previous config saved to /var/cache/conftool/dbconfig/20210128-072120-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1169 some more minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13995 and previous config saved to /var/cache/conftool/dbconfig/20210128-072036-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to s1 for the first time, with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P13994 and previous config saved to /var/cache/conftool/dbconfig/20210128-063806-marostegui.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1169 to dbctl T258361', diff saved to https://phabricator.wikimedia.org/P13993 and previous config saved to /var/cache/conftool/dbconfig/20210128-063655-marostegui.json |
[production] |
| 03:03 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 03:00 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2291.codfw.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2290.codfw.wmnet |
[production] |
| 02:13 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2288.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2288.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2290.codfw.wmnet |
[production] |
| 02:05 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2291.codfw.wmnet |
[production] |
| 02:02 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |
| 02:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |
| 01:35 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2291.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:35 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:35 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:33 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2291.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:33 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2290.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:32 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:32 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:32 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:32 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw2288.codfw.wmnet with reason: reimaging |
[production] |
| 01:31 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2288.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:31 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2290.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:31 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2288.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet |
[production] |
| 01:10 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2294.codfw.wmnet |
[production] |
| 01:09 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2293.codfw.wmnet |
[production] |
| 01:09 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2292.codfw.wmnet |
[production] |
| 00:56 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2294.codfw.wmnet |
[production] |
| 00:56 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2293.codfw.wmnet |
[production] |
| 00:56 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2292.codfw.wmnet |
[production] |
| 00:50 |
<Urbanecm> |
Evening B&C done |
[production] |
| 00:49 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 87c304c5439b1b7898f951db61d0a0a8a11ee4f7: Disable max-width on page namespace for wikisource (T260091; 2nd take) (duration: 01m 00s) |
[production] |
| 00:43 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1404.eqiad.wmnet |
[production] |
| 00:41 |
<foks> |
reset email for User:Uwe Martens |
[production] |
| 00:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1404.eqiad.wmnet |
[production] |
| 00:39 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1404.wmnet |
[production] |
| 00:33 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.27/includes/: c5c39ba8b3fce3f946e161191b814446aa5c1f4b: Fix fetching ipblock-exempt within BlockManager::getUserBlock (T271551, T270145) (duration: 01m 04s) |
[production] |
| 00:32 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw2293.codfw.wmnet with reason: reimaging |
[production] |
| 00:32 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw2293.codfw.wmnet with reason: reimaging |
[production] |
| 00:31 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.28/includes/: a67fe4f7cbf172b82153aaceaa93a067cdff2ae4: Fix fetching ipblock-exempt within BlockManager::getUserBlock (T271551, T270145) (duration: 01m 07s) |
[production] |
| 00:28 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2292.codfw.wmnet with reason: REIMAGE |
[production] |