|
2021-03-23
§
|
| 07:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 25%: Slowly repool db1146:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P15010 and previous config saved to /var/cache/conftool/dbconfig/20210323-073702-root.json |
[production] |
| 07:36 |
<elukey> |
create a 50g lvm volume on prometheus[12]00[34] for the k8s-mlserve cluster - T272918 |
[production] |
| 07:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:32 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 100%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P15009 and previous config saved to /var/cache/conftool/dbconfig/20210323-072352-root.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 50%: Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P15008 and previous config saved to /var/cache/conftool/dbconfig/20210323-072223-root.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 50%: Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15007 and previous config saved to /var/cache/conftool/dbconfig/20210323-072209-root.json |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 75%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P15006 and previous config saved to /var/cache/conftool/dbconfig/20210323-070849-root.json |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 25%: Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P15005 and previous config saved to /var/cache/conftool/dbconfig/20210323-070719-root.json |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 25%: Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P15004 and previous config saved to /var/cache/conftool/dbconfig/20210323-070705-root.json |
[production] |
| 07:02 |
<marostegui> |
Upgrade kernel on db1101 |
[production] |
| 06:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15003 and previous config saved to /var/cache/conftool/dbconfig/20210323-065947-marostegui.json |
[production] |
| 06:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 to enable report_host T266483', diff saved to https://phabricator.wikimedia.org/P15002 and previous config saved to /var/cache/conftool/dbconfig/20210323-065836-marostegui.json |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 50%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P15001 and previous config saved to /var/cache/conftool/dbconfig/20210323-065345-root.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 25%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P15000 and previous config saved to /var/cache/conftool/dbconfig/20210323-063842-root.json |
[production] |
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14999 and previous config saved to /var/cache/conftool/dbconfig/20210323-062942-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 10%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P14998 and previous config saved to /var/cache/conftool/dbconfig/20210323-062338-root.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1086', diff saved to https://phabricator.wikimedia.org/P14997 and previous config saved to /var/cache/conftool/dbconfig/20210323-062059-marostegui.json |
[production] |
| 06:20 |
<marostegui> |
Upgrade kernel on db1086 |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1086 (re)pooling @ 25%: Slowly repool db1086 after removing it from master', diff saved to https://phabricator.wikimedia.org/P14996 and previous config saved to /var/cache/conftool/dbconfig/20210323-060701-root.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1136 to s7 master and remove read-only from s7 T274336', diff saved to https://phabricator.wikimedia.org/P14995 and previous config saved to /var/cache/conftool/dbconfig/20210323-060216-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s7 as read-only for maintenance T274336', diff saved to https://phabricator.wikimedia.org/P14994 and previous config saved to /var/cache/conftool/dbconfig/20210323-060104-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting s7 eqiad failover from db1086 to db1136 - T274336 |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1174 to api T274336', diff saved to https://phabricator.wikimedia.org/P14993 and previous config saved to /var/cache/conftool/dbconfig/20210323-051346-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set weight 0 to db1136 before failover T274336', diff saved to https://phabricator.wikimedia.org/P14992 and previous config saved to /var/cache/conftool/dbconfig/20210323-051210-marostegui.json |
[production] |
| 00:07 |
<tstarling@deploy1002> |
Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s) |
[production] |
| 00:06 |
<tstarling@deploy1002> |
Synchronized wmf-config/CommonSettings.php: use RequestTimeout library step 2: enable new system (duration: 00m 57s) |
[production] |
| 00:04 |
<tstarling@deploy1002> |
Synchronized wmf-config/PhpAutoPrepend.php: use RequestTimeout library step 1: disable old request timeout system (duration: 00m 58s) |
[production] |
|
2021-03-22
§
|
| 23:52 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:49 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:34 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2250.codfw.wmnet |
[production] |
| 23:21 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:18 |
<ebernhardson@deploy1002> |
Synchronized php-1.36.0-wmf.35/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: T262612: Start glent m1 ab test (duration: 01m 53s) |
[production] |
| 23:18 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2250.codfw.wmnet |
[production] |
| 23:01 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2249.codfw.wmnet |
[production] |
| 22:52 |
<mutante> |
decom mw2249 |
[production] |
| 22:44 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2249.codfw.wmnet |
[production] |
| 21:08 |
<sbassett> |
Deployed security patch for T272244 |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2279.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2278.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2279.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2278.codfw.wmnet,service=canary |
[production] |
| 19:50 |
<mutante> |
gerrit2001 - restarted apache2 as well for consistency |
[production] |
| 19:47 |
<mutante> |
gerrit - restarting apache2 after we dropped MaxClients config line. This should make us fall back to Debian default MaxRequestWorkers. (since we use event MPM we should not be using MaxClients in the first place, says #httpd) (T277127) |
[production] |
| 18:20 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 25247c9cbba3d3741908164f2d15fb8497ce8b5e: hrwiki: Configure mentorship for Growth team features (T275684) (duration: 01m 00s) |
[production] |
| 18:13 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 951601f7a4c887f21e209b32dbd1cfd3da084816: Grant enwiki pagemovers the delete-redirect right (T278131) (duration: 00m 59s) |
[production] |
| 17:30 |
<Trey314159> |
reindexing Italian wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200) |
[production] |
| 16:49 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:48 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |