|
2021-03-22
ยง
|
| 23:52 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:49 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:34 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2250.codfw.wmnet |
[production] |
| 23:21 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:18 |
<ebernhardson@deploy1002> |
Synchronized php-1.36.0-wmf.35/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: T262612: Start glent m1 ab test (duration: 01m 53s) |
[production] |
| 23:18 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2250.codfw.wmnet |
[production] |
| 23:01 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2249.codfw.wmnet |
[production] |
| 22:52 |
<mutante> |
decom mw2249 |
[production] |
| 22:44 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2249.codfw.wmnet |
[production] |
| 21:08 |
<sbassett> |
Deployed security patch for T272244 |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2279.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2278.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2279.codfw.wmnet,service=canary |
[production] |
| 20:02 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2278.codfw.wmnet,service=canary |
[production] |
| 19:50 |
<mutante> |
gerrit2001 - restarted apache2 as well for consistency |
[production] |
| 19:47 |
<mutante> |
gerrit - restarting apache2 after we dropped MaxClients config line. This should make us fall back to Debian default MaxRequestWorkers. (since we use event MPM we should not be using MaxClients in the first place, says #httpd) (T277127) |
[production] |
| 18:20 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 25247c9cbba3d3741908164f2d15fb8497ce8b5e: hrwiki: Configure mentorship for Growth team features (T275684) (duration: 01m 00s) |
[production] |
| 18:13 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 951601f7a4c887f21e209b32dbd1cfd3da084816: Grant enwiki pagemovers the delete-redirect right (T278131) (duration: 00m 59s) |
[production] |
| 17:30 |
<Trey314159> |
reindexing Italian wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200) |
[production] |
| 16:49 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:48 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 16:47 |
<jayme@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:46 |
<jayme@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 16:37 |
<jayme@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:37 |
<jayme@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 16:12 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:07 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: Slowly repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P14990 and previous config saved to /var/cache/conftool/dbconfig/20210322-155808-root.json |
[production] |
| 15:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: Slowly repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P14989 and previous config saved to /var/cache/conftool/dbconfig/20210322-154304-root.json |
[production] |
| 15:38 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:33 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: Slowly repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P14988 and previous config saved to /var/cache/conftool/dbconfig/20210322-152800-root.json |
[production] |
| 15:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: Slowly repool db1144:3314', diff saved to https://phabricator.wikimedia.org/P14987 and previous config saved to /var/cache/conftool/dbconfig/20210322-151257-root.json |
[production] |
| 14:26 |
<jayme@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:25 |
<jayme@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:23 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:22 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:14 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:14 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3314 for schema change', diff saved to https://phabricator.wikimedia.org/P14986 and previous config saved to /var/cache/conftool/dbconfig/20210322-141146-marostegui.json |
[production] |
| 14:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 100%: Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P14985 and previous config saved to /var/cache/conftool/dbconfig/20210322-140800-root.json |
[production] |
| 14:07 |
<XioNoX> |
rename cloud-hosts1-b-eqiad to cloud-hosts1-eqiad - T277771 |
[production] |
| 14:07 |
<XioNoX> |
rename cloud-hosts1-b-eqiad to cloud-hosts1-eqiad |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P14984 and previous config saved to /var/cache/conftool/dbconfig/20210322-135256-root.json |
[production] |
| 13:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P14983 and previous config saved to /var/cache/conftool/dbconfig/20210322-133753-root.json |
[production] |
| 13:26 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 13:26 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 13:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: Slowly repool db1143', diff saved to https://phabricator.wikimedia.org/P14982 and previous config saved to /var/cache/conftool/dbconfig/20210322-132249-root.json |
[production] |
| 13:20 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |