|
2020-09-16
ยง
|
| 10:14 |
<liw@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.9 (duration: 46m 07s) |
[production] |
| 10:10 |
<ema> |
upload python-acme 0.31.0-2wm1 to buster-wikimedia T263006 |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12605 and previous config saved to /var/cache/conftool/dbconfig/20200916-100548-marostegui.json |
[production] |
| 10:01 |
<akosiaris> |
T187984 Shutdown mendelevium. |
[production] |
| 09:43 |
<jynus> |
deploying max_packet_size change to m3 instances, too |
[production] |
| 09:28 |
<liw@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.9 |
[production] |
| 09:26 |
<liw> |
moving train 1.36.0-wmf.9 to testwikis |
[production] |
| 09:22 |
<jynus> |
restarting gerrit service on gerrit1001, unresponsive |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12603 and previous config saved to /var/cache/conftool/dbconfig/20200916-091535-marostegui.json |
[production] |
| 09:13 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 0 - T262290 |
[production] |
| 09:08 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 1 - T262290 |
[production] |
| 08:52 |
<marostegui> |
Stop mysql on db1121, db1123, db1093 and db1109 for PDU work T261454 T261457 |
[production] |
| 08:52 |
<XioNoX> |
asw-d-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:50 |
<jynus> |
deploy new max_allowed_packet configuration to m1, m2 and m5 dbs |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12601 and previous config saved to /var/cache/conftool/dbconfig/20200916-084916-marostegui.json |
[production] |
| 08:42 |
<awight> |
finished security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:41 |
<awight@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/FileImporter/src/Services/ImportPlanValidator.php: Security patch for T262628 (duration: 00m 59s) |
[production] |
| 08:41 |
<XioNoX> |
asw-c-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:27 |
<XioNoX> |
asw-b-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:26 |
<awight> |
beginning security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:17 |
<XioNoX> |
asw-a-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:04 |
<akosiaris> |
T187984 Validated that ticket.wikimedia.org works, proceeding with a wider announcement |
[production] |
| 08:02 |
<XioNoX> |
asw2-d-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:49 |
<akosiaris> |
T187984 Switch over ticket.discovery.wmnet to otrs1001 |
[production] |
| 07:48 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:44 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:40 |
<XioNoX> |
asw2-c-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:37 |
<akosiaris> |
T187984 Tested inbound email successfully |
[production] |
| 07:29 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:26 |
<akosiaris> |
T187984 Tested outbound email, switching inbound email configuration and performing tests |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12600 and previous config saved to /var/cache/conftool/dbconfig/20200916-072614-marostegui.json |
[production] |
| 07:22 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:22 |
<jayme@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 07:21 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:12 |
<akosiaris> |
T187984 Disable gravatar in system configuration to avoid leaking agent PII through a 3rd party service |
[production] |
| 07:03 |
<akosiaris> |
T187984 validated that the OTRS installation is functional over SSH |
[production] |
| 07:02 |
<akosiaris> |
T187984 migration script done. Config updates, rebuilds, package upgrades/reinstall and index rebuilds done |
[production] |
| 06:28 |
<godog> |
codfw-prod: bump weight for ms-be2057 - T261633 |
[production] |
| 06:20 |
<kart_> |
Updated cxserver to 2020-08-30-011854-production (T253439, T260557) |
[production] |
| 06:20 |
<XioNoX> |
asw2-b-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 06:15 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:11 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 for the first time with minimum weight T261717', diff saved to https://phabricator.wikimedia.org/P12599 and previous config saved to /var/cache/conftool/dbconfig/20200916-061013-marostegui.json |
[production] |
| 06:08 |
<kartik@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12598 and previous config saved to /var/cache/conftool/dbconfig/20200916-060717-marostegui.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2015 to clone es2031 T261717', diff saved to https://phabricator.wikimedia.org/P12597 and previous config saved to /var/cache/conftool/dbconfig/20200916-055535-marostegui.json |
[production] |
| 05:53 |
<XioNoX> |
asw2-a-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12596 and previous config saved to /var/cache/conftool/dbconfig/20200916-055108-marostegui.json |
[production] |
| 05:50 |
<XioNoX> |
msw1-codfw> request system snapshot slice alternate - T262290 |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add es2027 and es2028 to dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12595 and previous config saved to /var/cache/conftool/dbconfig/20200916-053918-marostegui.json |
[production] |