|
2020-09-16
§
|
| 09:22 |
<jynus> |
restarting gerrit service on gerrit1001, unresponsive |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12603 and previous config saved to /var/cache/conftool/dbconfig/20200916-091535-marostegui.json |
[production] |
| 09:13 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 0 - T262290 |
[production] |
| 09:08 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 1 - T262290 |
[production] |
| 08:52 |
<marostegui> |
Stop mysql on db1121, db1123, db1093 and db1109 for PDU work T261454 T261457 |
[production] |
| 08:52 |
<XioNoX> |
asw-d-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:50 |
<jynus> |
deploy new max_allowed_packet configuration to m1, m2 and m5 dbs |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12601 and previous config saved to /var/cache/conftool/dbconfig/20200916-084916-marostegui.json |
[production] |
| 08:42 |
<awight> |
finished security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:41 |
<awight@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/FileImporter/src/Services/ImportPlanValidator.php: Security patch for T262628 (duration: 00m 59s) |
[production] |
| 08:41 |
<XioNoX> |
asw-c-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:27 |
<XioNoX> |
asw-b-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:26 |
<awight> |
beginning security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:17 |
<XioNoX> |
asw-a-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:04 |
<akosiaris> |
T187984 Validated that ticket.wikimedia.org works, proceeding with a wider announcement |
[production] |
| 08:02 |
<XioNoX> |
asw2-d-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:49 |
<akosiaris> |
T187984 Switch over ticket.discovery.wmnet to otrs1001 |
[production] |
| 07:48 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:44 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:40 |
<XioNoX> |
asw2-c-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:37 |
<akosiaris> |
T187984 Tested inbound email successfully |
[production] |
| 07:29 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:26 |
<akosiaris> |
T187984 Tested outbound email, switching inbound email configuration and performing tests |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12600 and previous config saved to /var/cache/conftool/dbconfig/20200916-072614-marostegui.json |
[production] |
| 07:22 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:22 |
<jayme@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 07:21 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:12 |
<akosiaris> |
T187984 Disable gravatar in system configuration to avoid leaking agent PII through a 3rd party service |
[production] |
| 07:03 |
<akosiaris> |
T187984 validated that the OTRS installation is functional over SSH |
[production] |
| 07:02 |
<akosiaris> |
T187984 migration script done. Config updates, rebuilds, package upgrades/reinstall and index rebuilds done |
[production] |
| 06:28 |
<godog> |
codfw-prod: bump weight for ms-be2057 - T261633 |
[production] |
| 06:20 |
<kart_> |
Updated cxserver to 2020-08-30-011854-production (T253439, T260557) |
[production] |
| 06:20 |
<XioNoX> |
asw2-b-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 06:15 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:11 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 for the first time with minimum weight T261717', diff saved to https://phabricator.wikimedia.org/P12599 and previous config saved to /var/cache/conftool/dbconfig/20200916-061013-marostegui.json |
[production] |
| 06:08 |
<kartik@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12598 and previous config saved to /var/cache/conftool/dbconfig/20200916-060717-marostegui.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2015 to clone es2031 T261717', diff saved to https://phabricator.wikimedia.org/P12597 and previous config saved to /var/cache/conftool/dbconfig/20200916-055535-marostegui.json |
[production] |
| 05:53 |
<XioNoX> |
asw2-a-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12596 and previous config saved to /var/cache/conftool/dbconfig/20200916-055108-marostegui.json |
[production] |
| 05:50 |
<XioNoX> |
msw1-codfw> request system snapshot slice alternate - T262290 |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add es2027 and es2028 to dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12595 and previous config saved to /var/cache/conftool/dbconfig/20200916-053918-marostegui.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12594 and previous config saved to /var/cache/conftool/dbconfig/20200916-053507-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 into vslow', diff saved to https://phabricator.wikimedia.org/P12593 and previous config saved to /var/cache/conftool/dbconfig/20200916-052343-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12592 and previous config saved to /var/cache/conftool/dbconfig/20200916-052241-marostegui.json |
[production] |
| 05:07 |
<marostegui> |
Repool labsdb1010 |
[production] |
| 02:22 |
<mutante> |
deneb - sudo systemctl start package_builder_Clean_up_build_directory to fix icinga alert after failed build attempts |
[production] |