|
2020-09-16
ยง
|
| 12:56 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 12:49 |
<cmjohnson1> |
start pdu swap in racks c6 and c7, d8 |
[production] |
| 12:36 |
<moritzm> |
powercycling mw2256 (went down with overheated CPU) |
[production] |
| 12:29 |
<moritzm> |
restarting exim on MXes to pick up GNUTLS update |
[production] |
| 11:29 |
<moritzm> |
restarting slapd on LDAP replicas to pick up GNUTLS update |
[production] |
| 11:18 |
<moritzm> |
installing gnutls28 security updates on remaining stretch hosts |
[production] |
| 11:12 |
<jforrester@deploy1001> |
Synchronized php-1.36.0-wmf.9/includes/filerepo/file: T263014 Revert "Remove support for (Archived|OldLocal)File::userCan without a user" (duration: 01m 04s) |
[production] |
| 10:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully pool es2027 and es2028 T261717', diff saved to https://phabricator.wikimedia.org/P12606 and previous config saved to /var/cache/conftool/dbconfig/20200916-103324-marostegui.json |
[production] |
| 10:20 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.9 |
[production] |
| 10:14 |
<liw@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.9 (duration: 46m 07s) |
[production] |
| 10:10 |
<ema> |
upload python-acme 0.31.0-2wm1 to buster-wikimedia T263006 |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12605 and previous config saved to /var/cache/conftool/dbconfig/20200916-100548-marostegui.json |
[production] |
| 10:01 |
<akosiaris> |
T187984 Shutdown mendelevium. |
[production] |
| 09:43 |
<jynus> |
deploying max_packet_size change to m3 instances, too |
[production] |
| 09:28 |
<liw@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.9 |
[production] |
| 09:26 |
<liw> |
moving train 1.36.0-wmf.9 to testwikis |
[production] |
| 09:22 |
<jynus> |
restarting gerrit service on gerrit1001, unresponsive |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12603 and previous config saved to /var/cache/conftool/dbconfig/20200916-091535-marostegui.json |
[production] |
| 09:13 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 0 - T262290 |
[production] |
| 09:08 |
<XioNoX> |
fasw-c-eqiad> request system snapshot slice alternate member 1 - T262290 |
[production] |
| 08:52 |
<marostegui> |
Stop mysql on db1121, db1123, db1093 and db1109 for PDU work T261454 T261457 |
[production] |
| 08:52 |
<XioNoX> |
asw-d-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:50 |
<jynus> |
deploy new max_allowed_packet configuration to m1, m2 and m5 dbs |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12601 and previous config saved to /var/cache/conftool/dbconfig/20200916-084916-marostegui.json |
[production] |
| 08:42 |
<awight> |
finished security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:41 |
<awight@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/FileImporter/src/Services/ImportPlanValidator.php: Security patch for T262628 (duration: 00m 59s) |
[production] |
| 08:41 |
<XioNoX> |
asw-c-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:27 |
<XioNoX> |
asw-b-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:26 |
<awight> |
beginning security backport for https://phabricator.wikimedia.org/T262628 |
[production] |
| 08:17 |
<XioNoX> |
asw-a-codfw> request system snapshot slice alternate all-members - T262290 |
[production] |
| 08:04 |
<akosiaris> |
T187984 Validated that ticket.wikimedia.org works, proceeding with a wider announcement |
[production] |
| 08:02 |
<XioNoX> |
asw2-d-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:49 |
<akosiaris> |
T187984 Switch over ticket.discovery.wmnet to otrs1001 |
[production] |
| 07:48 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:44 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:40 |
<XioNoX> |
asw2-c-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:37 |
<akosiaris> |
T187984 Tested inbound email successfully |
[production] |
| 07:29 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:26 |
<akosiaris> |
T187984 Tested outbound email, switching inbound email configuration and performing tests |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12600 and previous config saved to /var/cache/conftool/dbconfig/20200916-072614-marostegui.json |
[production] |
| 07:22 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:22 |
<jayme@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 07:21 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:12 |
<akosiaris> |
T187984 Disable gravatar in system configuration to avoid leaking agent PII through a 3rd party service |
[production] |
| 07:03 |
<akosiaris> |
T187984 validated that the OTRS installation is functional over SSH |
[production] |
| 07:02 |
<akosiaris> |
T187984 migration script done. Config updates, rebuilds, package upgrades/reinstall and index rebuilds done |
[production] |
| 06:28 |
<godog> |
codfw-prod: bump weight for ms-be2057 - T261633 |
[production] |
| 06:20 |
<kart_> |
Updated cxserver to 2020-08-30-011854-production (T253439, T260557) |
[production] |
| 06:20 |
<XioNoX> |
asw2-b-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 06:15 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |