251-300 of 10000 results (22ms)
2020-09-16 ยง
12:36 <moritzm> powercycling mw2256 (went down with overheated CPU) [production]
12:29 <moritzm> restarting exim on MXes to pick up GNUTLS update [production]
11:29 <moritzm> restarting slapd on LDAP replicas to pick up GNUTLS update [production]
11:18 <moritzm> installing gnutls28 security updates on remaining stretch hosts [production]
11:12 <jforrester@deploy1001> Synchronized php-1.36.0-wmf.9/includes/filerepo/file: T263014 Revert "Remove support for (Archived|OldLocal)File::userCan without a user" (duration: 01m 04s) [production]
10:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool es2027 and es2028 T261717', diff saved to https://phabricator.wikimedia.org/P12606 and previous config saved to /var/cache/conftool/dbconfig/20200916-103324-marostegui.json [production]
10:20 <liw@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.9 [production]
10:14 <liw@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.9 (duration: 46m 07s) [production]
10:10 <ema> upload python-acme 0.31.0-2wm1 to buster-wikimedia T263006 [production]
10:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12605 and previous config saved to /var/cache/conftool/dbconfig/20200916-100548-marostegui.json [production]
10:01 <akosiaris> T187984 Shutdown mendelevium. [production]
09:43 <jynus> deploying max_packet_size change to m3 instances, too [production]
09:28 <liw@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.9 [production]
09:26 <liw> moving train 1.36.0-wmf.9 to testwikis [production]
09:22 <jynus> restarting gerrit service on gerrit1001, unresponsive [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12603 and previous config saved to /var/cache/conftool/dbconfig/20200916-091535-marostegui.json [production]
09:13 <XioNoX> fasw-c-eqiad> request system snapshot slice alternate member 0 - T262290 [production]
09:08 <XioNoX> fasw-c-eqiad> request system snapshot slice alternate member 1 - T262290 [production]
08:52 <marostegui> Stop mysql on db1121, db1123, db1093 and db1109 for PDU work T261454 T261457 [production]
08:52 <XioNoX> asw-d-codfw> request system snapshot slice alternate all-members - T262290 [production]
08:50 <jynus> deploy new max_allowed_packet configuration to m1, m2 and m5 dbs [production]
08:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12601 and previous config saved to /var/cache/conftool/dbconfig/20200916-084916-marostegui.json [production]
08:42 <awight> finished security backport for https://phabricator.wikimedia.org/T262628 [production]
08:41 <awight@deploy1001> Synchronized php-1.36.0-wmf.8/extensions/FileImporter/src/Services/ImportPlanValidator.php: Security patch for T262628 (duration: 00m 59s) [production]
08:41 <XioNoX> asw-c-codfw> request system snapshot slice alternate all-members - T262290 [production]
08:27 <XioNoX> asw-b-codfw> request system snapshot slice alternate all-members - T262290 [production]
08:26 <awight> beginning security backport for https://phabricator.wikimedia.org/T262628 [production]
08:17 <XioNoX> asw-a-codfw> request system snapshot slice alternate all-members - T262290 [production]
08:04 <akosiaris> T187984 Validated that ticket.wikimedia.org works, proceeding with a wider announcement [production]
08:02 <XioNoX> asw2-d-eqiad> request system snapshot slice alternate all-members - T262290 [production]
07:49 <akosiaris> T187984 Switch over ticket.discovery.wmnet to otrs1001 [production]
07:48 <jayme@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:44 <jayme@cumin1001> START - Cookbook sre.dns.netbox [production]
07:40 <XioNoX> asw2-c-eqiad> request system snapshot slice alternate all-members - T262290 [production]
07:37 <akosiaris> T187984 Tested inbound email successfully [production]
07:29 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
07:26 <akosiaris> T187984 Tested outbound email, switching inbound email configuration and performing tests [production]
07:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12600 and previous config saved to /var/cache/conftool/dbconfig/20200916-072614-marostegui.json [production]
07:22 <jayme@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:22 <jayme@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
07:21 <jayme@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:12 <akosiaris> T187984 Disable gravatar in system configuration to avoid leaking agent PII through a 3rd party service [production]
07:03 <akosiaris> T187984 validated that the OTRS installation is functional over SSH [production]
07:02 <akosiaris> T187984 migration script done. Config updates, rebuilds, package upgrades/reinstall and index rebuilds done [production]
06:28 <godog> codfw-prod: bump weight for ms-be2057 - T261633 [production]
06:20 <kart_> Updated cxserver to 2020-08-30-011854-production (T253439, T260557) [production]
06:20 <XioNoX> asw2-b-eqiad> request system snapshot slice alternate all-members - T262290 [production]
06:15 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
06:11 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es2027 and es2028 for the first time with minimum weight T261717', diff saved to https://phabricator.wikimedia.org/P12599 and previous config saved to /var/cache/conftool/dbconfig/20200916-061013-marostegui.json [production]