|
2020-09-16
§
|
| 08:04 |
<akosiaris> |
T187984 Validated that ticket.wikimedia.org works, proceeding with a wider announcement |
[production] |
| 08:02 |
<XioNoX> |
asw2-d-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:49 |
<akosiaris> |
T187984 Switch over ticket.discovery.wmnet to otrs1001 |
[production] |
| 07:48 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:44 |
<jayme@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:40 |
<XioNoX> |
asw2-c-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 07:37 |
<akosiaris> |
T187984 Tested inbound email successfully |
[production] |
| 07:29 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 07:26 |
<akosiaris> |
T187984 Tested outbound email, switching inbound email configuration and performing tests |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 with more weight T261717', diff saved to https://phabricator.wikimedia.org/P12600 and previous config saved to /var/cache/conftool/dbconfig/20200916-072614-marostegui.json |
[production] |
| 07:22 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:22 |
<jayme@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 07:21 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:12 |
<akosiaris> |
T187984 Disable gravatar in system configuration to avoid leaking agent PII through a 3rd party service |
[production] |
| 07:03 |
<akosiaris> |
T187984 validated that the OTRS installation is functional over SSH |
[production] |
| 07:02 |
<akosiaris> |
T187984 migration script done. Config updates, rebuilds, package upgrades/reinstall and index rebuilds done |
[production] |
| 06:28 |
<godog> |
codfw-prod: bump weight for ms-be2057 - T261633 |
[production] |
| 06:20 |
<kart_> |
Updated cxserver to 2020-08-30-011854-production (T253439, T260557) |
[production] |
| 06:20 |
<XioNoX> |
asw2-b-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 06:15 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:11 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es2027 and es2028 for the first time with minimum weight T261717', diff saved to https://phabricator.wikimedia.org/P12599 and previous config saved to /var/cache/conftool/dbconfig/20200916-061013-marostegui.json |
[production] |
| 06:08 |
<kartik@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12598 and previous config saved to /var/cache/conftool/dbconfig/20200916-060717-marostegui.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2015 to clone es2031 T261717', diff saved to https://phabricator.wikimedia.org/P12597 and previous config saved to /var/cache/conftool/dbconfig/20200916-055535-marostegui.json |
[production] |
| 05:53 |
<XioNoX> |
asw2-a-eqiad> request system snapshot slice alternate all-members - T262290 |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12596 and previous config saved to /var/cache/conftool/dbconfig/20200916-055108-marostegui.json |
[production] |
| 05:50 |
<XioNoX> |
msw1-codfw> request system snapshot slice alternate - T262290 |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add es2027 and es2028 to dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12595 and previous config saved to /var/cache/conftool/dbconfig/20200916-053918-marostegui.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12594 and previous config saved to /var/cache/conftool/dbconfig/20200916-053507-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 into vslow', diff saved to https://phabricator.wikimedia.org/P12593 and previous config saved to /var/cache/conftool/dbconfig/20200916-052343-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12592 and previous config saved to /var/cache/conftool/dbconfig/20200916-052241-marostegui.json |
[production] |
| 05:07 |
<marostegui> |
Repool labsdb1010 |
[production] |
| 02:22 |
<mutante> |
deneb - sudo systemctl start package_builder_Clean_up_build_directory to fix icinga alert after failed build attempts |
[production] |
|
2020-09-15
§
|
| 23:20 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 1c0b0d161fe1024d6d08a27bbacf5b62c56c9c01: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 56s) |
[production] |
| 23:18 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 5beace32a396adfcce46b04e7f969b2f9f9effda: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 58s) |
[production] |
| 23:14 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: ac8bd3894f2dc8f2735cc9fa7b860af1d91c6707: flaggedrevs: Remove non-existent config options (duration: 00m 58s) |
[production] |
| 23:07 |
<urbanecm@deploy1001> |
Scap failed!: Call to mwscript eval.php stderr: not empty |
[production] |
| 23:00 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 62b21d55a8f0a94b8cd268d5024df0cf64013dd5: Revert "Remove abusefilter-view right grant from wmf-config" (T255506) (duration: 00m 59s) |
[production] |
| 20:44 |
<brennen> |
removing extraneous recursive symlink /srv/mediawiki-staging/php-1.36.0-wmf.9/php-1.36.0-wmf.8 |
[production] |
| 18:32 |
<Urbanecm> |
Morning B&C done |
[production] |
| 18:28 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 084729b7fd0716f11265f1b37570afc120b27109: Remove abusefilter-view right grant from wmf-config (T255506) (duration: 00m 56s) |
[production] |
| 18:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 1d3456570b80b1d8af1d2b71975496e54f87b24e: Enable MediaWiki client errors on frwiki (T255585) (duration: 00m 57s) |
[production] |
| 18:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 79004b7e503c7274fa56d2699b423b6919fbc869: Enable the reverted tag on all wikis (T164307) (duration: 00m 56s) |
[production] |
| 17:59 |
<krinkle@deploy1001> |
Synchronized src/ServiceConfig.php: If727ae4335 (duration: 00m 56s) |
[production] |
| 17:43 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out (duration: 37m 42s) |
[production] |
| 17:05 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out |
[production] |
| 17:05 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint (duration: 86m 46s) |
[production] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:59 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |