|
2020-09-16
§
|
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12596 and previous config saved to /var/cache/conftool/dbconfig/20200916-055108-marostegui.json |
[production] |
| 05:50 |
<XioNoX> |
msw1-codfw> request system snapshot slice alternate - T262290 |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add es2027 and es2028 to dbctl T261717', diff saved to https://phabricator.wikimedia.org/P12595 and previous config saved to /var/cache/conftool/dbconfig/20200916-053918-marostegui.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12594 and previous config saved to /var/cache/conftool/dbconfig/20200916-053507-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 into vslow', diff saved to https://phabricator.wikimedia.org/P12593 and previous config saved to /var/cache/conftool/dbconfig/20200916-052343-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es2011 and es2017 after cloning es2027 and es2028', diff saved to https://phabricator.wikimedia.org/P12592 and previous config saved to /var/cache/conftool/dbconfig/20200916-052241-marostegui.json |
[production] |
| 05:07 |
<marostegui> |
Repool labsdb1010 |
[production] |
| 02:22 |
<mutante> |
deneb - sudo systemctl start package_builder_Clean_up_build_directory to fix icinga alert after failed build attempts |
[production] |
|
2020-09-15
§
|
| 23:20 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 1c0b0d161fe1024d6d08a27bbacf5b62c56c9c01: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 56s) |
[production] |
| 23:18 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 5beace32a396adfcce46b04e7f969b2f9f9effda: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 58s) |
[production] |
| 23:14 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: ac8bd3894f2dc8f2735cc9fa7b860af1d91c6707: flaggedrevs: Remove non-existent config options (duration: 00m 58s) |
[production] |
| 23:07 |
<urbanecm@deploy1001> |
Scap failed!: Call to mwscript eval.php stderr: not empty |
[production] |
| 23:00 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 62b21d55a8f0a94b8cd268d5024df0cf64013dd5: Revert "Remove abusefilter-view right grant from wmf-config" (T255506) (duration: 00m 59s) |
[production] |
| 20:44 |
<brennen> |
removing extraneous recursive symlink /srv/mediawiki-staging/php-1.36.0-wmf.9/php-1.36.0-wmf.8 |
[production] |
| 18:32 |
<Urbanecm> |
Morning B&C done |
[production] |
| 18:28 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 084729b7fd0716f11265f1b37570afc120b27109: Remove abusefilter-view right grant from wmf-config (T255506) (duration: 00m 56s) |
[production] |
| 18:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 1d3456570b80b1d8af1d2b71975496e54f87b24e: Enable MediaWiki client errors on frwiki (T255585) (duration: 00m 57s) |
[production] |
| 18:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 79004b7e503c7274fa56d2699b423b6919fbc869: Enable the reverted tag on all wikis (T164307) (duration: 00m 56s) |
[production] |
| 17:59 |
<krinkle@deploy1001> |
Synchronized src/ServiceConfig.php: If727ae4335 (duration: 00m 56s) |
[production] |
| 17:43 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out (duration: 37m 42s) |
[production] |
| 17:05 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint, feeds time out |
[production] |
| 17:05 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint (duration: 86m 46s) |
[production] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:59 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:57 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 15:38 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint |
[production] |
| 15:33 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:33 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:30 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:30 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:28 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:28 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:26 |
<shdubsh> |
manual install prometheus-icinga-exporter upgrade on icinga2001 |
[production] |
| 14:53 |
<godog> |
switch grafana to eqiad - T259143 |
[production] |
| 14:48 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:42 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:38 |
<XioNoX> |
remove old SNMP community from all network devices |
[production] |
| 14:23 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 56s) |
[production] |
| 14:21 |
<otto@deploy1001> |
sync-file aborted: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 06s) |
[production] |
| 14:01 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 14:01 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:51 |
<cdanis@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:51 |
<cdanis@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:18 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 13:14 |
<cmjohnson1> |
beginning work inside racks c2, c3, c4 and c5 eqiad |
[production] |
| 12:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 from vslow, s8, add db1092 temporarily', diff saved to https://phabricator.wikimedia.org/P12589 and previous config saved to /var/cache/conftool/dbconfig/20200915-121849-marostegui.json |
[production] |
| 12:18 |
<jbond42> |
update libxml2 on stretch and jessie |
[production] |