|
2020-09-15
ยง
|
| 17:05 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint (duration: 86m 46s) |
[production] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:59 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:57 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 15:38 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@f7cda70]: Fix analytics by-country endpoint |
[production] |
| 15:33 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:33 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:30 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:30 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:28 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:28 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:26 |
<shdubsh> |
manual install prometheus-icinga-exporter upgrade on icinga2001 |
[production] |
| 14:53 |
<godog> |
switch grafana to eqiad - T259143 |
[production] |
| 14:48 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:42 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:38 |
<XioNoX> |
remove old SNMP community from all network devices |
[production] |
| 14:23 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 56s) |
[production] |
| 14:21 |
<otto@deploy1001> |
sync-file aborted: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 06s) |
[production] |
| 14:01 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 14:01 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:51 |
<cdanis@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:51 |
<cdanis@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 13:50 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:18 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 13:14 |
<cmjohnson1> |
beginning work inside racks c2, c3, c4 and c5 eqiad |
[production] |
| 12:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 from vslow, s8, add db1092 temporarily', diff saved to https://phabricator.wikimedia.org/P12589 and previous config saved to /var/cache/conftool/dbconfig/20200915-121849-marostegui.json |
[production] |
| 12:18 |
<jbond42> |
update libxml2 on stretch and jessie |
[production] |
| 12:08 |
<jbond42> |
rolling restart of php7.2-fpm |
[production] |
| 12:05 |
<elukey> |
roll restart cassandra on aqs* to pick up openjdk upgrades |
[production] |
| 12:05 |
<elukey@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 11:44 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 294931fc6eb9e365894ec0cf94c155d55ecae549: Revert "Disable DynamicPageList on ruwikinews" (T262240; T262391) (duration: 00m 58s) |
[production] |
| 11:17 |
<effie> |
roll out scap 3.15.0-1 to all - T261234 |
[production] |
| 11:12 |
<XioNoX> |
mass update SCS SNMP community in LibreNMS - T246890 |
[production] |
| 10:58 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 10:56 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:54 |
<XioNoX> |
mass update PDU SNMP community in LibreNMS - T246890 |
[production] |
| 10:48 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 10:36 |
<moritzm> |
uploaded libxml2 2.9.1+dfsg1-5+deb8u8+wmf1 for jessie-wikimedia |
[production] |
| 10:33 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 10:22 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "testwikiswikis to 1.36.0-wmf.9" |
[production] |
| 10:12 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 09:22 |
<marostegui> |
Stop MySQL on s5 and s8 eqiad primary master - lag will show up on labsdb hosts T261455 |
[production] |
| 09:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 09:08 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 09:05 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 09:05 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:04 |
<gehel> |
restart elasticsearch on elastic2029 (high GC |
[production] |