|
2020-01-08
ยง
|
| 18:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 18:04 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 18:03 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 18:03 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 16:25 |
<_joe_> |
running puppet on deploy1001 to remove my hot-patch to scap.cfg |
[production] |
| 16:20 |
<ema> |
rolling ats-be restart on !text@eqiad, !text@esams to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562849/ |
[production] |
| 16:00 |
<bblack> |
re-pooling esams text traffic in DNS |
[production] |
| 15:45 |
<ema> |
cumin -s10 -b1 'A:cp-text_eqiad' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:40 |
<vgutierrez> |
restarting ats-tls on esams text nodes |
[production] |
| 15:37 |
<ema> |
cumin -s10 -b1 'A:cp-text_esams' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:37 |
<bblack> |
authdns-update to depool esams |
[production] |
| 15:26 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 34s) |
[production] |
| 15:23 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 03m 56s) |
[production] |
| 15:19 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 33s) |
[production] |
| 15:12 |
<otto@deploy1001> |
Scap failed!: 4/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 15:11 |
<otto@deploy1001> |
sync-file aborted: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 00s) |
[production] |
| 15:10 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 10s) |
[production] |
| 15:02 |
<XioNoX> |
Routinator 0.6.4 looking good on rpki2001, upgrading rpki1001 - T242197 |
[production] |
| 15:00 |
<ottomata> |
deploying change to use new TLS port for eventgate-analytics - T242224 |
[production] |
| 14:35 |
<ema> |
repool cp4028 after successful X-Analytics-TLS patch test T237993 |
[production] |
| 14:23 |
<ema> |
depool cp4028 to test X-Analytics-TLS patch T237993 |
[production] |
| 14:07 |
<XioNoX> |
add routinator 0.6.4 to reprepro stretch-wikimedia - T242197 |
[production] |
| 14:00 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@dbd0ecd]: don't regenerate existing 7z files on rerun of the 7z recompression job (duration: 00m 05s) |
[production] |
| 14:00 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@dbd0ecd]: don't regenerate existing 7z files on rerun of the 7z recompression job |
[production] |
| 12:46 |
<_joe_> |
deleting releng/composer-php55:0.1.0 from the docker registry |
[production] |
| 12:36 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 12:34 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:510875|Update Skolt Sami language name (T223544)]] (duration: 01m 06s) |
[production] |
| 12:30 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.11/extensions/Cite: SWAT: [[gerrit:561169|Fix handling of `<references responsive="" />` (T241303)]] (duration: 01m 06s) |
[production] |
| 12:17 |
<tarrow@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562777|Enable tainted references on test.wikidata.org (T239621)]] (duration: 01m 19s) |
[production] |
| 12:08 |
<kart_> |
Updated cxserver to 2020-01-06-070550-production (T233405) |
[production] |
| 12:04 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:01 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:00 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:47 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubernetes2001.* |
[production] |
| 11:45 |
<akosiaris@cumin1001> |
conftool action : set/weight=10; selector: service=echostore |
[production] |
| 11:44 |
<vgutierrez> |
uploaded varnish 5.1.3-1wm12 to apt.wikimedia.org (buster) - T242093 |
[production] |
| 11:44 |
<akosiaris@cumin1001> |
conftool action : set/weight=10; selector: name=kubernetes1001.* |
[production] |
| 11:44 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubernetes1001.* |
[production] |
| 11:07 |
<moritzm> |
test failover of Ganeti master in eqsin T228099 |
[production] |
| 11:00 |
<moritzm> |
drain ganeti5003 to test new Ganeti setup in eqsin T228099 |
[production] |
| 10:53 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:53 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:41 |
<moritzm> |
rebooting netflow5001 to pick up microcode |
[production] |
| 10:08 |
<moritzm> |
enabling spec-ctr, ssbd. md-clear passthrough for new eqsin cluster T228099 |
[production] |
| 09:27 |
<moritzm> |
installing urldownloader1002 T241979 |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1085', diff saved to https://phabricator.wikimedia.org/P10088 and previous config saved to /var/cache/conftool/dbconfig/20200108-091124-marostegui.json |
[production] |
| 09:00 |
<moritzm> |
installing urldownloader1001 T241979 |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P10087 and previous config saved to /var/cache/conftool/dbconfig/20200108-082930-marostegui.json |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P10086 and previous config saved to /var/cache/conftool/dbconfig/20200108-082050-marostegui.json |
[production] |
| 08:09 |
<marostegui> |
Upgrade db1085 |
[production] |