|
2020-04-02
§
|
| 10:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:55 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:19 |
<marostegui> |
Deploy schema change on db1087, this will generate lag on s8 on wiki replicas |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for schema change', diff saved to https://phabricator.wikimedia.org/P10857 and previous config saved to /var/cache/conftool/dbconfig/20200402-101920-marostegui.json |
[production] |
| 10:17 |
<elukey> |
set up TLS encryption for all pmacct instances on netflow* to Kafka Jumbo |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104 after schema change', diff saved to https://phabricator.wikimedia.org/P10856 and previous config saved to /var/cache/conftool/dbconfig/20200402-101747-marostegui.json |
[production] |
| 09:47 |
<marostegui> |
Remove haproxy@10.64.37.14 from labsdb hosts - T231280 T248944 |
[production] |
| 09:40 |
<gehel> |
CORRECTION: depool maps2004 for data reimport - T249086 |
[production] |
| 09:40 |
<gehel> |
depool wdqs2004 for data reimport - T249086 |
[production] |
| 09:33 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) (duration: 00m 18s) |
[production] |
| 09:32 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@4f86d77]: (no justification provided) (duration: 00m 09s) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@4f86d77]: (no justification provided) |
[production] |
| 08:51 |
<marostegui> |
Deploy schema change db1104 |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104 for schema change', diff saved to https://phabricator.wikimedia.org/P10854 and previous config saved to /var/cache/conftool/dbconfig/20200402-085057-marostegui.json |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P10853 and previous config saved to /var/cache/conftool/dbconfig/20200402-085019-marostegui.json |
[production] |
| 08:28 |
<gehel> |
repooling wdqs1006 - catched up on lag |
[production] |
| 08:22 |
<vgutierrez> |
Enable inbound TLSv1.3 in upload@esams - T170567 |
[production] |
| 08:21 |
<vgutierrez> |
Enable TLS Session tickets in esams - T245616 |
[production] |
| 07:45 |
<moritzm> |
bounced ferm on ms-be1040 |
[production] |
| 07:27 |
<marostegui> |
Deploy schema change on db1092 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json |
[production] |
| 05:49 |
<marostegui> |
Deploy schema change on db1101:3318 |
[production] |
| 05:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json |
[production] |
| 05:29 |
<elukey> |
powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) |
[production] |
|
2020-04-01
§
|
| 22:44 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 22:43 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: |
[production] |
| 22:02 |
<volans> |
forcing logrotate on netflow2001 to compress yesterday's logs |
[production] |
| 21:53 |
<volans> |
force-rebooting ms-be1023, unresponsive - T249174 |
[production] |
| 21:50 |
<volans> |
stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop |
[production] |
| 19:48 |
<marxarelli> |
rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 |
[production] |
| 19:30 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 |
[production] |
| 19:21 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) |
[production] |
| 19:20 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 |
[production] |
| 19:18 |
<marxarelli> |
promoting group1 to 1.35.0-wmf.26 to group1 |
[production] |
| 17:21 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' |
[production] |
| 16:54 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' |
[production] |
| 16:39 |
<vgutierrez> |
pool cp2027 - T248816 |
[production] |
| 16:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:17 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@21363c1]: page range prefetch fixup (duration: 00m 09s) |
[production] |
| 16:17 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@21363c1]: page range prefetch fixup |
[production] |
| 15:33 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:27 |
<vgutierrez> |
depool & decommission cp20[16,19,23,27] - T249125 |
[production] |