|
2020-04-02
§
|
| 14:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10863 and previous config saved to /var/cache/conftool/dbconfig/20200402-141335-marostegui.json |
[production] |
| 14:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10862 and previous config saved to /var/cache/conftool/dbconfig/20200402-141149-marostegui.json |
[production] |
| 13:50 |
<marostegui> |
Compress wbqc_constraints on testcommonswiki and commonswiki (empty tables) - T248967 |
[production] |
| 13:44 |
<vgutierrez> |
update puppet compiler facts |
[production] |
| 13:40 |
<marostegui> |
Deploy schema change on db1111 |
[production] |
| 13:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111 for schema change', diff saved to https://phabricator.wikimedia.org/P10861 and previous config saved to /var/cache/conftool/dbconfig/20200402-133956-marostegui.json |
[production] |
| 13:32 |
<gehel> |
OSM data reimport on maps2004 - T249086 |
[production] |
| 12:55 |
<mutante> |
mw1390 - mw1399 - pooled and active but status "staged" in netbox, fixing to 'active' |
[production] |
| 12:52 |
<mutante> |
mw1297 - is pooled and serving traffic but status "staged" in netbox. set to "active" |
[production] |
| 11:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 after schema change', diff saved to https://phabricator.wikimedia.org/P10858 and previous config saved to /var/cache/conftool/dbconfig/20200402-114020-marostegui.json |
[production] |
| 10:56 |
<mutante> |
decom planet1001 (T248863) |
[production] |
| 10:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:55 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:19 |
<marostegui> |
Deploy schema change on db1087, this will generate lag on s8 on wiki replicas |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for schema change', diff saved to https://phabricator.wikimedia.org/P10857 and previous config saved to /var/cache/conftool/dbconfig/20200402-101920-marostegui.json |
[production] |
| 10:17 |
<elukey> |
set up TLS encryption for all pmacct instances on netflow* to Kafka Jumbo |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104 after schema change', diff saved to https://phabricator.wikimedia.org/P10856 and previous config saved to /var/cache/conftool/dbconfig/20200402-101747-marostegui.json |
[production] |
| 09:47 |
<marostegui> |
Remove haproxy@10.64.37.14 from labsdb hosts - T231280 T248944 |
[production] |
| 09:40 |
<gehel> |
CORRECTION: depool maps2004 for data reimport - T249086 |
[production] |
| 09:40 |
<gehel> |
depool wdqs2004 for data reimport - T249086 |
[production] |
| 09:33 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) (duration: 00m 18s) |
[production] |
| 09:32 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@4f86d77]: (no justification provided) (duration: 00m 09s) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@4f86d77]: (no justification provided) |
[production] |
| 08:51 |
<marostegui> |
Deploy schema change db1104 |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104 for schema change', diff saved to https://phabricator.wikimedia.org/P10854 and previous config saved to /var/cache/conftool/dbconfig/20200402-085057-marostegui.json |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P10853 and previous config saved to /var/cache/conftool/dbconfig/20200402-085019-marostegui.json |
[production] |
| 08:28 |
<gehel> |
repooling wdqs1006 - catched up on lag |
[production] |
| 08:22 |
<vgutierrez> |
Enable inbound TLSv1.3 in upload@esams - T170567 |
[production] |
| 08:21 |
<vgutierrez> |
Enable TLS Session tickets in esams - T245616 |
[production] |
| 07:45 |
<moritzm> |
bounced ferm on ms-be1040 |
[production] |
| 07:27 |
<marostegui> |
Deploy schema change on db1092 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json |
[production] |
| 05:49 |
<marostegui> |
Deploy schema change on db1101:3318 |
[production] |
| 05:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json |
[production] |
| 05:29 |
<elukey> |
powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) |
[production] |
|
2020-04-01
§
|
| 22:44 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 22:43 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: |
[production] |
| 22:02 |
<volans> |
forcing logrotate on netflow2001 to compress yesterday's logs |
[production] |
| 21:53 |
<volans> |
force-rebooting ms-be1023, unresponsive - T249174 |
[production] |
| 21:50 |
<volans> |
stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop |
[production] |
| 19:48 |
<marxarelli> |
rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 |
[production] |
| 19:30 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 |
[production] |
| 19:21 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) |
[production] |
| 19:20 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 |
[production] |
| 19:18 |
<marxarelli> |
promoting group1 to 1.35.0-wmf.26 to group1 |
[production] |
| 17:21 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' |
[production] |
| 16:54 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' |
[production] |
| 16:39 |
<vgutierrez> |
pool cp2027 - T248816 |
[production] |