|
2020-04-02
§
|
| 08:21 |
<vgutierrez> |
Enable TLS Session tickets in esams - T245616 |
[production] |
| 07:45 |
<moritzm> |
bounced ferm on ms-be1040 |
[production] |
| 07:27 |
<marostegui> |
Deploy schema change on db1092 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json |
[production] |
| 05:49 |
<marostegui> |
Deploy schema change on db1101:3318 |
[production] |
| 05:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json |
[production] |
| 05:29 |
<elukey> |
powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) |
[production] |
|
2020-04-01
§
|
| 22:44 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 22:43 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@4bfe647]: Deploy design/style-guide: |
[production] |
| 22:02 |
<volans> |
forcing logrotate on netflow2001 to compress yesterday's logs |
[production] |
| 21:53 |
<volans> |
force-rebooting ms-be1023, unresponsive - T249174 |
[production] |
| 21:50 |
<volans> |
stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop |
[production] |
| 19:48 |
<marxarelli> |
rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 |
[production] |
| 19:30 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 |
[production] |
| 19:21 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) |
[production] |
| 19:20 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 |
[production] |
| 19:18 |
<marxarelli> |
promoting group1 to 1.35.0-wmf.26 to group1 |
[production] |
| 17:21 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' |
[production] |
| 16:54 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' |
[production] |
| 16:39 |
<vgutierrez> |
pool cp2027 - T248816 |
[production] |
| 16:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:17 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@21363c1]: page range prefetch fixup (duration: 00m 09s) |
[production] |
| 16:17 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@21363c1]: page range prefetch fixup |
[production] |
| 15:33 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:27 |
<vgutierrez> |
depool & decommission cp20[16,19,23,27] - T249125 |
[production] |
| 15:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10845 and previous config saved to /var/cache/conftool/dbconfig/20200401-152258-marostegui.json |
[production] |
| 15:11 |
<herron> |
performing kafka-main rolling restarts to pick up security updates |
[production] |
| 14:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:49 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:49 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:46 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<vgutierrez> |
depool && decommission cp[2018,2020,2022,2024-2026].codfw.wmnet - T249115 |
[production] |
| 14:32 |
<gehel> |
depooling wdqs1006 to allow catching up on lag |
[production] |
| 14:30 |
<vgutierrez> |
pool cp2042 - T248816 |
[production] |
| 14:16 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:13 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:09 |
<XioNoX> |
remove AS-path prepending in esams |
[production] |
| 13:47 |
<XioNoX> |
remove AS-path prepending in eqsin |
[production] |
| 13:39 |
<vgutierrez> |
pool cp2041 - T248816 |
[production] |
| 13:34 |
<mutante> |
sodium (mirror): sudo -u mirror ftpsync to get Debian mirror updated (Icinga says it's old) |
[production] |
| 13:24 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |