|
2020-04-01
§
|
| 21:53 |
<volans> |
force-rebooting ms-be1023, unresponsive - T249174 |
[production] |
| 21:50 |
<volans> |
stopped and restarted kafkatee-webrequest.service on netflow2001, was in a restart loop |
[production] |
| 19:48 |
<marxarelli> |
rollback of 1.35.0-wmf.26 from group1 (T247773). blocked by T249162 |
[production] |
| 19:30 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: rollback 1.35.0-wmf.26 from group1 |
[production] |
| 19:21 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.26 (duration: 01m 06s) |
[production] |
| 19:20 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.26 |
[production] |
| 19:18 |
<marxarelli> |
promoting group1 to 1.35.0-wmf.26 to group1 |
[production] |
| 17:21 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqord*' commit 'enable sampling on eqord Iac15379cc' |
[production] |
| 16:54 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕐☕ homer 'cr*eqdfw*' commit 'enable sampling on eqdfw Iac15379cc' |
[production] |
| 16:39 |
<vgutierrez> |
pool cp2027 - T248816 |
[production] |
| 16:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:17 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@21363c1]: page range prefetch fixup (duration: 00m 09s) |
[production] |
| 16:17 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@21363c1]: page range prefetch fixup |
[production] |
| 15:33 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 15:31 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:27 |
<vgutierrez> |
depool & decommission cp20[16,19,23,27] - T249125 |
[production] |
| 15:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10845 and previous config saved to /var/cache/conftool/dbconfig/20200401-152258-marostegui.json |
[production] |
| 15:11 |
<herron> |
performing kafka-main rolling restarts to pick up security updates |
[production] |
| 14:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:49 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:49 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:46 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<vgutierrez> |
depool && decommission cp[2018,2020,2022,2024-2026].codfw.wmnet - T249115 |
[production] |
| 14:32 |
<gehel> |
depooling wdqs1006 to allow catching up on lag |
[production] |
| 14:30 |
<vgutierrez> |
pool cp2042 - T248816 |
[production] |
| 14:16 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:13 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:09 |
<XioNoX> |
remove AS-path prepending in esams |
[production] |
| 13:47 |
<XioNoX> |
remove AS-path prepending in eqsin |
[production] |
| 13:39 |
<vgutierrez> |
pool cp2041 - T248816 |
[production] |
| 13:34 |
<mutante> |
sodium (mirror): sudo -u mirror ftpsync to get Debian mirror updated (Icinga says it's old) |
[production] |
| 13:24 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:24 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:17 |
<marostegui> |
Deploy schema change on db1099:3318 |
[production] |
| 13:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10843 and previous config saved to /var/cache/conftool/dbconfig/20200401-131719-marostegui.json |
[production] |
| 13:13 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:10 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:19 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 12:19 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 12:19 |
<tgr@deploy1001> |
Synchronized wmf-config/config: SWAT: [[gerrit:584579|Sync growthexperiments dblist with actual state of wmgUseGrowthExperiments (T248844)]] (duration: 01m 06s) |
[production] |
| 12:18 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:18 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:17 |
<tgr@deploy1001> |
Synchronized dblists/growthexperiments.dblist: SWAT: [[gerrit:584579|Sync growthexperiments dblist with actual state of wmgUseGrowthExperiments (T248844)]] (duration: 01m 05s) |
[production] |