|
2020-01-27
§
|
| 23:10 |
<vgutierrez> |
rolling restart of varnish-frontend in cp4026 and cp4027 |
[production] |
| 23:06 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:06 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:01 |
<_joe_> |
restart apache on gerrit |
[production] |
| 22:58 |
<vgutierrez> |
restarting gerrit service |
[production] |
| 22:01 |
<vgutierrez> |
restarting varnish-fe on cp4028 |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3311 - T239453', diff saved to https://phabricator.wikimedia.org/P10277 and previous config saved to /var/cache/conftool/dbconfig/20200127-191614-marostegui.json |
[production] |
| 19:15 |
<marostegui> |
Remove partitions from db2085 enwiki - T239453 |
[production] |
| 13:58 |
<vgutierrez> |
repooling cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
restarting varnish-fe on cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
repooling cp4029 - T243634 |
[production] |
| 13:36 |
<vgutierrez> |
restarting varnish-fe on cp4029 - T243634 |
[production] |
| 12:10 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --from-id 1860 --to-id 1860 (T243705) |
[production] |
| 03:29 |
<gehel> |
restarting blazegraph on wdqs100[57] |
[production] |
|
2020-01-26
§
|
| 21:45 |
<akosiaris> |
repool maps1003 |
[production] |
| 21:45 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: name=maps1003.* |
[production] |
| 21:42 |
<akosiaris> |
test depool maps1003 |
[production] |
| 21:42 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: name=maps1003.* |
[production] |
| 21:38 |
<vgutierrez> |
powercycling cp3051 - T238305 |
[production] |
| 21:23 |
<akosiaris> |
restart kartotherian on maps1002 |
[production] |
| 21:19 |
<vgutierrez> |
restart varnish-fe and ats-tls on cp3056 |
[production] |
| 21:02 |
<bblack> |
ats-tls-restart on cp3064 |
[production] |
| 20:51 |
<bblack> |
esams text caches: reverting earlier sysctl mitigations |
[production] |
| 18:11 |
<volans> |
shutdown elastic2043 - T243715 |
[production] |
| 18:01 |
<volans> |
depooled elastic2043 - T243715 |
[production] |
| 18:01 |
<volans@cumin1001> |
conftool action : set/pooled=inactive; selector: name=elastic2043.codfw.wmnet |
[production] |
| 17:28 |
<elukey> |
restart varnishkafka-webrequest on cp3064 |
[production] |
| 17:25 |
<elukey> |
restart varnishkafka-webrequest on cp3056 |
[production] |
| 17:03 |
<bblack> |
reduce /proc/sys/net/ipv4/tcp_max_syn_backlog to 8192 on esams text caches |
[production] |
| 16:55 |
<bblack> |
reduce /proc/sys/net/ipv4/tcp_synack_retries to 1 on esams text caches |
[production] |
| 16:42 |
<cdanis> |
✔️ cdanis@cp4030.ulsfo.wmnet ~ 🕦☕ sudo depool |
[production] |
| 16:38 |
<bblack> |
applying GRE MTU mitigation from T232602 to all cp1, cp3, cp5 cache nodes |
[production] |
| 15:43 |
<XioNoX> |
3*prepend in esams/knams |
[production] |
| 15:26 |
<elukey> |
repool deployed |
[production] |
| 15:24 |
<elukey> |
repool esams |
[production] |
| 15:01 |
<cdanis> |
deployed |
[production] |
| 15:00 |
<cdanis> |
depool esams |
[production] |
| 14:56 |
<XioNoX> |
enabling netflow sampling on the knams-esams links (esams side) |
[production] |
| 11:25 |
<effie> |
restarted tilerator and tileratorui on maps1002 |
[production] |
| 11:23 |
<effie> |
restarted tilerator and tileratorui on maps1001 |
[production] |
| 10:38 |
<effie> |
deployed |
[production] |
| 10:37 |
<effie> |
Pool esams back |
[production] |
| 01:12 |
<cdanis> |
deployed |
[production] |
| 01:12 |
<cdanis> |
depool esams with new geo-maps-esams-offline |
[production] |