|
2020-01-27
§
|
| 17:46 |
<zhuyifei1999_> |
restarted webservice via `webservice --backend kubernetes php7.3 stop` `webservice --backend kubernetes php7.3 start` T115231 |
[tools.dplbot] |
| 13:58 |
<vgutierrez> |
repooling cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
restarting varnish-fe on cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
repooling cp4029 - T243634 |
[production] |
| 13:36 |
<vgutierrez> |
restarting varnish-fe on cp4029 - T243634 |
[production] |
| 12:45 |
<arturo> |
[codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]> update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` T243556 |
[admin] |
| 12:44 |
<arturo> |
[codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description "main DNS domain for VMs" --email "root@wmflabs.org" --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` T243556 |
[admin] |
| 12:10 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --from-id 1860 --to-id 1860 (T243705) |
[production] |
| 07:05 |
<zhuyifei1999_> |
wrong package. uninstalled. the correct one is bpfcc-tools and seems only available in buster+. T115231 |
[tools] |
| 07:01 |
<zhuyifei1999_> |
apt installing bcc on tools-worker-1037 to see who is sending SIGTERM, will uninstall after done. dependency: bin86. T115231 |
[tools] |
| 05:38 |
<elukey> |
re-run webrequest text 2020-01-26T20/21 with higher dataloss thresholds (false positives) |
[analytics] |
| 03:29 |
<gehel> |
restarting blazegraph on wdqs100[57] |
[production] |
| 02:49 |
<elukey> |
re-run refine eventlogging manually to clear out refine failed events |
[analytics] |
|
2020-01-26
§
|
| 23:23 |
<James_F> |
Zuul: Mark mediawiki/extensions/BlueSpiceMenues as archived T240788 |
[releng] |
| 22:20 |
<James_F> |
Zuul: Mark mediawiki/extensions/SemanticTasks as archived T221549 |
[releng] |
| 21:45 |
<akosiaris> |
repool maps1003 |
[production] |
| 21:45 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: name=maps1003.* |
[production] |
| 21:42 |
<akosiaris> |
test depool maps1003 |
[production] |
| 21:42 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: name=maps1003.* |
[production] |
| 21:38 |
<vgutierrez> |
powercycling cp3051 - T238305 |
[production] |
| 21:23 |
<akosiaris> |
restart kartotherian on maps1002 |
[production] |
| 21:19 |
<vgutierrez> |
restart varnish-fe and ats-tls on cp3056 |
[production] |
| 21:02 |
<bblack> |
ats-tls-restart on cp3064 |
[production] |
| 20:51 |
<bblack> |
esams text caches: reverting earlier sysctl mitigations |
[production] |
| 18:51 |
<hauskatze> |
maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php ep[campus,online,coordinator,instructor] | T241504 |
[releng] |
| 18:11 |
<volans> |
shutdown elastic2043 - T243715 |
[production] |
| 18:01 |
<volans> |
depooled elastic2043 - T243715 |
[production] |
| 18:01 |
<volans@cumin1001> |
conftool action : set/pooled=inactive; selector: name=elastic2043.codfw.wmnet |
[production] |
| 17:58 |
<elukey> |
re-run failed refine job for MobileWebUIActionsTracking 2020-01-26T12 |
[analytics] |
| 17:32 |
<elukey> |
restart varnishkafka on cp3056/cp3064 due to network issues on the hosts |
[analytics] |
| 17:28 |
<elukey> |
restart varnishkafka-webrequest on cp3064 |
[production] |
| 17:25 |
<elukey> |
restart varnishkafka-webrequest on cp3056 |
[production] |
| 17:03 |
<bblack> |
reduce /proc/sys/net/ipv4/tcp_max_syn_backlog to 8192 on esams text caches |
[production] |
| 16:55 |
<bblack> |
reduce /proc/sys/net/ipv4/tcp_synack_retries to 1 on esams text caches |
[production] |
| 16:42 |
<cdanis> |
✔️ cdanis@cp4030.ulsfo.wmnet ~ 🕦☕ sudo depool |
[production] |
| 16:38 |
<bblack> |
applying GRE MTU mitigation from T232602 to all cp1, cp3, cp5 cache nodes |
[production] |
| 15:43 |
<XioNoX> |
3*prepend in esams/knams |
[production] |
| 15:26 |
<elukey> |
repool deployed |
[production] |
| 15:24 |
<elukey> |
repool esams |
[production] |
| 15:01 |
<cdanis> |
deployed |
[production] |
| 15:00 |
<cdanis> |
depool esams |
[production] |
| 14:56 |
<XioNoX> |
enabling netflow sampling on the knams-esams links (esams side) |
[production] |
| 11:25 |
<effie> |
restarted tilerator and tileratorui on maps1002 |
[production] |
| 11:23 |
<effie> |
restarted tilerator and tileratorui on maps1001 |
[production] |
| 11:02 |
<hauskatze> |
github - deleted mirror for extension-BlueSpiceMenues; refs. T240788 |
[releng] |
| 10:38 |
<effie> |
deployed |
[production] |
| 10:37 |
<effie> |
Pool esams back |
[production] |
| 01:12 |
<cdanis> |
deployed |
[production] |
| 01:12 |
<cdanis> |
depool esams with new geo-maps-esams-offline |
[production] |