|
2020-01-27
§
|
| 23:40 |
<eileen> |
civicrm revision changed from fbd5c35fb0 to ac730a6bcb, config revision is 837b9d0703 |
[production] |
| 23:23 |
<wm-bot> |
<lucaswerkmeister> deployed 54b9e37118 (more RTL fixes) |
[tools.lexeme-forms] |
| 23:14 |
<wm-bot> |
<lucaswerkmeister> deployed 72ec256823 (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log] |
[tools.lexeme-forms] |
| 23:10 |
<vgutierrez> |
rolling restart of varnish-frontend in cp4026 and cp4027 |
[production] |
| 23:06 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:06 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:01 |
<_joe_> |
restart apache on gerrit |
[production] |
| 22:58 |
<vgutierrez> |
restarting gerrit service |
[production] |
| 22:01 |
<vgutierrez> |
restarting varnish-fe on cp4028 |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3311 - T239453', diff saved to https://phabricator.wikimedia.org/P10277 and previous config saved to /var/cache/conftool/dbconfig/20200127-191614-marostegui.json |
[production] |
| 19:15 |
<marostegui> |
Remove partitions from db2085 enwiki - T239453 |
[production] |
| 17:46 |
<zhuyifei1999_> |
restarted webservice via `webservice --backend kubernetes php7.3 stop` `webservice --backend kubernetes php7.3 start` T115231 |
[tools.dplbot] |
| 13:58 |
<vgutierrez> |
repooling cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
restarting varnish-fe on cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
repooling cp4029 - T243634 |
[production] |
| 13:36 |
<vgutierrez> |
restarting varnish-fe on cp4029 - T243634 |
[production] |
| 12:45 |
<arturo> |
[codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]> update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` T243556 |
[admin] |
| 12:44 |
<arturo> |
[codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description "main DNS domain for VMs" --email "root@wmflabs.org" --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` T243556 |
[admin] |
| 12:10 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --from-id 1860 --to-id 1860 (T243705) |
[production] |
| 07:05 |
<zhuyifei1999_> |
wrong package. uninstalled. the correct one is bpfcc-tools and seems only available in buster+. T115231 |
[tools] |
| 07:01 |
<zhuyifei1999_> |
apt installing bcc on tools-worker-1037 to see who is sending SIGTERM, will uninstall after done. dependency: bin86. T115231 |
[tools] |
| 05:38 |
<elukey> |
re-run webrequest text 2020-01-26T20/21 with higher dataloss thresholds (false positives) |
[analytics] |
| 03:29 |
<gehel> |
restarting blazegraph on wdqs100[57] |
[production] |
| 02:49 |
<elukey> |
re-run refine eventlogging manually to clear out refine failed events |
[analytics] |
|
2020-01-26
§
|
| 23:23 |
<James_F> |
Zuul: Mark mediawiki/extensions/BlueSpiceMenues as archived T240788 |
[releng] |
| 22:20 |
<James_F> |
Zuul: Mark mediawiki/extensions/SemanticTasks as archived T221549 |
[releng] |
| 21:45 |
<akosiaris> |
repool maps1003 |
[production] |
| 21:45 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: name=maps1003.* |
[production] |
| 21:42 |
<akosiaris> |
test depool maps1003 |
[production] |
| 21:42 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: name=maps1003.* |
[production] |
| 21:38 |
<vgutierrez> |
powercycling cp3051 - T238305 |
[production] |
| 21:23 |
<akosiaris> |
restart kartotherian on maps1002 |
[production] |
| 21:19 |
<vgutierrez> |
restart varnish-fe and ats-tls on cp3056 |
[production] |
| 21:02 |
<bblack> |
ats-tls-restart on cp3064 |
[production] |
| 20:51 |
<bblack> |
esams text caches: reverting earlier sysctl mitigations |
[production] |
| 18:51 |
<hauskatze> |
maurelio@deployment-deploy01:~$ foreachwiki maintenance/emptyUserGroup.php ep[campus,online,coordinator,instructor] | T241504 |
[releng] |
| 18:11 |
<volans> |
shutdown elastic2043 - T243715 |
[production] |
| 18:01 |
<volans> |
depooled elastic2043 - T243715 |
[production] |
| 18:01 |
<volans@cumin1001> |
conftool action : set/pooled=inactive; selector: name=elastic2043.codfw.wmnet |
[production] |
| 17:58 |
<elukey> |
re-run failed refine job for MobileWebUIActionsTracking 2020-01-26T12 |
[analytics] |
| 17:32 |
<elukey> |
restart varnishkafka on cp3056/cp3064 due to network issues on the hosts |
[analytics] |
| 17:28 |
<elukey> |
restart varnishkafka-webrequest on cp3064 |
[production] |
| 17:25 |
<elukey> |
restart varnishkafka-webrequest on cp3056 |
[production] |