|
2020-01-28
§
|
| 23:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3314 T239453', diff saved to https://phabricator.wikimedia.org/P10287 and previous config saved to /var/cache/conftool/dbconfig/20200128-235336-marostegui.json |
[production] |
| 23:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3314 T239453', diff saved to https://phabricator.wikimedia.org/P10286 and previous config saved to /var/cache/conftool/dbconfig/20200128-234601-marostegui.json |
[production] |
| 23:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Start repooling db1084 with its original weight', diff saved to https://phabricator.wikimedia.org/P10285 and previous config saved to /var/cache/conftool/dbconfig/20200128-234219-marostegui.json |
[production] |
| 23:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1121 T232446', diff saved to https://phabricator.wikimedia.org/P10284 and previous config saved to /var/cache/conftool/dbconfig/20200128-234037-marostegui.json |
[production] |
| 17:24 |
<arturo> |
[codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. (T243766) |
[admin] |
| 15:06 |
<addshore> |
Start addshore@mwmaint1002:~$ ./T219123.sh # Taking over from @ladsgroup for T219123 |
[production] |
| 13:48 |
<arturo> |
crontab jobs activated again |
[tools.jarbot] |
| 13:35 |
<arturo> |
`aborrero@tools-clushmaster-02:~$ clush -w @exec-stretch 'for i in $(ps aux | grep [t]ools.j | awk -F" " "{print \$2}") ; do echo "killing $i" ; sudo kill $i ; done || true'` (T243831) |
[tools] |
| 11:18 |
<arturo> |
disabled all cronjobs per request from WMF SRE team: https://en.wikipedia.org/w/index.php?title=User_talk%3AJarBot&type=revision&diff=937974097&oldid=719916908 |
[tools.jarbot] |
| 11:15 |
<arturo> |
stopped grid jobs per request from WMF SRE team: https://en.wikipedia.org/w/index.php?title=User_talk%3AJarBot&type=revision&diff=937974097&oldid=719916908 |
[tools.jarbot] |
| 10:18 |
<arturo> |
[codfw1dev] created DNS record `bastion-codfw1dev-01.codfw1dev.wmcloud.org A 185.15.57.2` (T242976, T229441) |
[admin] |
| 10:13 |
<arturo> |
[codfw1dev] the zone `codfw1dev.wmcloud.org` belongs now to the `cloudinfra-codfw1dev` project (T242976) |
[admin] |
| 10:11 |
<arturo> |
[codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description "main DNS domain for public addresses" --email "root@wmflabs.org" --type PRIMARY --ttl 3600 codfw1dev.wmcloud.org.` (T242976 and T243766) |
[admin] |
| 10:03 |
<arturo> |
delegated `codfw1dev.wmcloud.org` to designate @ codfw1dev ns0.openstack.codfw1dev.wikimediacloud.org (T242976 and T243766) |
[cloudinfra] |
| 09:59 |
<effie> |
rolling restart mobileapps in codfw |
[production] |
| 09:53 |
<arturo> |
restart apache2 in labweb1001/1002 because horizon errors |
[admin] |
| 09:53 |
<arturo> |
the DNS zone wmcloud.org now belongs to this project (T242976) |
[cloudinfra] |
| 09:47 |
<arturo> |
created DNS zone wmcloud.org in eqiad1, transfer it to the cloudinfra project (T242976) right now only use is to delegate codfw1dev.wmcloud.org subdomain to designate in the other deployment |
[admin] |
| 02:05 |
<mutante> |
gerrit1002 - gzipping a bunch of /var/log/gerrit/ log files (T243808) |
[production] |
| 00:17 |
<wm-bot> |
<lucaswerkmeister> deployed 61fe7e59fb (typofix) |
[tools.lexeme-forms] |
| 00:08 |
<wm-bot> |
<lucaswerkmeister> deployed e0e916e0a5 (more Persian translations and RTL fixes) |
[tools.lexeme-forms] |
|
2020-01-27
§
|
| 23:40 |
<eileen> |
civicrm revision changed from fbd5c35fb0 to ac730a6bcb, config revision is 837b9d0703 |
[production] |
| 23:23 |
<wm-bot> |
<lucaswerkmeister> deployed 54b9e37118 (more RTL fixes) |
[tools.lexeme-forms] |
| 23:14 |
<wm-bot> |
<lucaswerkmeister> deployed 72ec256823 (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log] |
[tools.lexeme-forms] |
| 23:10 |
<vgutierrez> |
rolling restart of varnish-frontend in cp4026 and cp4027 |
[production] |
| 23:06 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:06 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:01 |
<_joe_> |
restart apache on gerrit |
[production] |
| 22:58 |
<vgutierrez> |
restarting gerrit service |
[production] |
| 22:01 |
<vgutierrez> |
restarting varnish-fe on cp4028 |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3311 - T239453', diff saved to https://phabricator.wikimedia.org/P10277 and previous config saved to /var/cache/conftool/dbconfig/20200127-191614-marostegui.json |
[production] |
| 19:15 |
<marostegui> |
Remove partitions from db2085 enwiki - T239453 |
[production] |
| 17:46 |
<zhuyifei1999_> |
restarted webservice via `webservice --backend kubernetes php7.3 stop` `webservice --backend kubernetes php7.3 start` T115231 |
[tools.dplbot] |
| 13:58 |
<vgutierrez> |
repooling cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
restarting varnish-fe on cp4030 - T243634 |
[production] |
| 13:54 |
<vgutierrez> |
repooling cp4029 - T243634 |
[production] |
| 13:36 |
<vgutierrez> |
restarting varnish-fe on cp4029 - T243634 |
[production] |
| 12:45 |
<arturo> |
[codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]> update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` T243556 |
[admin] |
| 12:44 |
<arturo> |
[codfw1dev] `root@cloudcontrol2001-dev:~# openstack zone create --description "main DNS domain for VMs" --email "root@wmflabs.org" --type PRIMARY --ttl 3600 codfw1dev.wikimedia.cloud.` T243556 |
[admin] |
| 12:10 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --from-id 1860 --to-id 1860 (T243705) |
[production] |
| 07:05 |
<zhuyifei1999_> |
wrong package. uninstalled. the correct one is bpfcc-tools and seems only available in buster+. T115231 |
[tools] |
| 07:01 |
<zhuyifei1999_> |
apt installing bcc on tools-worker-1037 to see who is sending SIGTERM, will uninstall after done. dependency: bin86. T115231 |
[tools] |
| 05:38 |
<elukey> |
re-run webrequest text 2020-01-26T20/21 with higher dataloss thresholds (false positives) |
[analytics] |
| 03:29 |
<gehel> |
restarting blazegraph on wdqs100[57] |
[production] |
| 02:49 |
<elukey> |
re-run refine eventlogging manually to clear out refine failed events |
[analytics] |