|
2020-11-18
ยง
|
| 14:30 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:30 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:17 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:13 |
<Urbanecm> |
Purge https://2030.wikimedia.org/ via purgeList.php (T264797) |
[production] |
| 14:09 |
<elukey> |
copied /etc/krb5.keytab from krb1001 to krb2001 (the last one contained only one principal for 2001, the first one both for 1001 and 2001) |
[production] |
| 14:05 |
<moritzm> |
installing openldap security updates on ro replicas |
[production] |
| 14:03 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 14:02 |
<elukey> |
restart krb5-kpropd.service on krb2001 to force the pick up of new client configs |
[production] |
| 13:35 |
<bblack> |
cache_text: Executing "varnishadm -n frontend param.set nuke_limit 1000" - T266373 |
[production] |
| 13:34 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 13:30 |
<moritzm> |
installing openldap security updates on corp replicas |
[production] |
| 13:08 |
<Urbanecm> |
EU B&C done (~15 minutes ago) |
[production] |
| 12:43 |
<akosiaris> |
sync staging cluster's helmfile.d/admin state. Aside from calico, the rest is a noop |
[production] |
| 12:43 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 12:40 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.16/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 5488f56c7458fa8fb9be5f41f131e00b26a84cc0: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) |
[production] |
| 12:25 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.18/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 45d71a37f381e81e5382c8e10ac4063c9665beb8: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) |
[production] |
| 12:13 |
<Urbanecm> |
Purge https://en.wikipedia.org/static/images/project-logos/{bnwiki,bnwiki-1.5x,bnwiki-2x}.png (T265553) |
[production] |
| 12:13 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=releases |
[production] |
| 12:11 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: 70aabf7ec8e1b549e78978e48967fb70d21316de: Regenerate Bengali Wikipedia logo (T265553) (duration: 01m 06s) |
[production] |
| 12:06 |
<akosiaris@cumin1001> |
conftool action : set/ttl=300; selector: dnsdisc=wikifeeds |
[production] |
| 12:01 |
<akosiaris@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=restbase-async,name=eqiad |
[production] |
| 12:00 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1009 in pc3 after restarting mysql T266483 (duration: 01m 06s) |
[production] |
| 12:00 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=blubberoid,name=eqiad |
[production] |
| 11:56 |
<Urbanecm> |
Start of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=frwiki; T246539) |
[production] |
| 11:56 |
<marostegui> |
Restart mysql on pc1009 T266483 |
[production] |
| 11:56 |
<Urbanecm> |
End of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=nlwiki; T246539) |
[production] |
| 11:56 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1009 and place pc1010 instead of it T266483 (duration: 01m 18s) |
[production] |
| 11:40 |
<XioNoX> |
eqiad row D: remove un-needed "enable" keywords |
[production] |
| 10:59 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) |
[production] |
| 10:59 |
<jbond@cumin1001> |
START - Cookbook sre.puppet.renew-cert |
[production] |
| 10:58 |
<jbond42> |
renew sretest1002 ssl cert to test cookbook |
[production] |
| 10:51 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 10:51 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 10:48 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 10:48 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 10:25 |
<godog> |
ms-be1022 - disable failed sdb |
[production] |
| 10:01 |
<XioNoX> |
eqiad row D: Standardize interfaces descriptions |
[production] |
| 09:56 |
<moritzm> |
uploaded libexif 0.6.21-2+deb8u4+wmf1 to jessie-wikimedia |
[production] |
| 09:22 |
<elukey> |
set dns_canonicalize_hostname = false to all kerberos clients |
[production] |
| 09:13 |
<jbond42> |
renew puppet certificate of seaborgium |
[production] |
| 08:34 |
<marostegui> |
Stop MySQL on es1011, es1012, es1014 T268100 T268101 T268102 |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove es1012 from dbctl T268101', diff saved to https://phabricator.wikimedia.org/P13326 and previous config saved to /var/cache/conftool/dbconfig/20201118-082942-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1012 before decommissioning it', diff saved to https://phabricator.wikimedia.org/P13325 and previous config saved to /var/cache/conftool/dbconfig/20201118-082636-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13324 and previous config saved to /var/cache/conftool/dbconfig/20201118-082618-root.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 80%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13323 and previous config saved to /var/cache/conftool/dbconfig/20201118-081115-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13322 and previous config saved to /var/cache/conftool/dbconfig/20201118-075612-root.json |
[production] |
| 07:45 |
<marostegui> |
Deploy schema change on db1098:3316 T267335 T267399 |
[production] |