|
2020-11-18
ยง
|
| 13:08 |
<Urbanecm> |
EU B&C done (~15 minutes ago) |
[production] |
| 12:43 |
<akosiaris> |
sync staging cluster's helmfile.d/admin state. Aside from calico, the rest is a noop |
[production] |
| 12:43 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 12:42 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 12:40 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.16/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 5488f56c7458fa8fb9be5f41f131e00b26a84cc0: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) |
[production] |
| 12:25 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.18/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 45d71a37f381e81e5382c8e10ac4063c9665beb8: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) |
[production] |
| 12:13 |
<Urbanecm> |
Purge https://en.wikipedia.org/static/images/project-logos/{bnwiki,bnwiki-1.5x,bnwiki-2x}.png (T265553) |
[production] |
| 12:13 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=releases |
[production] |
| 12:11 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: 70aabf7ec8e1b549e78978e48967fb70d21316de: Regenerate Bengali Wikipedia logo (T265553) (duration: 01m 06s) |
[production] |
| 12:06 |
<akosiaris@cumin1001> |
conftool action : set/ttl=300; selector: dnsdisc=wikifeeds |
[production] |
| 12:01 |
<akosiaris@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=restbase-async,name=eqiad |
[production] |
| 12:00 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1009 in pc3 after restarting mysql T266483 (duration: 01m 06s) |
[production] |
| 12:00 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=blubberoid,name=eqiad |
[production] |
| 11:56 |
<Urbanecm> |
Start of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=frwiki; T246539) |
[production] |
| 11:56 |
<marostegui> |
Restart mysql on pc1009 T266483 |
[production] |
| 11:56 |
<Urbanecm> |
End of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=nlwiki; T246539) |
[production] |
| 11:56 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1009 and place pc1010 instead of it T266483 (duration: 01m 18s) |
[production] |
| 11:40 |
<XioNoX> |
eqiad row D: remove un-needed "enable" keywords |
[production] |
| 10:59 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) |
[production] |
| 10:59 |
<jbond@cumin1001> |
START - Cookbook sre.puppet.renew-cert |
[production] |
| 10:58 |
<jbond42> |
renew sretest1002 ssl cert to test cookbook |
[production] |
| 10:51 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 10:51 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 10:48 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 10:48 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 10:25 |
<godog> |
ms-be1022 - disable failed sdb |
[production] |
| 10:01 |
<XioNoX> |
eqiad row D: Standardize interfaces descriptions |
[production] |
| 09:56 |
<moritzm> |
uploaded libexif 0.6.21-2+deb8u4+wmf1 to jessie-wikimedia |
[production] |
| 09:22 |
<elukey> |
set dns_canonicalize_hostname = false to all kerberos clients |
[production] |
| 09:13 |
<jbond42> |
renew puppet certificate of seaborgium |
[production] |
| 08:34 |
<marostegui> |
Stop MySQL on es1011, es1012, es1014 T268100 T268101 T268102 |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove es1012 from dbctl T268101', diff saved to https://phabricator.wikimedia.org/P13326 and previous config saved to /var/cache/conftool/dbconfig/20201118-082942-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1012 before decommissioning it', diff saved to https://phabricator.wikimedia.org/P13325 and previous config saved to /var/cache/conftool/dbconfig/20201118-082636-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13324 and previous config saved to /var/cache/conftool/dbconfig/20201118-082618-root.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 80%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13323 and previous config saved to /var/cache/conftool/dbconfig/20201118-081115-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13322 and previous config saved to /var/cache/conftool/dbconfig/20201118-075612-root.json |
[production] |
| 07:45 |
<marostegui> |
Deploy schema change on db1098:3316 T267335 T267399 |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 60%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13321 and previous config saved to /var/cache/conftool/dbconfig/20201118-074108-root.json |
[production] |
| 07:28 |
<Urbanecm> |
Start of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=nlwiki; T246539) |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 50%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13320 and previous config saved to /var/cache/conftool/dbconfig/20201118-072605-root.json |
[production] |
| 07:16 |
<marostegui> |
Run check table on s6 on db1125:3316 T267090 |
[production] |
| 07:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 30%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13319 and previous config saved to /var/cache/conftool/dbconfig/20201118-071101-root.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 25%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13318 and previous config saved to /var/cache/conftool/dbconfig/20201118-065558-root.json |
[production] |
| 06:53 |
<elukey> |
restart also mirror maker on kafka-main1001/1003 (seems not related but just to clear old errors and a possible weird state) |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1018 (re)pooling @ 100%: Slowly pool es1018 after cloning es1032 T261717', diff saved to https://phabricator.wikimedia.org/P13317 and previous config saved to /var/cache/conftool/dbconfig/20201118-064556-root.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1032 (re)pooling @ 20%: Slowly pool es1032 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13316 and previous config saved to /var/cache/conftool/dbconfig/20201118-064054-root.json |
[production] |
| 06:37 |
<elukey> |
restart kafka-mirror-main-codfw_to_main-eqiad@0.service on kafka-main1002 - consumer msg rate low since kafka-main2003 went down for codfw c7 failure |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1018 (re)pooling @ 75%: Slowly pool es1018 after cloning es1032 T261717', diff saved to https://phabricator.wikimedia.org/P13315 and previous config saved to /var/cache/conftool/dbconfig/20201118-063052-root.json |
[production] |