701-750 of 10000 results (28ms)
2020-11-18 ยง
15:09 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
15:09 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
15:05 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
15:03 <Urbanecm> Purge https://2030.wikimedia.org/ via purgeList.php (T264797) [production]
14:34 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
14:30 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
14:30 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
14:17 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
14:13 <Urbanecm> Purge https://2030.wikimedia.org/ via purgeList.php (T264797) [production]
14:09 <elukey> copied /etc/krb5.keytab from krb1001 to krb2001 (the last one contained only one principal for 2001, the first one both for 1001 and 2001) [production]
14:05 <moritzm> installing openldap security updates on ro replicas [production]
14:03 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
14:02 <elukey> restart krb5-kpropd.service on krb2001 to force the pick up of new client configs [production]
13:35 <bblack> cache_text: Executing "varnishadm -n frontend param.set nuke_limit 1000" - T266373 [production]
13:34 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
13:30 <moritzm> installing openldap security updates on corp replicas [production]
13:08 <Urbanecm> EU B&C done (~15 minutes ago) [production]
12:43 <akosiaris> sync staging cluster's helmfile.d/admin state. Aside from calico, the rest is a noop [production]
12:43 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . [production]
12:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
12:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . [production]
12:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . [production]
12:40 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.16/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 5488f56c7458fa8fb9be5f41f131e00b26a84cc0: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) [production]
12:25 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.18/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/NewcomerTasksCacheRefreshJob.php: 45d71a37f381e81e5382c8e10ac4063c9665beb8: Fix NewcomerTasksCacheRefreshJob (T268008) (duration: 01m 05s) [production]
12:13 <Urbanecm> Purge https://en.wikipedia.org/static/images/project-logos/{bnwiki,bnwiki-1.5x,bnwiki-2x}.png (T265553) [production]
12:13 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=releases [production]
12:11 <urbanecm@deploy1001> Synchronized static/images/project-logos/: 70aabf7ec8e1b549e78978e48967fb70d21316de: Regenerate Bengali Wikipedia logo (T265553) (duration: 01m 06s) [production]
12:06 <akosiaris@cumin1001> conftool action : set/ttl=300; selector: dnsdisc=wikifeeds [production]
12:01 <akosiaris@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=restbase-async,name=eqiad [production]
12:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool pc1009 in pc3 after restarting mysql T266483 (duration: 01m 06s) [production]
12:00 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=blubberoid,name=eqiad [production]
11:56 <Urbanecm> Start of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=frwiki; T246539) [production]
11:56 <marostegui> Restart mysql on pc1009 T266483 [production]
11:56 <Urbanecm> End of mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log in a tmux at mwmaint1002 (wiki=nlwiki; T246539) [production]
11:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool pc1009 and place pc1010 instead of it T266483 (duration: 01m 18s) [production]
11:40 <XioNoX> eqiad row D: remove un-needed "enable" keywords [production]
10:59 <jbond@cumin1001> END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) [production]
10:59 <jbond@cumin1001> START - Cookbook sre.puppet.renew-cert [production]
10:58 <jbond42> renew sretest1002 ssl cert to test cookbook [production]
10:51 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:51 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:48 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:48 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:25 <godog> ms-be1022 - disable failed sdb [production]
10:01 <XioNoX> eqiad row D: Standardize interfaces descriptions [production]
09:56 <moritzm> uploaded libexif 0.6.21-2+deb8u4+wmf1 to jessie-wikimedia [production]
09:22 <elukey> set dns_canonicalize_hostname = false to all kerberos clients [production]
09:13 <jbond42> renew puppet certificate of seaborgium [production]
08:34 <marostegui> Stop MySQL on es1011, es1012, es1014 T268100 T268101 T268102 [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es1012 from dbctl T268101', diff saved to https://phabricator.wikimedia.org/P13326 and previous config saved to /var/cache/conftool/dbconfig/20201118-082942-marostegui.json [production]