151-200 of 10000 results (54ms)
2020-01-16 §
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P10168 and previous config saved to /var/cache/conftool/dbconfig/20200116-072219-marostegui.json [production]
06:58 <marostegui> stop db1107 and db1080 replication in sync [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1080', diff saved to https://phabricator.wikimedia.org/P10166 and previous config saved to /var/cache/conftool/dbconfig/20200116-065505-marostegui.json [production]
02:46 <Krinkle> krinkle@mwmaint1002 Change code_repo.repo_viewvc from 'http://svn.wikimedia.org/viewvc/pywikipedia' to '' for repo_id 2 (pywikipedia) for. Ref 2162cf2fc46cfe. [production]
02:35 <Krinkle> krinkle@mwmaint1002 Change code_repo.repo_viewvc from 'https://svn.wikimedia.org/viewvc/mediawiki' to '' for 'MediaWiki' repo_name. Ref 2162cf2fc46cfe, T205361. [production]
00:40 <bstorm_> set max_connections on db1133 (m5-master) back to 500 since the neutron connections seem fairly stable now T242817 [production]
00:23 <catrope@deploy1001> Synchronized static/images/project-logos/: Restore pre-censorship trwiki logos (T242932) (duration: 01m 05s) [production]
00:17 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Enable topics for suggested edits on testwiki (duration: 01m 04s) [production]
2020-01-15 §
22:40 <mutante> phabricator - disabling 'bzimport' user (T242860) [production]
21:03 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.14/languages/messages/MessagesMrj.php: Fix fallbacks of mrj (Hill Mari) T242409 T242796 (duration: 01m 05s) [production]
20:47 <mutante> gerrit - adding Zoranzoki to members of extension-GoogleAdSense (endorsed by extension owner Siebrand) (T241509) [production]
20:28 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Touched IS.php for sync (duration: 01m 05s) [production]
20:27 <jforrester@deploy1001> sync-file aborted: Enable partial blocks on last wiki, (duration: 00m 01s) [production]
20:17 <krinkle@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/MultimediaViewer/resources/: T229484 (duration: 01m 06s) [production]
19:57 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable partial blocks on last wiki, Commons T242570 (duration: 01m 03s) [production]
19:54 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable banner for wikis that recently opted in to partial blocks T240300 T242570 T242569 (duration: 01m 05s) [production]
18:10 <anomie@deploy1001> Synchronized wmf-config/CommonSettings.php: Set OAuth 2 access token expiry to "infinity" (duration: 01m 04s) [production]
17:50 <anomie@deploy1001> Synchronized private/PrivateSettings.php: Setting RSA keys for OAuth 2.0 (T242872) (duration: 01m 05s) [production]
16:27 <elukey> import key 0xDBBF9D42B7B4BD70 (Apache BigTop) manually on install1002's gpg [production]
15:54 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/WikibaseQualityConstraints/extension.json: [[gerrit:565012|Fix service injection for special page (T242846)]] (duration: 01m 08s) [production]
15:40 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/Wikibase/client/includes/Api/PageTerms.php: [[gerrit:565034|Fix invalid iteration over false in PageTerms (T242856)]] (duration: 01m 06s) [production]
15:37 <vgutierrez> rolling restart of ats-tls instances - T196558 T242778 [production]
15:28 <ema> cp3064: ats-tls-restart to apply https://gerrit.wikimedia.org/r/559711 T196558 [production]
15:20 <moritzm> installing OpenSSL security updates on db* hosts [production]
15:02 <moritzm> installing OpenSSL security updates on mw* [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1252.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1251.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1250.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1249.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1248.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1247.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1246.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1245.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1244.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1243.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1242.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1241.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1240.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1239.eqiad.wmnet [production]
14:54 <jiji@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1238.eqiad.wmnet [production]
14:54 <effie> lower weights on slower servers mw1238-mw1252 [production]
14:53 <effie> pool mw1238, mw1240, mw1246 [production]
14:43 <XioNoX> reject RPKI invalids in dfw - T220669 [production]
14:30 <moritzm> rolling restart of FPM on mw1261-mw1265 to pick up OpenSSL security update [production]
14:25 <XioNoX> reject RPKI invalids in ams - T220669 [production]
14:18 <godog> reenable puppet on cp hosts, after https://gerrit.wikimedia.org/r/c/operations/puppet/+/563430 deployment [production]
14:08 <effie> depool mw1238, mw1240, mw1246 [production]
14:06 <liw@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.15 (duration: 01m 07s) [production]
14:05 <liw@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.15 [production]
13:58 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]