1701-1750 of 10000 results (44ms)
2020-04-08 §
16:16 <ema> cache_upload: rolling varnish-fe restarts to bump transient storage limit T185968 [production]
15:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:19 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:11 <ema> cp3051: param.set shortlived=0 to try ease pressure on transient memory [production]
14:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1112 after schema change', diff saved to https://phabricator.wikimedia.org/P10947 and previous config saved to /var/cache/conftool/dbconfig/20200408-142341-marostegui.json [production]
14:14 <jeh@deploy1001> Finished deploy [horizon/deploy@0d18f67]: update horizon submodule to enable server groups (duration: 03m 30s) [production]
14:10 <jeh@deploy1001> Started deploy [horizon/deploy@0d18f67]: update horizon submodule to enable server groups [production]
13:40 <mutante> stopped and masked zuul-merger service on contint2001 via puppet (T224591) [production]
13:30 <ema> cp3050: stop vhtcpd, start purged T249583 [production]
13:22 <vgutierrez> enable inbound TLSv1.3 in text@ulsfo - T170567 [production]
13:05 <ema> purged 0.1 uploaded to buster-wikimedia T249583 [production]
12:31 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: re-sync (duration: 01m 07s) [production]
12:29 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:585219|Enable GrowthExperiments suggested edits on uk, hu, hy, eu wikipedias (T247308)]] (duration: 01m 08s) [production]
12:17 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584135|Enable GrowthExperiments welcome survey on Ukrainian, Hungarian, Armenian Wikipedias (T238295) (duration: 01m 08s) [production]
12:09 <tgr@deploy1001> Synchronized wmf-config/: SWAT: [[gerrit:584183|Enable GrowthExperiments on French Wiktionary (T235964)]] (duration: 01m 06s) [production]
11:56 <tgr@deploy1001> Synchronized dblists/: SWAT: [[gerrit:584183|Enable GrowthExperiments on French Wiktionary (T235964)]] (duration: 01m 03s) [production]
11:48 <mutante> logstash1009 - restarted logstash [production]
11:43 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:585766|Enable WikibaseQualityConstraints on test commons (T248117)]] (duration: 01m 05s) [production]
11:43 <marostegui> Deploy schema change on db1112, this will generate lag on labs s3 [production]
11:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 for schema change', diff saved to https://phabricator.wikimedia.org/P10942 and previous config saved to /var/cache/conftool/dbconfig/20200408-114315-marostegui.json [production]
11:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1078 after schema change', diff saved to https://phabricator.wikimedia.org/P10941 and previous config saved to /var/cache/conftool/dbconfig/20200408-113901-marostegui.json [production]
11:29 <tgr@deploy1001> Synchronized wmf-config/: SWAT: [[gerrit:584133|Deploy GrowthExperiments on Serbian Wikipedia (T241181)]] (duration: 01m 06s) [production]
11:28 <tgr@deploy1001> Synchronized dblists/: SWAT: [[gerrit:584133|Deploy GrowthExperiments on Serbian Wikipedia (T241181)]] (duration: 01m 17s) [production]
11:05 <XioNoX> push urpf log only to codfw - T244147 [production]
10:39 <jbond42> restarting idp.wikimedia.org [production]
10:14 <marostegui> Deploy schema change on db1078 [production]
10:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P10940 and previous config saved to /var/cache/conftool/dbconfig/20200408-101431-marostegui.json [production]
09:30 <jynus> stopping and removing db1095:s8 instance [production]
09:20 <godog> upgrade grafana on cloudmetrics hosts - T244208 [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1075 after schema change', diff saved to https://phabricator.wikimedia.org/P10939 and previous config saved to /var/cache/conftool/dbconfig/20200408-091728-marostegui.json [production]
09:11 <gehel> setting weight=10 for all pooled wdqs servers in codfw - T246343 [production]
09:10 <marostegui> Reload proxies on dbproxy1018 and dbproxy1019 to depool labsdb1011 - T249188 T248592 [production]
09:07 <gehel> pooling wdqs200[78] - new servers ready to go! - T246343 [production]
08:46 <marostegui> Rename wb_terms and recreate views on labsdb1009-labsdb1011 - T248592 T248086 [production]
08:39 <godog> upgrade grafana on grafana1002 - T244208 [production]
08:17 <_joe_> switching parsoid to envoy (take 2) in eqiad [production]
07:23 <marostegui> Deploy schema change on db1075 [production]
07:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1075 for schema change', diff saved to https://phabricator.wikimedia.org/P10937 and previous config saved to /var/cache/conftool/dbconfig/20200408-072331-marostegui.json [production]
06:31 <marostegui> Deploy schema change on db1095:3313 [production]
06:11 <marostegui> Stop haproxy on dbproxy1011 - T231520 [production]
05:44 <vgutierrez> rolling upgrade ATS to 8.0.6-1wm6 in cp[5006,5012,3065,3064,2042,2041,1090,1089] [production]
05:34 <marostegui> Deploy schema change on dbstore1004:3313 [production]
05:33 <_joe_> repooling wtp1025, with envoy and logging any error above 404 T249535 [production]
04:36 <vgutierrez> rolling restart of ats-tls - T249335 [production]
2020-04-07 §
20:39 <andrewbogott> correction: briefly downtiming ldap-eqiad-replica0 and ldap-eqiad-replica1. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes [production]
20:37 <andrewbogott> briefly downtiming serpens and seaborgium. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes [production]
20:34 <hoo> (Take 3) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]
20:17 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.27 refs T247774 [production]
20:09 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.27 (duration: 60m 34s) [production]
20:08 <hoo> (Take 2) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]