4301-4350 of 10000 results (35ms)
2020-04-20 §
11:25 <awight@deploy1001> Synchronized php-1.35.0-wmf.28/extensions/TwoColConflict: SWAT: [[gerrit:591016|Configurable EditStepAttempt oversampling for conflicts (T249616)]] (duration: 01m 03s) [production]
11:05 <mutante> rsyncing static-bugzilla files from bromine to miscweb1002 (T247650) [production]
11:02 <mutante> bromine/vega: stop rsyncd which was removed from puppet [production]
10:49 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:591023| Bumping portals to master (563985)]] (duration: 00m 57s) [production]
10:48 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:591023| Bumping portals to master (563985)]] (duration: 01m 03s) [production]
10:37 <elukey> apt-get purge rsync on mwlog* after https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589600/ [production]
10:08 <XioNoX> uRPF, sample + discard in eqiad - T244147 [production]
10:06 <XioNoX> uRPF, sample + discard in eqord - T244147 [production]
09:51 <XioNoX> uRPF, sample + discard in dfw - T244147 [production]
09:38 <XioNoX> uRPF, sample + discard in ulsfo - T244147 [production]
09:19 <Urbanecm> Security deploy for T250594 [production]
08:46 <vgutierrez> restart ats-tls in cp3064 - T249335 [production]
08:35 <jayme> imported helmfile 0.66.0-1+deb10u1 to main for buster-wikimedia [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Temporary pool db1097:3314 into API', diff saved to https://phabricator.wikimedia.org/P11019 and previous config saved to /var/cache/conftool/dbconfig/20200420-082019-marostegui.json [production]
08:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1081', diff saved to https://phabricator.wikimedia.org/P11018 and previous config saved to /var/cache/conftool/dbconfig/20200420-081911-marostegui.json [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1089', diff saved to https://phabricator.wikimedia.org/P11017 and previous config saved to /var/cache/conftool/dbconfig/20200420-081623-marostegui.json [production]
08:14 <marostegui> Remove img_deleted column from db1089 (enwiki), db1081 (commonswiki, db1111 (wikidatawiki) - T250055 [production]
08:09 <jynus> restarting s3 instance on db1095 to reduce its buffer pool T250602 [production]
07:22 <_joe_> restarting php-fpm on the eqiad appservers to pick up the new max_execution_time [production]
07:20 <marostegui> Re add tl_namespace index to db1104 and db1092 - T250060 [production]
06:44 <moritzm> installing python2.7 security updates on jessie [production]
06:41 <elukey> execute find -mtime +30 -delete in /var/log/airflow/scheduler on an-airflow1001 to free space [production]
06:25 <moritzm> installing libxdmcp security updates on jessie [production]
06:16 <moritzm> installing bash updates on jessie [production]
05:54 <vgutierrez> rolling restart of ats-tls in cp[3052,3054,3056,3058,3060,4028,4029,4030,4031,4032] - T249335 [production]
05:53 <marostegui> Deploy schema change on s8 eqiad hosts T250060 [production]
05:50 <marostegui> Deploy schema change on s8 codfw - lag will show up T250060 [production]
04:55 <ariel@deploy1001> Finished deploy [dumps/dumps@b813c8a]: no private table dumps, check for existence of 7z,bz2 page content files before dumping, various unit tests (duration: 00m 04s) [production]
04:55 <ariel@deploy1001> Started deploy [dumps/dumps@b813c8a]: no private table dumps, check for existence of 7z,bz2 page content files before dumping, various unit tests [production]
2020-04-19 §
16:19 <reedy@deploy1001> Synchronized wmf-config/LabsServices.php: labs: Move RB traffic to new stretch host (duration: 01m 11s) [production]
16:05 <vgutierrez> rolling restart of ats-tls in text@esams - T249335 [production]
05:51 <marostegui> Power back on db1140 T250602 [production]
2020-04-18 §
22:50 <addshore> pool wdqs1006 blazegraph caught up T242453 [production]
20:30 <cdanis@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=eqiad [production]
20:27 <thcipriani> restart gerrit-replica [production]
16:40 <dcausse> forcing replica count to 1 on some cloudelastic@chi indices [production]
15:13 <Amir1> applying schema change of T139090 on labswiki (wikitech) [production]
14:03 <cdanis@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=eqiad [production]
12:19 <addshore> restarting blazegraph on wdqs1006 blazegraph stuck T242453 [production]
12:15 <addshore> depool wdqs1006 blazegraph stuck T242453 [production]
12:15 <addshore> depool wdqs1006 blazegraph stuck [production]
06:07 <XioNoX> change OSPF metrics to prefer ulsfo tunnel transport [production]
2020-04-17 §
19:33 <Krinkle> Depool mw1407.eqiad.wmnet for opcache testing. Do not repool without first reverting https://gerrit.wikimedia.org/r/589674. [production]
19:32 <Krinkle> Depool mw1407.eqiad.wmnet for opcache and LCStoreStaticArray testing. – T99740 [production]
17:41 <cmjohnson1> replacing network cable pc1009 T250257 [production]
17:34 <cmjohnson1> moving msw1 to msw-c racks mounted switch cable ports from port 49 to port 50 [production]
17:22 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:22 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
16:15 <Urbanecm> Revert recent email change of User:CPHL@SUL's email [production]
16:05 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]