|
2020-04-20
§
|
| 14:54 |
<addshore> |
restart blazegraph on wdqs1006 |
[production] |
| 14:53 |
<addshore> |
depool wdqs1006 as it stopped updating |
[production] |
| 14:28 |
<marostegui> |
Upgrade db2096 (x1 codfw master) |
[production] |
| 14:24 |
<marostegui> |
Upgrade db2101 |
[production] |
| 14:18 |
<marostegui> |
Upgrade dbstore1005 |
[production] |
| 14:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1127 after upgrade', diff saved to https://phabricator.wikimedia.org/P11025 and previous config saved to /var/cache/conftool/dbconfig/20200420-141711-marostegui.json |
[production] |
| 14:13 |
<marostegui> |
Upgrade db2131 |
[production] |
| 14:10 |
<marostegui> |
Upgrade db1127 |
[production] |
| 14:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1127 for upgrade', diff saved to https://phabricator.wikimedia.org/P11023 and previous config saved to /var/cache/conftool/dbconfig/20200420-141017-marostegui.json |
[production] |
| 14:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1089 after schema change', diff saved to https://phabricator.wikimedia.org/P11022 and previous config saved to /var/cache/conftool/dbconfig/20200420-140642-marostegui.json |
[production] |
| 13:50 |
<marostegui> |
Deploy schema change on codfw master - T250055 |
[production] |
| 13:30 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Undeploying graphoid on beta (duration: 01m 07s) |
[production] |
| 13:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1081 after schema change, restore db1097:3314 original weights', diff saved to https://phabricator.wikimedia.org/P11021 and previous config saved to /var/cache/conftool/dbconfig/20200420-131823-marostegui.json |
[production] |
| 12:40 |
<XioNoX> |
remove all disabled termsfrom cr2-eqiad |
[production] |
| 12:31 |
<XioNoX> |
remove all disabled BGP neighbors on cr2-esams |
[production] |
| 12:11 |
<mateusbs17> |
Running `REINDEX DATABASE gis` in maps2004.codfw.wmnet (which is depooled at the moment) |
[production] |
| 11:41 |
<mutante> |
puppetmaster - revoking cert for webserver-misc-apps.discovery.wmnet and recreating it with additional static microsite names (T247650) |
[production] |
| 11:27 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:591017|Temporarily enable event oversampling for conflicts (T249616)]] (duration: 01m 00s) |
[production] |
| 11:25 |
<awight@deploy1001> |
Synchronized php-1.35.0-wmf.28/extensions/TwoColConflict: SWAT: [[gerrit:591016|Configurable EditStepAttempt oversampling for conflicts (T249616)]] (duration: 01m 03s) |
[production] |
| 11:05 |
<mutante> |
rsyncing static-bugzilla files from bromine to miscweb1002 (T247650) |
[production] |
| 11:02 |
<mutante> |
bromine/vega: stop rsyncd which was removed from puppet |
[production] |
| 10:49 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:591023| Bumping portals to master (563985)]] (duration: 00m 57s) |
[production] |
| 10:48 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:591023| Bumping portals to master (563985)]] (duration: 01m 03s) |
[production] |
| 10:37 |
<elukey> |
apt-get purge rsync on mwlog* after https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589600/ |
[production] |
| 10:08 |
<XioNoX> |
uRPF, sample + discard in eqiad - T244147 |
[production] |
| 10:06 |
<XioNoX> |
uRPF, sample + discard in eqord - T244147 |
[production] |
| 09:51 |
<XioNoX> |
uRPF, sample + discard in dfw - T244147 |
[production] |
| 09:38 |
<XioNoX> |
uRPF, sample + discard in ulsfo - T244147 |
[production] |
| 09:19 |
<Urbanecm> |
Security deploy for T250594 |
[production] |
| 08:46 |
<vgutierrez> |
restart ats-tls in cp3064 - T249335 |
[production] |
| 08:35 |
<jayme> |
imported helmfile 0.66.0-1+deb10u1 to main for buster-wikimedia |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Temporary pool db1097:3314 into API', diff saved to https://phabricator.wikimedia.org/P11019 and previous config saved to /var/cache/conftool/dbconfig/20200420-082019-marostegui.json |
[production] |
| 08:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081', diff saved to https://phabricator.wikimedia.org/P11018 and previous config saved to /var/cache/conftool/dbconfig/20200420-081911-marostegui.json |
[production] |
| 08:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1089', diff saved to https://phabricator.wikimedia.org/P11017 and previous config saved to /var/cache/conftool/dbconfig/20200420-081623-marostegui.json |
[production] |
| 08:14 |
<marostegui> |
Remove img_deleted column from db1089 (enwiki), db1081 (commonswiki, db1111 (wikidatawiki) - T250055 |
[production] |
| 08:09 |
<jynus> |
restarting s3 instance on db1095 to reduce its buffer pool T250602 |
[production] |
| 07:22 |
<_joe_> |
restarting php-fpm on the eqiad appservers to pick up the new max_execution_time |
[production] |
| 07:20 |
<marostegui> |
Re add tl_namespace index to db1104 and db1092 - T250060 |
[production] |
| 06:44 |
<moritzm> |
installing python2.7 security updates on jessie |
[production] |
| 06:41 |
<elukey> |
execute find -mtime +30 -delete in /var/log/airflow/scheduler on an-airflow1001 to free space |
[production] |
| 06:25 |
<moritzm> |
installing libxdmcp security updates on jessie |
[production] |
| 06:16 |
<moritzm> |
installing bash updates on jessie |
[production] |
| 05:54 |
<vgutierrez> |
rolling restart of ats-tls in cp[3052,3054,3056,3058,3060,4028,4029,4030,4031,4032] - T249335 |
[production] |
| 05:53 |
<marostegui> |
Deploy schema change on s8 eqiad hosts T250060 |
[production] |
| 05:50 |
<marostegui> |
Deploy schema change on s8 codfw - lag will show up T250060 |
[production] |
| 04:55 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@b813c8a]: no private table dumps, check for existence of 7z,bz2 page content files before dumping, various unit tests (duration: 00m 04s) |
[production] |
| 04:55 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@b813c8a]: no private table dumps, check for existence of 7z,bz2 page content files before dumping, various unit tests |
[production] |