|
2018-04-02
§
|
| 20:59 |
<twentyafterfour> |
MediaWiki Train: rolling back to 1.31.0-wmf.26 refs T183966, T190960 |
[production] |
| 20:38 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/database/DatabaseMysqlBase.php: sync 36c52351487aa70a651dbc6157e0a5882dfd9e7f refs T190960 (duration: 01m 16s) |
[production] |
| 20:22 |
<mholloway-shell@tin> |
Finished deploy [mobileapps/deploy@940bd48]: Update mobileapps to 58a0a88 (duration: 05m 52s) |
[production] |
| 20:17 |
<mholloway-shell@tin> |
Started deploy [mobileapps/deploy@940bd48]: Update mobileapps to 58a0a88 |
[production] |
| 19:56 |
<herron> |
puppetdb postgres update complete — puppet agents re-enabled |
[production] |
| 19:46 |
<herron> |
temporarily disabling puppet agents for puppetdb postgres security update |
[production] |
| 19:41 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/database/DatabaseMysqlBase.php: sync 779e7fd3de08101ab626136c07045c98ea162c5b refs T190960 (duration: 01m 16s) |
[production] |
| 19:17 |
<twentyafterfour@tin> |
Synchronized php: group1 wikis to 1.31.0-wmf.27 (duration: 01m 15s) |
[production] |
| 19:16 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.27 |
[production] |
| 19:09 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/: sync I57dd8d3d0ca96d6fb2f9e83f062f29b1d53224dd refs T183966 T190960 (duration: 01m 19s) |
[production] |
| 19:06 |
<twentyafterfour> |
sync rdbms: avoid lag estimates in getLagFromPtHeartbeat ruined by snapshots Bug: T190960 Change-Id: I57dd8d3d0ca96d6fb2f9e83f062f29b1d53224dd |
[production] |
| 19:04 |
<twentyafterfour> |
Getting the train back on track: deploying 1.31.0-wmf.27 to Group0 |
[production] |
| 17:49 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch the remaining high-traffic jobs to EventBus, test wikis only, file 2/2 - T190327 (duration: 01m 15s) |
[production] |
| 17:48 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@9e1b203]: Switch remaining high traffic jobs for test wikis. T190327 (duration: 00m 43s) |
[production] |
| 17:47 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@9e1b203]: Switch remaining high traffic jobs for test wikis. T190327 |
[production] |
| 17:47 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch the remaining high-traffic jobs to EventBus, test wikis only, file 1/2 - T190327 (duration: 01m 16s) |
[production] |
| 17:36 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Shift serach traffic for enwiki to codfw (duration: 01m 17s) |
[production] |
| 17:21 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@49f4eed]: GUI update (duration: 09m 49s) |
[production] |
| 17:11 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@49f4eed]: GUI update |
[production] |
| 16:37 |
<madhuvishy> |
Rolling out new symlinks to /public/dumps for labstore1006 dumps nfs mount T188643 |
[production] |
| 15:59 |
<madhuvishy> |
Absenting /public/dumps mount from labstore1003 across the VPS fleet T188643 |
[production] |
| 15:56 |
<ebernhardson> |
restart elasticsearch on elastic1024, been stuck at 100% cpu for 3+ hours |
[production] |
| 15:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Change db2035 IP - T191193 (duration: 01m 15s) |
[production] |
| 15:40 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Change db2035 IP - T191193 (duration: 01m 15s) |
[production] |
| 15:28 |
<marostegui> |
Stop MySQL and power off db2035 (s2 codfw master - this will stop replication on s2 codfw slaves) for rack change - T191193 |
[production] |
| 15:06 |
<madhuvishy> |
Reenabled puppet and rolled out mounting new dumps NFS shares from labstore1006|7 on VPS instances T188643 |
[production] |
| 14:40 |
<cmjohnson1> |
disabling puppet on decom host db1020 |
[production] |
| 14:28 |
<madhuvishy> |
Disabling puppet across VPS instances with dumps mounted (https://phabricator.wikimedia.org/P6921) T188643 |
[production] |
| 14:22 |
<marostegui> |
Drop contest* tables from s3 - T186867 |
[production] |
| 14:12 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=15; selector: dc=eqiad,service=recommendation-api,cluster=scb,name=scb1003.* |
[production] |
| 14:12 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=15; selector: dc=eqiad,service=recommendation-api,cluster=scb,name=scb1004.* |
[production] |
| 14:10 |
<akosiaris> |
lower weight for scb1001, scb1002 from 10 to 8 for all services. T191199. scb1003, scb1004 have a weight of 15 already |
[production] |
| 14:09 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=8; selector: dc=eqiad,cluster=scb,name=scb1002.* |
[production] |
| 14:09 |
<akosiaris@puppetmaster1001> |
conftool action : set/weight=8; selector: dc=eqiad,cluster=scb,name=scb1001.* |
[production] |
| 13:54 |
<ariel@tin> |
Finished deploy [dumps/dumps@0363d50]: add check that xml files don't have binary corruption (nulls) after the header (duration: 00m 04s) |
[production] |
| 13:54 |
<ariel@tin> |
Started deploy [dumps/dumps@0363d50]: add check that xml files don't have binary corruption (nulls) after the header |
[production] |
| 13:48 |
<twentyafterfour@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Sync initializesettings for T190445 (duration: 01m 16s) |
[production] |
| 13:36 |
<twentyafterfour@tin> |
Synchronized wmf-config/throttle.php: SWAT: Sync throttle rules for T191187 (duration: 01m 15s) |
[production] |
| 13:30 |
<twentyafterfour@tin> |
Synchronized wmf-config/throttle.php: SWAT: Sync throttle rules for T191168 (duration: 01m 16s) |
[production] |
| 13:27 |
<jynus> |
restarting pdfrender on scd1003 (Socket timeout) |
[production] |
| 12:49 |
<akosiaris> |
upgrade mediawiki servers for hhvm upgrade |
[production] |
| 12:06 |
<marostegui> |
Deploy schema change on dbstore1002 - s3 - T187089 T185128 T153182 |
[production] |
| 11:51 |
<akosiaris> |
repool mediawiki canary servers after hhvm upgrade |
[production] |
| 11:44 |
<akosiaris> |
depool mediawiki canary servers for hhvm upgrade |
[production] |
| 10:16 |
<jdrewniak@tin> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:423456|Bumping portals to master (T128546)]] (duration: 01m 16s) |
[production] |
| 10:15 |
<jdrewniak@tin> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:423456|Bumping portals to master (T128546)]] (duration: 01m 16s) |
[production] |
| 09:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Remove references to virt1000 (duration: 01m 16s) |
[production] |
| 09:12 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Remove references to virt1000 (duration: 01m 16s) |
[production] |
| 08:50 |
<marostegui> |
Deploy schema change on s3 codfw master db2043 (this will generate lag on codfw) - T187089 T185128 T153182 |
[production] |
| 08:21 |
<jynus> |
stop mariadb at labsdb1009 and labsdb1010 |
[production] |