|
2018-04-03
§
|
| 07:50 |
<elukey> |
roll restart zookeeper on druid100[456] to enable prometheus monitoring |
[production] |
| 07:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1078 with low weight (duration: 00m 58s) |
[production] |
| 07:12 |
<jynus> |
upgrade and restart of labsdb1010 |
[production] |
| 07:10 |
<marostegui> |
Stop MySQL on db1078 for mariadb and kernel upgrade |
[production] |
| 06:43 |
<elukey> |
execute systemctl reset-failed kafka-mirror-main-eqiad_to_jumbo-eqiad.service on kafka102[23] |
[production] |
| 06:18 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Change db2035 rack comment (duration: 00m 58s) |
[production] |
| 05:37 |
<marostegui> |
Deploy schema change on db1078 - s3 - T187089 T185128 T153182 |
[production] |
| 05:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 59s) |
[production] |
| 05:18 |
<marostegui> |
Enable back gtid on db2035 - T191193 |
[production] |
| 02:44 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.27) (duration: 19m 03s) |
[production] |
| 00:11 |
<Amir1> |
Evening SWAT is done |
[production] |
|
2018-04-02
§
|
| 23:56 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:405550|Add several domains of Ukraine government to wgCopyUploadsDomains (T185399)]] (duration: 00m 59s) |
[production] |
| 23:45 |
<ladsgroup@tin> |
Synchronized tests/cirrusTest.php: [[gerrit:423532|Shift all search traffic to codfw, part II (T191236)]] (duration: 00m 58s) |
[production] |
| 23:44 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:423532|Shift all search traffic to codfw (T191236)]] (duration: 00m 59s) |
[production] |
| 23:29 |
<Amir1> |
Persian Wikipedia logos have been purged using purgeList.php on terbium |
[production] |
| 23:26 |
<ladsgroup@tin> |
Synchronized static/images/project-logos: [[gerrit:423562|Update logo for the Persian Wikipedia (T191174)]] (duration: 00m 59s) |
[production] |
| 22:41 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/CSSMin.php: sync https://gerrit.wikimedia.org/r/423574 (duration: 00m 58s) |
[production] |
| 22:11 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: all wikis to 1.31.0-wmf.27 |
[production] |
| 21:59 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/CSSMin.php: (no justification provided) (duration: 01m 16s) |
[production] |
| 21:28 |
<twentyafterfour@tin> |
Synchronized php: group1 wikis to 1.31.0-wmf.27 (duration: 01m 14s) |
[production] |
| 21:26 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.27 |
[production] |
| 21:22 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/database/: Revert ceb7d61ee7ef3edc6705abd41ec86b3afcd9c491 refs T183966 T190960 (duration: 00m 59s) |
[production] |
| 21:09 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: all wikis to 1.31.0-wmf.26 |
[production] |
| 20:59 |
<twentyafterfour> |
MediaWiki Train: rolling back to 1.31.0-wmf.26 refs T183966, T190960 |
[production] |
| 20:38 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/database/DatabaseMysqlBase.php: sync 36c52351487aa70a651dbc6157e0a5882dfd9e7f refs T190960 (duration: 01m 16s) |
[production] |
| 20:22 |
<mholloway-shell@tin> |
Finished deploy [mobileapps/deploy@940bd48]: Update mobileapps to 58a0a88 (duration: 05m 52s) |
[production] |
| 20:17 |
<mholloway-shell@tin> |
Started deploy [mobileapps/deploy@940bd48]: Update mobileapps to 58a0a88 |
[production] |
| 19:56 |
<herron> |
puppetdb postgres update complete — puppet agents re-enabled |
[production] |
| 19:46 |
<herron> |
temporarily disabling puppet agents for puppetdb postgres security update |
[production] |
| 19:41 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/database/DatabaseMysqlBase.php: sync 779e7fd3de08101ab626136c07045c98ea162c5b refs T190960 (duration: 01m 16s) |
[production] |
| 19:17 |
<twentyafterfour@tin> |
Synchronized php: group1 wikis to 1.31.0-wmf.27 (duration: 01m 15s) |
[production] |
| 19:16 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.27 |
[production] |
| 19:09 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.27/includes/libs/rdbms/: sync I57dd8d3d0ca96d6fb2f9e83f062f29b1d53224dd refs T183966 T190960 (duration: 01m 19s) |
[production] |
| 19:06 |
<twentyafterfour> |
sync rdbms: avoid lag estimates in getLagFromPtHeartbeat ruined by snapshots Bug: T190960 Change-Id: I57dd8d3d0ca96d6fb2f9e83f062f29b1d53224dd |
[production] |
| 19:04 |
<twentyafterfour> |
Getting the train back on track: deploying 1.31.0-wmf.27 to Group0 |
[production] |
| 17:49 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch the remaining high-traffic jobs to EventBus, test wikis only, file 2/2 - T190327 (duration: 01m 15s) |
[production] |
| 17:48 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@9e1b203]: Switch remaining high traffic jobs for test wikis. T190327 (duration: 00m 43s) |
[production] |
| 17:47 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@9e1b203]: Switch remaining high traffic jobs for test wikis. T190327 |
[production] |
| 17:47 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch the remaining high-traffic jobs to EventBus, test wikis only, file 1/2 - T190327 (duration: 01m 16s) |
[production] |
| 17:36 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Shift serach traffic for enwiki to codfw (duration: 01m 17s) |
[production] |
| 17:21 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@49f4eed]: GUI update (duration: 09m 49s) |
[production] |
| 17:11 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@49f4eed]: GUI update |
[production] |
| 16:37 |
<madhuvishy> |
Rolling out new symlinks to /public/dumps for labstore1006 dumps nfs mount T188643 |
[production] |
| 15:59 |
<madhuvishy> |
Absenting /public/dumps mount from labstore1003 across the VPS fleet T188643 |
[production] |
| 15:56 |
<ebernhardson> |
restart elasticsearch on elastic1024, been stuck at 100% cpu for 3+ hours |
[production] |
| 15:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Change db2035 IP - T191193 (duration: 01m 15s) |
[production] |
| 15:40 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Change db2035 IP - T191193 (duration: 01m 15s) |
[production] |
| 15:28 |
<marostegui> |
Stop MySQL and power off db2035 (s2 codfw master - this will stop replication on s2 codfw slaves) for rack change - T191193 |
[production] |
| 15:06 |
<madhuvishy> |
Reenabled puppet and rolled out mounting new dumps NFS shares from labstore1006|7 on VPS instances T188643 |
[production] |
| 14:40 |
<cmjohnson1> |
disabling puppet on decom host db1020 |
[production] |