|
2018-04-18
§
|
| 09:30 |
<marostegui> |
Deploy schema change on db1096:3315 - T191519 T188299 T190148 |
[production] |
| 09:30 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 01m 22s) |
[production] |
| 09:27 |
<godog> |
reenable puppet fleetwide after https://gerrit.wikimedia.org/r/c/421860 |
[production] |
| 09:16 |
<moritzm> |
imported lz4 0.0~r131-2~wmf1+trusty1 for trusty-wikimedia to apt.wikimedia.org (needed to build HHVM 3.18 for trusty) |
[production] |
| 09:09 |
<godog> |
stop puppet agent fleetwide before applying https://gerrit.wikimedia.org/r/c/421860/ |
[production] |
| 09:08 |
<moritzm> |
reimaging mw1281 to stretch |
[production] |
| 09:04 |
<_joe_> |
restart HHVM on mw1223,mw1224, also repool them after investigation in crashes |
[production] |
| 08:59 |
<vgutierrez> |
Repool (Re-enable BGP) in lvs3003 - T191897 |
[production] |
| 08:44 |
<elukey> |
execute cumin 'analytics10[28-69]*' 'rm /etc/apt/preferences.d/r_* && apt-get update' to clear jessie backports apt config - T192348 |
[production] |
| 07:39 |
<vgutierrez> |
Depool and reimage lvs3003 as stretch - T191897 |
[production] |
| 06:49 |
<marostegui> |
Deploy schema change on s5 codfw master (db2052) this will generate lag in codfw - T191519 T188299 T190148 |
[production] |
| 06:43 |
<moritzm> |
installing ruby security updates for trusty |
[production] |
| 05:27 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1114 after changing RX buffers - T191996 (duration: 01m 09s) |
[production] |
| 05:20 |
<marostegui> |
Change RX buffers on db1114 - T191996 |
[production] |
| 05:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 01m 15s) |
[production] |
| 05:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 16s) |
[production] |
| 05:02 |
<marostegui> |
Deploy schema change on db1071 (s8 primary master) - T185128 T153182 |
[production] |
| 02:51 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.29) (duration: 05m 55s) |
[production] |
| 00:05 |
<aaron@tin> |
Synchronized wmf-config/mc-labs.php: 8ad186728d: use mcrouter key prefixes (deployment-prep only) (duration: 01m 15s) |
[production] |
|
2018-04-17
§
|
| 23:31 |
<ebernhardson@tin> |
Synchronized wmf-config/CommonSettings-labs.php: labs config noop (duration: 01m 15s) |
[production] |
| 23:17 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: T191236: Shift search traffic back to eqiad (duration: 01m 17s) |
[production] |
| 23:08 |
<gilles> |
Private wiki thumbnail traffic now going to eqiad T191643 |
[production] |
| 23:07 |
<gilles@tin> |
Synchronized wmf-config/filebackend.php: Fix private wiki DC configuration: [[gerrit:424618|Serve private wiki thumbnails with Thumbor (T191643)]] (duration: 01m 18s) |
[production] |
| 21:34 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: ext-dist config changes for rel1_31 (duration: 01m 16s) |
[production] |
| 20:13 |
<demon@tin> |
rebuilt and synchronized wikiversions files: group0 to wmf.30 |
[production] |
| 19:59 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@f08fbcc]: GUI update |
[production] |
| 19:48 |
<demon@tin> |
Finished scap: bootstrap wmf.30 (duration: 112m 35s) |
[production] |
| 19:01 |
<imarlier@tin> |
Finished deploy [performance/navtiming@22483a4]: Navtiming refactor for increased testability, and to add wrapper for easy service use (duration: 00m 02s) |
[production] |
| 19:01 |
<imarlier@tin> |
Started deploy [performance/navtiming@22483a4]: Navtiming refactor for increased testability, and to add wrapper for easy service use |
[production] |
| 18:52 |
<urandom> |
rebooting restbase-dev1006 (kernel oom killer misbehaving) |
[production] |
| 18:45 |
<urandom> |
rebooting restbase-dev1005 (kernel oom killer misbehaving) |
[production] |
| 18:41 |
<urandom> |
rebooting restbase-dev1004 (kernel oom killer misbehaving) |
[production] |
| 17:56 |
<demon@tin> |
Started scap: bootstrap wmf.30 |
[production] |
| 17:27 |
<ejegg> |
updated payments-wiki from 320a6c2600 to 4a8aada491 |
[production] |
| 17:16 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 100% of anons for enwiki - T191101 (duration: 00m 59s) |
[production] |
| 16:57 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool es1017 fully (duration: 01m 16s) |
[production] |
| 16:37 |
<elukey> |
incremental rollout of the new zookeeper jmx config to druid1* and conf* |
[production] |
| 16:34 |
<urandom> |
decommissioning Cassandra, restbase1010-a -- T189822 |
[production] |
| 16:02 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable page previews for 75% of anons for enwiki - T191101 (duration: 00m 58s) |
[production] |
| 15:50 |
<arturo> |
enable puppet in labstore1004 |
[production] |
| 15:37 |
<vgutierrez> |
Repool (Enable BGP) on lvs3004 - T191897 |
[production] |
| 15:33 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Change db2048 IP - T191193 (duration: 00m 58s) |
[production] |
| 15:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Change db2048 IP - T191193 (duration: 00m 58s) |
[production] |
| 15:23 |
<marostegui> |
Stopping mysql on db2048 will break replication on codfw s1 slaves |
[production] |
| 15:23 |
<marostegui> |
Stop MySQL on db2048 for rack movement - T191193 |
[production] |
| 15:22 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1067, es1017 with low load (duration: 01m 02s) |
[production] |
| 14:59 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1114 after changing the network cable - T191996 (duration: 01m 02s) |
[production] |
| 14:55 |
<gehel> |
starting data reimport after re-image for wdqs2001 - T189192 |
[production] |
| 14:52 |
<marostegui> |
Stop MySQL on db2042 to move it to another rack - https://phabricator.wikimedia.org/T191193 |
[production] |
| 14:36 |
<ariel@tin> |
Finished deploy [dumps/dumps@1073d75]: more exception logging from xmlstream (duration: 00m 03s) |
[production] |