|
2018-05-16
§
|
| 06:56 |
<moritzm> |
upgrading mwdebug servers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 |
[production] |
| 06:45 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s3 codfw - T194663 |
[production] |
| 06:25 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s7 - T194663 |
[production] |
| 06:19 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s4 - T194663 |
[production] |
| 06:19 |
<elukey> |
update analytics-in4 on cr1/cr2 eqiad to allow conf100[4-6] (new zookeeper hosts) |
[production] |
| 06:17 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s8 - T194663 |
[production] |
| 06:15 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s6 - T194663 |
[production] |
| 06:11 |
<marostegui> |
Drop unused tables msg_resource msg_resource_links from s5 - T194663 |
[production] |
| 05:54 |
<elukey> |
removed acpi_power_meter manually from conf1004 (blacklisted module in puppet), Acpi errors in dmesg |
[production] |
| 05:22 |
<marostegui> |
Deploy schema change on db1086 - T191519 T188299 T190148 |
[production] |
| 05:21 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 01m 23s) |
[production] |
| 05:14 |
<marostegui> |
Deploy schema change on s2 primary master db1054 - T191519 T188299 T190148 |
[production] |
| 04:43 |
<kartik@tin> |
Finished deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 (T191285) (duration: 03m 52s) |
[production] |
| 04:39 |
<kartik@tin> |
Started deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 (T191285) |
[production] |
| 03:46 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed May 16 03:46:12 UTC 2018 (duration 7m 41s) |
[production] |
| 03:38 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.4) (duration: 16m 26s) |
[production] |
| 02:40 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.3) (duration: 08m 25s) |
[production] |
|
2018-05-15
§
|
| 23:33 |
<mutante> |
creating ganeti VM webperf2002.eqiad.wmnet on ganeti2004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390) |
[production] |
| 23:11 |
<dzahn@neodymium> |
conftool action : set/pooled=yes; selector: name=mw2139.codfw.wmnet |
[production] |
| 23:10 |
<mutante> |
mw2139 - reimaged, scap pull, apache-fast-test baseurls from naos, repooled with confctl (T194426) |
[production] |
| 23:06 |
<mutante> |
creating ganeti VM webperf1002.eqiad.wmnet on ganeti1004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390) |
[production] |
| 22:39 |
<samwilson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Deploying GlobalPreferences T190425 (duration: 01m 21s) |
[production] |
| 22:21 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.4 refs T191050 |
[production] |
| 22:18 |
<twentyafterfour@tin> |
Synchronized php-1.32.0-wmf.4/includes/api/ApiLogin.php: fix syntax error (duration: 01m 39s) |
[production] |
| 21:34 |
<twentyafterfour@tin> |
Finished scap: testwikis to 1.32.0-wmf.4 refs T191050 (duration: 66m 19s) |
[production] |
| 20:27 |
<twentyafterfour@tin> |
Started scap: testwikis to 1.32.0-wmf.4 refs T191050 |
[production] |
| 20:22 |
<mutante> |
[radium:~] $ sudo apt-get autoremove |
[production] |
| 19:53 |
<cwd> |
re-enabled process-control |
[production] |
| 19:42 |
<twentyafterfour@tin> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_770814178" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 47s) |
[production] |
| 19:40 |
<cwd> |
disabled process-control for sql trigger refresh |
[production] |
| 19:39 |
<twentyafterfour@tin> |
Started scap: testwikis wikis to 1.32.0-wmf.4 |
[production] |
| 19:05 |
<mutante> |
mw2139 - wmf-auto-reimage --conftoool --new (because it got "Failed to icinga_downtime" and has a new mainboard (T194426) |
[production] |
| 19:03 |
<mutante> |
mw2139 - wmf-auto-reimage --conftoool --no-verify (T194426) |
[production] |
| 17:27 |
<twentyafterfour> |
branching 1.32.0-wmf.4 refs T191050 |
[production] |
| 17:15 |
<ottomata> |
rolling restart kafka-jumbo100[456] |
[production] |
| 16:36 |
<elukey> |
rolling restart of aqs on aqs* nodes to pick up the new druid config |
[production] |
| 16:27 |
<milimetric@tin> |
Finished deploy [analytics/refinery@679cf09]: Update partition drop script after schema change (duration: 07m 13s) |
[production] |
| 16:25 |
<mobrovac@tin> |
Started restart [cpjobqueue/deploy@58935d5]: (no justification provided) |
[production] |
| 16:21 |
<mobrovac@tin> |
Started restart [changeprop/deploy@e468d8e]: (no justification provided) |
[production] |
| 16:20 |
<milimetric@tin> |
Started deploy [analytics/refinery@679cf09]: Update partition drop script after schema change |
[production] |
| 16:12 |
<elukey> |
roll restart kafka on kafka-jumbo to pick up new zookeeper settings |
[production] |
| 16:11 |
<joal@tin> |
Finished deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch (duration: 05m 47s) |
[production] |
| 16:05 |
<joal@tin> |
Started deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch |
[production] |
| 15:52 |
<elukey> |
rolling restart of hadoop master daemons to pick up new zookeeper settings |
[production] |
| 15:20 |
<elukey> |
roll restart of Kafka Analytics to pick up new zookeeper settings |
[production] |
| 14:59 |
<elukey> |
roll restart of kafka daemons on kafka100[1-3] to pick up new zookeeper settings and group.initial.rebalance.delay.ms = 10s |
[production] |
| 14:28 |
<mobrovac@tin> |
Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change |
[production] |
| 14:28 |
<mobrovac@tin> |
Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change |
[production] |
| 14:19 |
<ottomata> |
temporarily disabling puppet on analytics1003 to run refine-eventbus after jumbo based camus eventbus import finishes |
[production] |
| 14:14 |
<elukey> |
swap conf1001 with conf1004 in the zookeeper main eqiad's config + roll restart of the service |
[production] |