|
2016-02-14
§
|
| 23:53 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270656 |
[releng] |
| 23:25 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270654 |
[releng] |
| 23:18 |
<jynus> |
repaired table and restarted replication on sanitarium (s3) |
[production] |
| 23:13 |
<legoktm> |
also deploying https://gerrit.wikimedia.org/r/#/c/265098/ |
[releng] |
| 23:11 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270651 |
[releng] |
| 23:06 |
<andrewbogott> |
moved tools dbs off of labsdb1002 via https://gerrit.wikimedia.org/r/#/c/270650/ |
[production] |
| 22:40 |
<_joe_> |
labsdb lvm partition /srv unavailable because /dev/sdc is apparently broken |
[production] |
| 16:55 |
<_joe_> |
powercycling mw1140, stuck in OOM, no ssh no console login |
[production] |
| 14:16 |
<bd808> |
Updated to 186f4aa; commands and phab task mentions now ignored for a large number of known bots |
[tools.stashbot] |
| 13:57 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: vslow, dump to db1054; do not send api to db1054; 1036 only rc (duration: 01m 17s) |
[production] |
| 13:56 |
<travis-ci> |
bd808/tools-stashbot#5 (master - 186f4aa Bryan Davis): The build passed. - https://travis-ci.org/bd808/tools-stashbot/builds/109161172 |
[tools.stashbot] |
| 13:14 |
<hoo@mira> |
Synchronized wmf-config/db-eqiad.php: Depool db1021 (duration: 01m 26s) |
[production] |
| 05:31 |
<travis-ci> |
bd808/tools-stashbot#4 (master - b349296 Bryan Davis): The build passed. - https://travis-ci.org/bd808/tools-stashbot/builds/109113976 |
[tools.stashbot] |
| 05:21 |
<travis-ci> |
bd808/tools-stashbot#3 (master - cb95bea Bryan Davis): The build passed. - https://travis-ci.org/bd808/tools-stashbot/builds/109113152 |
[tools.stashbot] |
| 05:18 |
<bd808> |
tools.stashbot Testing after restart (T126419) |
[releng] |
| 05:12 |
<travis-ci> |
bd808/tools-stashbot#2 (master - 7f51fbf Bryan Davis): The build passed. - https://travis-ci.org/bd808/tools-stashbot/builds/109112611 |
[tools.stashbot] |
| 02:40 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Feb 14 02:40:27 UTC 2016 (duration 9m 8s) |
[production] |
| 02:31 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 12m 01s) |
[production] |
| 01:06 |
<travis-ci> |
bd808/tools-stashbot#1 (master - 22f54c4 Bryan Davis): The build passed. - https://travis-ci.org/bd808/tools-stashbot/builds/109092935 |
[tools.stashbot] |
| 01:02 |
<bd808> |
Post `restart` test {T126419} |
[tools.stashbot] |
|
2016-02-13
§
|
| 22:30 |
<Krenair> |
Fixed puppet on phab-03 and harbormaster1 by signing the certs on deploy |
[phabricator] |
| 21:28 |
<bd808> |
Post `restart` test {T126419} |
[tools.stashbot] |
| 21:12 |
<bd808> |
Post restart test {{T126419}} |
[tools.stashbot] |
| 06:42 |
<bd808> |
restarted nutcracker on deployment-mediawiki01 |
[releng] |
| 06:32 |
<bd808> |
jobrunner on deployment-jobrunner01 enabled after reverting changes from T87928 that caused T126830 |
[releng] |
| 06:30 |
<urandom> |
`nodetool stop -- COMPACTION && nodetool cleanup' on restbase1002.eqiad, an abundance of caution (https://phabricator.wikimedia.org/P2612) |
[production] |
| 05:51 |
<bd808> |
disabled jobrunner process on jobrunner01; queue full of jobs broken by T126830 |
[releng] |
| 05:31 |
<bd808> |
trebuchet clone of /srv/jobrunner/jobrunner broken on jobrunner01; failing puppet runs |
[releng] |
| 05:25 |
<bd808> |
jobrunner process on deployment-jobrunner01 badly broken; investigating |
[releng] |
| 05:20 |
<bd808> |
Ran https://phabricator.wikimedia.org/P2273 on deployment-jobrunner01.deployment-prep.eqiad.wmflabs; freed ~500M; disk utilization still at 94% |
[releng] |
| 05:00 |
<bd808> |
Added Alex Monk (Krenair) as projectadmin |
[phabricator] |
| 04:37 |
<urandom> |
(ephemerally) dropping compactor thread count from 10 to 8 on restbase1002.eqiad |
[production] |
| 01:18 |
<mutante> |
omg testing this log feature that logs straight to tickets (T108720) |
[production] |
| 00:42 |
<krinkle@mira> |
Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 14s) |
[production] |
| 00:28 |
<krinkle@mira> |
Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 17s) |
[production] |
| 00:10 |
<mutante> |
ruthenium - restarting parsoid, now works out of /srv/ |
[production] |
|
2016-02-12
§
|
| 23:54 |
<hashar> |
beta cluster broken since 20:30 UTC https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor havent looked |
[releng] |
| 23:37 |
<mutante> |
ruthenium - moving parsoid path, cleaning up old resources |
[production] |
| 22:33 |
<ori@mira> |
Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/extension.json: I315628aef3: Don't use 'qlow' for NetSpeed=B (duration: 01m 16s) |
[production] |
| 21:30 |
<tgr@mira> |
Synchronized wmf-config/InitialiseSettings.php: T125455: log session-ip channel to logstash (duration: 01m 17s) |
[production] |
| 21:22 |
<ori@mira> |
Synchronized docroot and w: Ifc5b02cba4: Speed trials: add no-srcset variant (duration: 01m 16s) |
[production] |
| 19:56 |
<chasemp> |
nfs traffic shaping pilot round 2 |
[tools] |
| 19:26 |
<bd808@mira> |
Synchronized php-1.27.0-wmf.13/extensions/Disambiguator/Disambiguator.hooks.php: Check for array index existence (7b5f87f) (T126651) (duration: 01m 15s) |
[production] |
| 19:16 |
<bd808> |
Wikis back up thankfully |
[production] |
| 19:15 |
<bd808@mira> |
Synchronized php-1.27.0-wmf.13/includes/session/SessionManager.php: Log multiple IPs using the same session or the same user account (4d8b8ca) (T125455) (duration: 01m 17s) |
[production] |
| 19:15 |
<bd808> |
Synced files for T125455 in wrong order; broke all wikis |
[production] |
| 19:14 |
<bd808@mira> |
Synchronized php-1.27.0-wmf.13/includes/Setup.php: Log multiple IPs using the same session or the same user account (4d8b8ca) (duration: 01m 18s) |
[production] |