|
2016-04-07
§
|
| 22:05 |
<Pchelolo> |
finished update RESTBase to 7f69f86ee9 |
[production] |
| 22:00 |
<Pchelolo> |
start update RESTBase to 7f69f86ee9 |
[production] |
| 21:32 |
<mattflaschen@tin> |
Synchronized wmf-config/LabsServices.php: Beta Cluster change (duration: 00m 30s) |
[production] |
| 21:30 |
<krinkle@tin> |
Synchronized w/: I69653efe0f1968: rm old symlinks (duration: 00m 46s) |
[production] |
| 19:55 |
<bblack> |
stop->disable varnishrls service on non-text clusters (upload, maps, misc) - ( https://gerrit.wikimedia.org/r/#/c/281439/ ) |
[production] |
| 19:34 |
<ottomata> |
restarting eventlogging so it runs out of the scap deploy in eventlogging/analytics |
[production] |
| 19:15 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.20 |
[production] |
| 19:00 |
<ejegg> |
updated paymentswiki fraud filter settings |
[production] |
| 16:48 |
<godog> |
start raid expansion on restbase2002 T127951 |
[production] |
| 16:37 |
<godog> |
repool restbase2003, raid expansion finished, depool restbase2002 |
[production] |
| 15:30 |
<thcipriani@tin> |
Synchronized wmf-config/ProductionServices.php: SWAT: Use local resources in codfw for parsoid, url-downloader and mathoid [[gerrit:279355]] (duration: 00m 25s) |
[production] |
| 15:22 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: Use ProductionServices for the jobqueue configuration 3/3 [[gerrit:279350]] (duration: 00m 31s) |
[production] |
| 15:22 |
<thcipriani@tin> |
Synchronized wmf-config/jobqueue.php: SWAT: Use ProductionServices for the jobqueue configuration 2/3 [[gerrit:279350]] (duration: 00m 27s) |
[production] |
| 15:21 |
<thcipriani@tin> |
Synchronized wmf-config/ProductionServices.php: SWAT: Use ProductionServices for the jobqueue configuration 1/3 [[gerrit:279350]] (duration: 00m 30s) |
[production] |
| 15:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor Beta Feature on Wikisources, Wiktionaries [[gerrit:280828]] (duration: 00m 30s) |
[production] |
| 14:47 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 30s) |
[production] |
| 14:47 |
<gehel> |
syncing config to activate switch of CirrusSearch to codfw |
[production] |
| 14:35 |
<gehel> |
switching CirrusSearch to use Elasticsearch cluster in codfw (again), testing on mw1017 first |
[production] |
| 13:58 |
<chasemp> |
labstore restart nfs-export that crashed |
[production] |
| 13:53 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 35s) |
[production] |
| 13:50 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 31s) |
[production] |
| 13:48 |
<gehel> |
switching CirrusSearch to use Elasticsearch cluster in codfw |
[production] |
| 13:28 |
<_joe_> |
restarted hhvm on mw1015 |
[production] |
| 13:22 |
<gehel@tin> |
Synchronized wmf-config: Fix TTMServer elastic config (duration: 00m 32s) |
[production] |
| 12:50 |
<hoo@tin> |
Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 00m 26s) |
[production] |
| 12:49 |
<_joe_> |
restarting hhvm on mw1213, deadlock in HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 12:46 |
<gehel@tin> |
Synchronized wmf-config: Point the codfw label back to the codfw cluster (duration: 00m 27s) |
[production] |
| 12:37 |
<gehel@tin> |
Synchronized wmf-config: Don't use HTTPS+pooling for labswiki (duration: 00m 27s) |
[production] |
| 12:22 |
<gehel@tin> |
Synchronized wmf-config: (no message) (duration: 00m 35s) |
[production] |
| 12:08 |
<gehel> |
enabling HTTPS + connection pooling for CirrusSerach on all mediawiki nodes (T131839) |
[production] |
| 12:06 |
<godog> |
remove restbase2004-b cassandra data directory |
[production] |
| 11:57 |
<godog> |
bump raid rebuild to 20MB/s on restbase2003 |
[production] |
| 10:03 |
<_joe_> |
restarted hhvm on mw1114 (one of the child processes was defunct) and on mw1173 (deadlock in HPHP::Treadmill::getAgeOldestRequest) |
[production] |
| 09:35 |
<gehel> |
deploying HHVM HTTP pool sizing on all MW nodes (T131839 / https://gerrit.wikimedia.org/r/#/c/281881), not used yet, no impact expected |
[production] |
| 09:28 |
<elukey> |
de-pooling/re-pooling aqs100[23].eqiad.wmnet for nodejs upgrade |
[production] |
| 09:19 |
<elukey> |
re-added aqs1001.eqiad.wmnet to LVS pool via confctl |
[production] |
| 08:59 |
<elukey> |
removed aqs1001.eqiad.wmnet from LVS pool via confd for nodejs upgrade |
[production] |
| 08:22 |
<godog> |
stop cassandra bootstrap of restbase2004-b, not enough disk |
[production] |
| 08:17 |
<akosiaris> |
renamed duplicate cxserver-deploy and cxserver repos in phabricator to citoid-deploy and citoid respectively |
[production] |
| 07:04 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Depool crashed db2047, needs to be reimaged T132011 (duration: 00m 38s) |
[production] |
| 03:08 |
<bd808> |
backfilled missing SAL entries from 2016-04-04T17:30Z to 2016-04-05T20:36Z to https://tools.wmflabs.org/sal/production |
[production] |
| 03:08 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Apr 7 03:08:28 UTC 2016 (duration 9m 52s) |
[production] |
| 02:58 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.20) (duration: 10m 00s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.19) (duration: 17m 51s) |
[production] |
| 02:09 |
<chasemp> |
reboot db2047 (it's totally stuck w/ BUG: soft lockup - CPU#26 stuck for 22s! [migration/26:267] across) |
[production] |