|
2016-04-07
§
|
| 16:48 |
<godog> |
start raid expansion on restbase2002 T127951 |
[production] |
| 16:37 |
<godog> |
repool restbase2003, raid expansion finished, depool restbase2002 |
[production] |
| 15:30 |
<thcipriani@tin> |
Synchronized wmf-config/ProductionServices.php: SWAT: Use local resources in codfw for parsoid, url-downloader and mathoid [[gerrit:279355]] (duration: 00m 25s) |
[production] |
| 15:22 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: Use ProductionServices for the jobqueue configuration 3/3 [[gerrit:279350]] (duration: 00m 31s) |
[production] |
| 15:22 |
<thcipriani@tin> |
Synchronized wmf-config/jobqueue.php: SWAT: Use ProductionServices for the jobqueue configuration 2/3 [[gerrit:279350]] (duration: 00m 27s) |
[production] |
| 15:21 |
<thcipriani@tin> |
Synchronized wmf-config/ProductionServices.php: SWAT: Use ProductionServices for the jobqueue configuration 1/3 [[gerrit:279350]] (duration: 00m 30s) |
[production] |
| 15:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor Beta Feature on Wikisources, Wiktionaries [[gerrit:280828]] (duration: 00m 30s) |
[production] |
| 14:47 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 30s) |
[production] |
| 14:47 |
<gehel> |
syncing config to activate switch of CirrusSearch to codfw |
[production] |
| 14:35 |
<gehel> |
switching CirrusSearch to use Elasticsearch cluster in codfw (again), testing on mw1017 first |
[production] |
| 13:58 |
<chasemp> |
labstore restart nfs-export that crashed |
[production] |
| 13:53 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 35s) |
[production] |
| 13:50 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 31s) |
[production] |
| 13:48 |
<gehel> |
switching CirrusSearch to use Elasticsearch cluster in codfw |
[production] |
| 13:28 |
<_joe_> |
restarted hhvm on mw1015 |
[production] |
| 13:22 |
<gehel@tin> |
Synchronized wmf-config: Fix TTMServer elastic config (duration: 00m 32s) |
[production] |
| 12:50 |
<hoo@tin> |
Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 00m 26s) |
[production] |
| 12:49 |
<_joe_> |
restarting hhvm on mw1213, deadlock in HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 12:46 |
<gehel@tin> |
Synchronized wmf-config: Point the codfw label back to the codfw cluster (duration: 00m 27s) |
[production] |
| 12:37 |
<gehel@tin> |
Synchronized wmf-config: Don't use HTTPS+pooling for labswiki (duration: 00m 27s) |
[production] |
| 12:22 |
<gehel@tin> |
Synchronized wmf-config: (no message) (duration: 00m 35s) |
[production] |
| 12:08 |
<gehel> |
enabling HTTPS + connection pooling for CirrusSerach on all mediawiki nodes (T131839) |
[production] |
| 12:06 |
<godog> |
remove restbase2004-b cassandra data directory |
[production] |
| 11:57 |
<godog> |
bump raid rebuild to 20MB/s on restbase2003 |
[production] |
| 10:03 |
<_joe_> |
restarted hhvm on mw1114 (one of the child processes was defunct) and on mw1173 (deadlock in HPHP::Treadmill::getAgeOldestRequest) |
[production] |
| 09:35 |
<gehel> |
deploying HHVM HTTP pool sizing on all MW nodes (T131839 / https://gerrit.wikimedia.org/r/#/c/281881), not used yet, no impact expected |
[production] |
| 09:28 |
<elukey> |
de-pooling/re-pooling aqs100[23].eqiad.wmnet for nodejs upgrade |
[production] |
| 09:19 |
<elukey> |
re-added aqs1001.eqiad.wmnet to LVS pool via confctl |
[production] |
| 08:59 |
<elukey> |
removed aqs1001.eqiad.wmnet from LVS pool via confd for nodejs upgrade |
[production] |
| 08:22 |
<godog> |
stop cassandra bootstrap of restbase2004-b, not enough disk |
[production] |
| 08:17 |
<akosiaris> |
renamed duplicate cxserver-deploy and cxserver repos in phabricator to citoid-deploy and citoid respectively |
[production] |
| 07:04 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Depool crashed db2047, needs to be reimaged T132011 (duration: 00m 38s) |
[production] |
| 03:08 |
<bd808> |
backfilled missing SAL entries from 2016-04-04T17:30Z to 2016-04-05T20:36Z to https://tools.wmflabs.org/sal/production |
[production] |
| 03:08 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Apr 7 03:08:28 UTC 2016 (duration 9m 52s) |
[production] |
| 02:58 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.20) (duration: 10m 00s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.19) (duration: 17m 51s) |
[production] |
| 02:09 |
<chasemp> |
reboot db2047 (it's totally stuck w/ BUG: soft lockup - CPU#26 stuck for 22s! [migration/26:267] across) |
[production] |
|
2016-04-06
§
|
| 23:44 |
<mattflaschen@tin> |
Synchronized php-1.27.0-wmf.19/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 29s) |
[production] |
| 23:40 |
<mattflaschen@tin> |
Synchronized php-1.27.0-wmf.20/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 37s) |
[production] |
| 23:11 |
<mattflaschen@tin> |
Synchronized docroot/wikipedia.org/apple-app-site-association: Support handoff and credential sharing with the iOS app (duration: 00m 34s) |
[production] |
| 22:49 |
<bd808> |
Testing stashbot response in #wikimedia-fundraising |
[production] |
| 20:17 |
<subbu> |
finished deploying parsoid sha 5f6c0c60 |
[production] |
| 20:10 |
<subbu> |
synced code; restarted parsoid on wtp1001 as a canary |
[production] |
| 20:06 |
<subbu> |
starting parsoid deploy |
[production] |
| 19:50 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: Promote labswiki to 1.27.0-wmf.20 following temporary rollback and fix |
[production] |
| 19:49 |
<dduvall@tin> |
Synchronized php-1.27.0-wmf.20/extensions/SemanticMediaWiki/includes/SMW_ParserExtensions.php: Replace usage of Title::newFromRedirect() (duration: 00m 38s) |
[production] |
| 19:33 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: (no message) |
[production] |
| 19:24 |
<akosiaris> |
pool maps-test200{1,2,3} for kartotherian.svc.codfw.wmnet |
[production] |
| 19:22 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.20 |
[production] |
| 19:22 |
<akosiaris> |
bounce hhvm on mw1135, mw1145 |
[production] |