|
2016-04-07
§
|
| 13:58 |
<chasemp> |
labstore restart nfs-export that crashed |
[production] |
| 13:53 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 35s) |
[production] |
| 13:50 |
<gehel@tin> |
Synchronized wmf-config: switching CirrusSearch to use Elasticsearch cluster in codfw (duration: 00m 31s) |
[production] |
| 13:48 |
<gehel> |
switching CirrusSearch to use Elasticsearch cluster in codfw |
[production] |
| 13:28 |
<_joe_> |
restarted hhvm on mw1015 |
[production] |
| 13:22 |
<gehel@tin> |
Synchronized wmf-config: Fix TTMServer elastic config (duration: 00m 32s) |
[production] |
| 12:50 |
<hoo@tin> |
Synchronized wmf-config/Wikibase.php: Bump $wgCacheEpoch on Wikidata after Property conversions (duration: 00m 26s) |
[production] |
| 12:49 |
<_joe_> |
restarting hhvm on mw1213, deadlock in HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 12:46 |
<gehel@tin> |
Synchronized wmf-config: Point the codfw label back to the codfw cluster (duration: 00m 27s) |
[production] |
| 12:37 |
<gehel@tin> |
Synchronized wmf-config: Don't use HTTPS+pooling for labswiki (duration: 00m 27s) |
[production] |
| 12:22 |
<gehel@tin> |
Synchronized wmf-config: (no message) (duration: 00m 35s) |
[production] |
| 12:08 |
<gehel> |
enabling HTTPS + connection pooling for CirrusSerach on all mediawiki nodes (T131839) |
[production] |
| 12:06 |
<godog> |
remove restbase2004-b cassandra data directory |
[production] |
| 11:57 |
<godog> |
bump raid rebuild to 20MB/s on restbase2003 |
[production] |
| 10:03 |
<_joe_> |
restarted hhvm on mw1114 (one of the child processes was defunct) and on mw1173 (deadlock in HPHP::Treadmill::getAgeOldestRequest) |
[production] |
| 09:35 |
<gehel> |
deploying HHVM HTTP pool sizing on all MW nodes (T131839 / https://gerrit.wikimedia.org/r/#/c/281881), not used yet, no impact expected |
[production] |
| 09:28 |
<elukey> |
de-pooling/re-pooling aqs100[23].eqiad.wmnet for nodejs upgrade |
[production] |
| 09:19 |
<elukey> |
re-added aqs1001.eqiad.wmnet to LVS pool via confctl |
[production] |
| 08:59 |
<elukey> |
removed aqs1001.eqiad.wmnet from LVS pool via confd for nodejs upgrade |
[production] |
| 08:22 |
<godog> |
stop cassandra bootstrap of restbase2004-b, not enough disk |
[production] |
| 08:17 |
<akosiaris> |
renamed duplicate cxserver-deploy and cxserver repos in phabricator to citoid-deploy and citoid respectively |
[production] |
| 07:04 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Depool crashed db2047, needs to be reimaged T132011 (duration: 00m 38s) |
[production] |
| 03:08 |
<bd808> |
backfilled missing SAL entries from 2016-04-04T17:30Z to 2016-04-05T20:36Z to https://tools.wmflabs.org/sal/production |
[production] |
| 03:08 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Apr 7 03:08:28 UTC 2016 (duration 9m 52s) |
[production] |
| 02:58 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.20) (duration: 10m 00s) |
[production] |
| 02:38 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.19) (duration: 17m 51s) |
[production] |
| 02:09 |
<chasemp> |
reboot db2047 (it's totally stuck w/ BUG: soft lockup - CPU#26 stuck for 22s! [migration/26:267] across) |
[production] |
|
2016-04-06
§
|
| 23:44 |
<mattflaschen@tin> |
Synchronized php-1.27.0-wmf.19/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 29s) |
[production] |
| 23:40 |
<mattflaschen@tin> |
Synchronized php-1.27.0-wmf.20/extensions/Flow/includes/Data/Listener/NotificationListener.php: Fix new topic notifications (duration: 00m 37s) |
[production] |
| 23:11 |
<mattflaschen@tin> |
Synchronized docroot/wikipedia.org/apple-app-site-association: Support handoff and credential sharing with the iOS app (duration: 00m 34s) |
[production] |
| 22:49 |
<bd808> |
Testing stashbot response in #wikimedia-fundraising |
[production] |
| 20:17 |
<subbu> |
finished deploying parsoid sha 5f6c0c60 |
[production] |
| 20:10 |
<subbu> |
synced code; restarted parsoid on wtp1001 as a canary |
[production] |
| 20:06 |
<subbu> |
starting parsoid deploy |
[production] |
| 19:50 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: Promote labswiki to 1.27.0-wmf.20 following temporary rollback and fix |
[production] |
| 19:49 |
<dduvall@tin> |
Synchronized php-1.27.0-wmf.20/extensions/SemanticMediaWiki/includes/SMW_ParserExtensions.php: Replace usage of Title::newFromRedirect() (duration: 00m 38s) |
[production] |
| 19:33 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: (no message) |
[production] |
| 19:24 |
<akosiaris> |
pool maps-test200{1,2,3} for kartotherian.svc.codfw.wmnet |
[production] |
| 19:22 |
<dduvall@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.20 |
[production] |
| 19:22 |
<akosiaris> |
bounce hhvm on mw1135, mw1145 |
[production] |
| 19:16 |
<akosiaris> |
restart tilerator, tileratorui on maps-test200{1,2,3} |
[production] |
| 19:15 |
<akosiaris> |
restart kartotherian on maps-test200{1,2,3} |
[production] |
| 18:58 |
<YuviPanda> |
reboot notebook1001 |
[production] |
| 18:50 |
<akosiaris> |
disable salt-minion on maps-test200{1,2,3} for maps services deployment. nodejs upgrade is in place |
[production] |
| 17:00 |
<thcipriani@tin> |
Finished scap: SWAT: Add user_wpzero AbuseFilter variable [[gerrit:281867]] (duration: 27m 39s) |
[production] |
| 16:32 |
<thcipriani@tin> |
Started scap: SWAT: Add user_wpzero AbuseFilter variable [[gerrit:281867]] |
[production] |
| 16:06 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Language Overlay experiment [[gerrit:277837]] (duration: 00m 26s) |
[production] |
| 15:54 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on uawikimedia [[gerrit:281403]] (duration: 00m 28s) |
[production] |
| 15:33 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: REVERT Enable Translate extension on uawikimedia [[gerrit:281403]] (duration: 00m 25s) |
[production] |
| 15:31 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on uawikimedia [[gerrit:281403]] (duration: 00m 27s) |
[production] |