|
2016-01-25
§
|
| 09:31 |
<_joe_> |
rolling reboot of the eqiad appserver cluster |
[production] |
| 09:27 |
<moritzm> |
installed fuse security update on labnodepool1001 (the other fuse installations are on Ubuntu, which doesn't ship the udev rule, but uses mountall instead) |
[production] |
| 07:47 |
<paravoid> |
stat1002: umount -f /mnt/hdfs |
[production] |
| 07:34 |
<_joe_> |
rebooting alsafi, unresponsive to ssh |
[production] |
| 07:24 |
<_joe_> |
restarting hhvm on mw1148, stuck in HPHP::Treadmill::startRequest (__lll_lock_wait) |
[production] |
| 07:23 |
<_joe_> |
restarting hhvm on mw1143, stuck into HPHP::SynchronizableMulti::waitImpl (__pthread_cond_wait) |
[production] |
| 03:10 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s) |
[production] |
| 03:03 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s) |
[production] |
| 03:02 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserOutput.php: (no message) (duration: 00m 27s) |
[production] |
| 02:30 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Jan 25 02:30:13 UTC 2016 (duration 6m 52s) |
[production] |
| 02:23 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.10) (duration: 09m 09s) |
[production] |
|
2016-01-23
§
|
| 19:03 |
<ebernhardson@tin> |
Synchronized wmf-config/CirrusSearch-production.php: config change to repoint morelike search from eqiad to codfw (duration: 00m 26s) |
[production] |
| 19:02 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/: Support code for repointing morelike queries from eqiad to codfw (duration: 00m 30s) |
[production] |
| 19:00 |
<ebernhardson> |
repoint most expensive search queries (morelike) at codfw cluster to reduce load. 1/2 of eqiad cluster maxed on cpu |
[production] |
| 16:47 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki wowiki |
[production] |
| 13:25 |
<jynus> |
upgrading and restarting db1046 |
[production] |
| 13:13 |
<jynus> |
db1046 maintenance finished- restarting mysql to apply latest configuration |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Jan 23 02:32:15 UTC 2016 (duration 7m 3s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.10) (duration: 09m 09s) |
[production] |
| 01:33 |
<bd808@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: Back to 1.27.0-wmf10 again after fixking l10n cache problems |
[production] |
| 01:28 |
<bd808@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: Temporarily back to 1.27.0-wmf11; need to rebuild l10n cache |
[production] |
| 01:16 |
<bd808@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: Revert all wikis to 1.27.0-wmf.10 |
[production] |
| 00:08 |
<bd808@tin> |
Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265872/ (duration: 00m 25s) |
[production] |
| 00:07 |
<bd808@tin> |
Synchronized php-1.27.0-wmf.11/includes/session/CookieSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265871/ (duration: 00m 25s) |
[production] |
|
2016-01-22
§
|
| 23:43 |
<legoktm@tin> |
Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265870/ (duration: 00m 26s) |
[production] |
| 23:42 |
<legoktm@tin> |
Synchronized php-1.27.0-wmf.11/includes/session/CookieSessionProvider.php: https://gerrit.wikimedia.org/r/#/c/265869/ (duration: 00m 26s) |
[production] |
| 23:22 |
<mobrovac> |
restbase cassandra truncating local_group_wiktionary_T_term_definition.data |
[production] |
| 22:33 |
<mdholloway> |
mobileapps deployed 2900faa |
[production] |
| 22:23 |
<twentyafterfour@tin> |
Finished scap: deploy https://gerrit.wikimedia.org/r/#/c/263415/ and clean up old branches (duration: 07m 02s) |
[production] |
| 22:16 |
<twentyafterfour@tin> |
Started scap: deploy https://gerrit.wikimedia.org/r/#/c/263415/ and clean up old branches |
[production] |
| 22:06 |
<bblack> |
upgrading vhtcpd on all caches |
[production] |
| 21:49 |
<anomie@tin> |
Synchronized php-1.27.0-wmf.11/includes/: Fix T124468, for real this time (duration: 00m 36s) |
[production] |
| 21:48 |
<anomie@tin> |
Synchronized php-1.27.0-wmf.11/includes/: Fix T124468 (duration: 00m 38s) |
[production] |
| 21:17 |
<legoktm> |
running migrateAccount.php --attachbroken over list of all unattached users (T74791) |
[production] |
| 20:04 |
<mutante> |
ruthenium - rebooting for reinstall |
[production] |
| 19:42 |
<aaron@tin> |
Synchronized wmf-config/CommonSettings.php: Revert "Bump $wgJobBackoffThrottling to lower the htmlcacheupdate backlog" (duration: 00m 32s) |
[production] |
| 18:51 |
<jynus> |
"repairing" enwiki.oldtable on dbstore1001 |
[production] |
| 18:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Aborting pc1001 maintenance (duration: 00m 31s) |
[production] |
| 18:15 |
<legoktm> |
running CentralAuth's resetGlobalUserTokens.php to force session resets for all users T124440 |
[production] |
| 18:02 |
<anomie@tin> |
Synchronized php-1.27.0-wmf.11/includes/user/User.php: Fix T124414 (duration: 00m 33s) |
[production] |
| 17:53 |
<legoktm> |
manually attaching User:Mower Genetics and User:Themeetingplace because they made edits somehow (T74791) |
[production] |
| 17:46 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Stop logging the CirrusSearchRequests channel (duration: 00m 32s) |
[production] |
| 17:44 |
<legoktm> |
running migrateAccount.php --attachbroken over lists on T74791 |
[production] |
| 17:39 |
<_joe_> |
removed an archived CirrusSearchRequests.log on fluorine, now we have enough room for the weekend |
[production] |
| 17:29 |
<anomie@tin> |
Synchronized php-1.27.0-wmf.11/extensions/CentralAuth/includes: Fix T124406 (duration: 00m 35s) |
[production] |
| 17:04 |
<mobrovac> |
mobileapps deploying bba45456 |
[production] |
| 17:00 |
<reedy@tin> |
Synchronized docroot and w: Extra noc symlinks (duration: 00m 32s) |
[production] |
| 16:58 |
<jynus@tin> |
Synchronized wmf-config/InitialiseSettings.php: monolog: reduce on-disk logging of DBPerformance to warning (duration: 00m 32s) |
[production] |