|
2016-01-25
§
|
| 19:04 |
<chasemp> |
nfsd has 224 threads atm and was bumped up over the weekend |
[production] |
| 18:59 |
<twentyafterfour> |
started redis-server on deployment-redis01 by commenting out latency-monitor-threshold from the redis.conf |
[releng] |
| 18:58 |
<ori> |
removed unused wikiversions.cdb on mira and tin |
[production] |
| 18:28 |
<jynus> |
retroactively logging the depool of mw1217, mw1178 and mw1257 3 hours ago (Jan 25 15:45:26) |
[production] |
| 16:49 |
<ema> |
Finished migration of mobile traffic to text cluster in ulsfo https://phabricator.wikimedia.org/T109286 |
[production] |
| 16:38 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (third try) (duration: 01m 35s) |
[production] |
| 16:26 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (second try after running puppet) (duration: 03m 23s) |
[production] |
| 16:25 |
<_joe_> |
restarting salt-minion on all deployment targets |
[production] |
| 16:24 |
<_joe_> |
running salt deploy.fixurl on all deployment targets |
[production] |
| 16:09 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Preparing ips for new parsercache deployments (duration: 03m 32s) |
[production] |
| 15:35 |
<ema> |
Starting migration of mobile traffic to text cluster in ulsfo https://phabricator.wikimedia.org/T109286 |
[production] |
| 15:22 |
<hashar> |
CI: fixing kernels not upgrading via: rm /boot/grub/menu.lst ; update-grub -y (i.e.: regenerate the Grub menu from scratch) |
[releng] |
| 15:14 |
<chasemp> |
restart of pdns and pdns-recursor on labservices1001 |
[production] |
| 14:56 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: deploy new parsercache hardware (pc1004) substituting pc1001 (duration: 03m 25s) |
[production] |
| 14:54 |
<ottomata> |
restarting eventlogging mysql consumers after TokuDB conversion has finished |
[analytics] |
| 14:21 |
<hashar> |
integration-slave-trusty-1015.integration.eqiad.wmflabs is gone. I have failed the kernel upgrade / grub update |
[releng] |
| 13:16 |
<mforns> |
restarted EventLogging after Kafka leader reelection |
[analytics] |
| 13:16 |
<elukey> |
ran kafka preferred-replica-election on kafka1022 to balance the leaders |
[production] |
| 13:07 |
<elukey> |
restarting kafka on kafka1022 |
[production] |
| 12:57 |
<elukey> |
restarting kafka on kafka1013 |
[production] |
| 12:38 |
<elukey> |
restarting kafka on kafka1014 |
[production] |
| 12:20 |
<jynus> |
compressed and truncated iridium's phab daemons.log - it was taking 20% of disk space |
[production] |
| 12:08 |
<mforns> |
restarted EventLogging to sync with kafka1018 restart |
[analytics] |
| 12:04 |
<ema> |
restarting kafka on kafka1018 |
[production] |
| 11:26 |
<jynus> |
stopping mysql at pc1001 and cloning to pc1004 |
[production] |
| 10:57 |
<mforns> |
restarted EventLogging to overcome connection problems with Kafka |
[analytics] |
| 10:55 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Depool pc1001 for maintenance (clone to pc1004) (duration: 01m 41s) |
[production] |
| 10:11 |
<_joe_> |
switching the active deployment host to mira |
[production] |
| 09:56 |
<ema> |
limiting GCLogFileSize and restarting kafka on kafka1012 |
[production] |
| 09:31 |
<_joe_> |
rolling reboot of the eqiad appserver cluster |
[production] |
| 09:27 |
<moritzm> |
installed fuse security update on labnodepool1001 (the other fuse installations are on Ubuntu, which doesn't ship the udev rule, but uses mountall instead) |
[production] |
| 07:47 |
<paravoid> |
stat1002: umount -f /mnt/hdfs |
[production] |
| 07:34 |
<_joe_> |
rebooting alsafi, unresponsive to ssh |
[production] |
| 07:24 |
<_joe_> |
restarting hhvm on mw1148, stuck in HPHP::Treadmill::startRequest (__lll_lock_wait) |
[production] |
| 07:23 |
<_joe_> |
restarting hhvm on mw1143, stuck into HPHP::SynchronizableMulti::waitImpl (__pthread_cond_wait) |
[production] |
| 03:10 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s) |
[production] |
| 03:03 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserCache.php: (no message) (duration: 00m 25s) |
[production] |
| 03:02 |
<tstarling@tin> |
Synchronized php-1.27.0-wmf.10/includes/parser/ParserOutput.php: (no message) (duration: 00m 27s) |
[production] |
| 02:30 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Jan 25 02:30:13 UTC 2016 (duration 6m 52s) |
[production] |
| 02:23 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.10) (duration: 09m 09s) |
[production] |
| 01:35 |
<bd808> |
integration-slave-jessie-1001:/mnt full; cleaned up 15G of files in /mnt/pbuilder/build |
[releng] |