9201-9250 of 10000 results (51ms)
2016-09-26 §
17:39 <ejegg> updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [production]
13:52 <marostegui> phabricator is back in write mode - search is degraded. we are regenerating the indexes [production]
13:52 <chasemp> iridium phab ./bin/search index --all [production]
03:39 <cwdent_> disabled civicrm dedupe contacts job [production]
2016-09-25 §
16:25 <Amir1> deploying 9cc2009 on ORES nodes (T146581) [production]
15:24 <elukey> executed https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs on stat1002 (fusermount didn't succeed to umount though) [production]
14:39 <gehel> restarting blazegraph on wdqs1002 - T146576 [production]
14:30 <gehel> putting wdqs1002 in maintenance mode, server looks unstable, investigating... - T146576 [production]
14:28 <godog> temporarily redirect ores-celery-worker logs to /srv/log/celery/syslog.log and remove old daemon.log.1 to avoid scb100* filling up the / filesystem [production]
2016-09-24 §
19:30 <ema> hhvm 1283-1290 rolling restart [production]
12:21 <godog> apply temporary cleanup of old (+20m) thumbor temporary files - T146262 [production]
10:47 <_joe_> systemctl restart thumbor-instances.service on thumbor1001 freed 3 GB of space [production]
02:44 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Sep 24 02:44:59 UTC 2016 (duration 5m 57s) [production]
02:39 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.20) (duration: 16m 49s) [production]
2016-09-23 §
22:05 <matt_flaschen> Deployed patch for T146425 [production]
21:42 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Additional logging to track down autocomplete timing regression (duration: 00m 50s) [production]
20:52 <gehel> cleaning up leftover system unit files on wdqs1* [production]
18:41 <gehel> killing stuck tilerator notification processes on maps1001 - T145534 [production]
17:57 <mutante> mira restarted cron [production]
17:53 <ejegg> updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 [production]
17:46 <thcipriani@tin> Synchronized README: Test sync for new mira (duration: 01m 27s) [production]
17:43 <mutante> mira - changing UID of l10nupdate to 10002, chown'ing files (1001 -> 10002) [production]
17:35 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Add timing marks to narrow down autocomplete timing regression (duration: 00m 50s) [production]
17:31 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/CompletionSuggester.php: Add timing marks to narrow down autocomplete timing regression (duration: 18m 43s) [production]
17:04 <mutante> stat1002 - before it was hanging and then fixed due to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs [production]
17:03 <mutante> stat1002 - starting nagios-nrpe-server [production]
14:55 <jynus> deployed dns update (removing db1010) T129395 [production]
12:20 <moritzm> rearmed keyholder on mira [production]
12:03 <_joe_> rolling restart of mw1280-90, high cpu usage due to memory leaks. [production]
10:16 <moritzm> reimaging mira to jessie (again, previously installer config still pointed to trusty) [production]
10:05 <Amir1> ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki' [production]
10:00 <Amir1> ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki [production]
09:58 <hashar@tin> Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s) [production]
09:58 <moritzm> rearmed keyholder on mira [production]
09:48 <jynus> disabling alerts and shutting down db1010 in preparation for decommissioning T129395 [production]
09:08 <moritzm> reimaging mira to jessie [production]
09:06 <elukey> reboot eventlog2001.codfw.wmnet for kernel upgrades [production]
08:52 <elukey> upgrading varnishkafka to 1.0.12-1 in cache:misc [production]
08:44 <ema> depooled nginx restart on cp4003 and cp1045 for libssl upgrade [production]
08:30 <elukey> upgrading varnishkafka to 1.0.12-1 in cache:maps [production]
07:33 <elukey> executed 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' for all the api and appservers to remove/prevent cronspam (root:adm files also related to new reimaged hosts, Rsyslog needs to be configured before hhvm) - T132324 [production]
07:02 <moritzm> rebooting francium for kernel security update [production]
04:03 <aaron@tin> Synchronized php-1.28.0-wmf.20/includes/deferred: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 48s) [production]
04:02 <aaron@tin> Synchronized php-1.28.0-wmf.20/includes/libs/rdbms: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 51s) [production]
02:46 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Sep 23 02:46:04 UTC 2016 (duration 6m 10s) [production]
02:39 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 04s) [production]
02:13 <maxsem@tin> Synchronized php-1.28.0-wmf.20/extensions/SecurePoll/: https://gerrit.wikimedia.org/r/#/c/312450/1 (duration: 00m 51s) [production]
02:10 <mutante> mw1206, mw1224 - restarted hhvm and apache [production]
01:49 <bblack> depooled mw1224 service apache2 [production]
00:38 <Krenair> mw1224 apache stuck, not restarting for now in case someone wants to investigate later. possibly T89912? [production]