4251-4300 of 10000 results (49ms)
2016-09-28 §
20:47 <bd808> logmsgbot seems to be down: "error: [Errno 111] Connection refused" from scap sync-file [production]
20:46 <bd808> scap sync-file wmf-config/throttle.php "IP cap lift for eswiki on 2016-09-30 (T146788)" [production]
19:48 <awight> update fundraising CRM from 6b2bd98fce5006030423bccc4f4b7fd9b5d14821 to 86371233a9218a526f85cd76d7008f966042fde7 [production]
19:22 <XenoRyet> reverted SmashPig from 4b930ada57d99070713be18e235238078e1c1e48 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 [production]
17:22 <jynus> restarting db1069.s3 (stagnant replication) [production]
16:42 <awight> update CRM from 88000acc826592e535491bde6a47f8976deaf07a to 6b2bd98fce5006030423bccc4f4b7fd9b5d14821 [production]
08:15 <twentyafterfour> twentyafterfour@iridium:/srv/phab/phabricator$ sudo bin/search index --type PhabricatorProject --force [production]
03:34 <eileen> tools upgrade from b0be0f9ca04191c4bab869bb81191c5c77c432ca to 5427a601ed64ecfe01ef84eda13f28c0dd08a3af [production]
01:20 <eileen> updating CiviCRM from 26e5214e66f2e4b5de89c27dd5121dfde43269e2 to 88000acc826592e535491bde6a47f8976deaf07a [production]
01:05 <eileen> civicrm upgrade from d30a5e454e6614214faef8354a3a23d71d573c7f to 26e5214e66f2e4b5de89c27dd5121dfde43269e2 [production]
2016-09-27 §
05:32 <_joe_> rebooting ms-be1002, stuck in a failed disk [production]
2016-09-26 §
19:37 <awight> enabling awight_test5 banner at 1% of nlwiki [production]
18:09 <ejegg> rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d [production]
17:59 <ejegg> updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [production]
17:47 <ejegg> rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d [production]
17:39 <ejegg> updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c [production]
13:52 <marostegui> phabricator is back in write mode - search is degraded. we are regenerating the indexes [production]
13:52 <chasemp> iridium phab ./bin/search index --all [production]
03:39 <cwdent_> disabled civicrm dedupe contacts job [production]
2016-09-25 §
16:25 <Amir1> deploying 9cc2009 on ORES nodes (T146581) [production]
15:24 <elukey> executed https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs on stat1002 (fusermount didn't succeed to umount though) [production]
14:39 <gehel> restarting blazegraph on wdqs1002 - T146576 [production]
14:30 <gehel> putting wdqs1002 in maintenance mode, server looks unstable, investigating... - T146576 [production]
14:28 <godog> temporarily redirect ores-celery-worker logs to /srv/log/celery/syslog.log and remove old daemon.log.1 to avoid scb100* filling up the / filesystem [production]
2016-09-24 §
19:30 <ema> hhvm 1283-1290 rolling restart [production]
12:21 <godog> apply temporary cleanup of old (+20m) thumbor temporary files - T146262 [production]
10:47 <_joe_> systemctl restart thumbor-instances.service on thumbor1001 freed 3 GB of space [production]
02:44 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Sep 24 02:44:59 UTC 2016 (duration 5m 57s) [production]
02:39 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.20) (duration: 16m 49s) [production]
2016-09-23 §
22:05 <matt_flaschen> Deployed patch for T146425 [production]
21:42 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Additional logging to track down autocomplete timing regression (duration: 00m 50s) [production]
20:52 <gehel> cleaning up leftover system unit files on wdqs1* [production]
18:41 <gehel> killing stuck tilerator notification processes on maps1001 - T145534 [production]
17:57 <mutante> mira restarted cron [production]
17:53 <ejegg> updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 [production]
17:46 <thcipriani@tin> Synchronized README: Test sync for new mira (duration: 01m 27s) [production]
17:43 <mutante> mira - changing UID of l10nupdate to 10002, chown'ing files (1001 -> 10002) [production]
17:35 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Add timing marks to narrow down autocomplete timing regression (duration: 00m 50s) [production]
17:31 <ebernhardson@tin> Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/CompletionSuggester.php: Add timing marks to narrow down autocomplete timing regression (duration: 18m 43s) [production]
17:04 <mutante> stat1002 - before it was hanging and then fixed due to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs [production]
17:03 <mutante> stat1002 - starting nagios-nrpe-server [production]
14:55 <jynus> deployed dns update (removing db1010) T129395 [production]
12:20 <moritzm> rearmed keyholder on mira [production]
12:03 <_joe_> rolling restart of mw1280-90, high cpu usage due to memory leaks. [production]
10:16 <moritzm> reimaging mira to jessie (again, previously installer config still pointed to trusty) [production]
10:05 <Amir1> ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki' [production]
10:00 <Amir1> ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki [production]
09:58 <hashar@tin> Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s) [production]
09:58 <moritzm> rearmed keyholder on mira [production]
09:48 <jynus> disabling alerts and shutting down db1010 in preparation for decommissioning T129395 [production]