1-50 of 10000 results (12ms)
2015-01-02 §
21:19 <qchris> Ran kafka leader re-election to bring analytics1021 back into the set of leaders [production]
11:48 <godog> reboot es2004, debugging gmond stuck on start/stop [production]
04:59 <springle> Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s) [production]
03:29 <springle> clone and deploy es2002 es2003 es2004 [production]
2015-01-01 §
15:19 <springle> upgrade db1061 trusty [production]
15:19 <springle> upgrade db1061 trusty [production]
2014-12-31 §
23:45 <springle> Synchronized wmf-config/db-eqiad.php: depool db1061 (duration: 00m 05s) [production]
14:02 <springle> Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 06s) [production]
09:47 <godog> updating precise-wikimedia from third-party repo (hwraid) [production]
09:45 <godog> previous reprepro update also accidentally updated elasticsearch in trusty-wikimedia to 1.3.7 [production]
09:43 <godog> updating trusty-wikimedia from third-party repo (hwraid) [production]
02:22 <springle> upgrade db1065 trusty [production]
02:16 <springle> Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 05s) [production]
02:03 <awight> updated payments from 78b72063e4e0cc76b7e168be1e626d5e10e34d4a to 62c81d4574e5e994ff8f3cac7115eff335bd5265 [production]
00:52 <bd808> restarted elasticsearch on logstash1001 [production]
00:49 <awight> updated payments from e81f473acc5b31b49dd27714c40f9b71c3462e26 to 78b72063e4e0cc76b7e168be1e626d5e10e34d4a [production]
00:42 <bd808> log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. [production]
00:33 <bd808> restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch [production]
2014-12-30 §
21:52 <bd808> restarted elasticsearch on logstash1002; it had dropped from the cluster [production]
20:46 <yurik> Synchronized wmf-config/CommonSettings.php: ZeroPortal 182227 (duration: 00m 06s) [production]
19:06 <paravoid> manually stopping acct on neon and setting /etc/default/acct ACCT_ENABLE to 0 [production]
16:38 <godog> killing uwsgi on tunsten, blew memory [production]
14:46 <Nemo_bis> morebots is being rude today [production]
14:36 <hoo> Synchronized wmf-config/CommonSettings.php: Enable unregistered users editing on it.m.wikipedia.org after Dec 31 (duration: 00m 06s) [production]
2014-12-29 §
20:19 <awight> payments updated from ce7fb9af37c4bba2a84668387b61729df4f9723c to e81f473acc5b31b49dd27714c40f9b71c3462e26 [production]
10:35 <godog> reboot ms-be2011, stuck while removing a LD, no console [production]
2014-12-27 §
23:33 <paravoid> restarting puppetmasters [production]
20:29 <gwicke> dropped old keyspaces titan{,2,3} on xenon to free space for titan4 [production]
19:53 <ori> gallium: restarted jenkins [production]
16:19 <Reedy> jenkins started again... [production]
16:17 <Reedy> jenkins killed [production]
16:12 <Reedy> attempting to kill jenkins [production]
16:11 <Reedy> jenkins is hung with high cpu/memory usage [production]
12:55 <springle> Synchronized wmf-config/db-eqiad.php: bump up s1 api load sent to db1066 (duration: 00m 06s) [production]
12:11 <springle> Synchronized wmf-config/db-eqiad.php: repool db1066 and db1028, warm up (duration: 00m 06s) [production]
05:49 <ori> cerium disk space critical, so moved /mnt/data/cassandra/java_{1418354329,1418533386,1418537719}.hprof to /tmp/hprof_files, freeing up ~17G of space. [production]
2014-12-25 §
20:44 <_joe_> restarting hhvm on mw1239, stuck in HPHP::is_valid_var_name probably after trying to call ini_set [production]
00:32 <hoo> Synchronized wmf-config/Bug54847.php: Fix for invalid hashes (this prevented some people from logging in) (duration: 00m 05s) [production]
00:26 <spage> Synchronized php-1.25wmf12/extensions/PageTriage/modules/ext.pageTriage.views.toolbar/ext.pageTriage.delete.js: Unbreak page curation on enwiki for Xmas (duration: 00m 05s) [production]
00:20 <spage> Synchronized php-1.25wmf13/extensions/PageTriage/modules/ext.pageTriage.views.toolbar/ext.pageTriage.delete.js: Unbreak page curation (duration: 00m 06s) [production]
2014-12-24 §
18:42 <paravoid> manually running debmirror on carbon to sync over the holidays; "pkill -f debmirror" should suffice if there is a problem [production]
14:57 <akosiaris> disabled puppet on helium while testing copy jobs [production]
13:59 <Jeff_Green> package updates and reboots for several fundraising servers... [production]
01:17 <hoo> Ran mysql:wikiadmin@db1033 [centralauth]> DELETE FROM bug_54847_password_resets WHERE r_username = 'Stilfehler'; [production]
2014-12-23 §
22:21 <hoo> Synchronized wmf-config/: Syncing Kaldari's beta-only change (duration: 00m 07s) [production]
20:34 <K4-713> Re-enabled Thank You send job [production]
19:53 <anomie> Synchronized wmf-config: Labs-only change (duration: 00m 06s) [production]
19:43 <K4-713> disabled Thank You email send job [production]
19:40 <chasemp> updated phab sprint app to 0.6.1.4 [production]
18:32 <paravoid> restarting icing [production]