8951-9000 of 10000 results (31ms)
2015-04-16 §
02:29 <l10nupdate> Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 05m 57s) [production]
02:24 <andrewbogott> but the ‘token’ table is still too big to manage [production]
02:24 <andrewbogott> restarted mysql on virt1000 because keystone was stuck. It seems to have helped, eventually [production]
02:24 <andrewbogott> restarted keystone and nova-scheduler in a failed attempt to unstick things [production]
02:23 <andrewbogott> testing the log by logging a test [production]
2015-04-15 §
23:21 <Krinkle> beta-update-databases-eqiad stuck waiting for executors on a node that has plenty executors available [releng]
21:15 <hashar> Jenkins browser test jobs sometime deadlock because of the IRC notification plugin https://phabricator.wikimedia.org/T96183 [releng]
20:34 <hashar> hard restarting Jenkins [releng]
19:24 <Krinkle> Aborting browser tests jobs. Stuck for over 5 hours. [releng]
19:24 <Krinkle> Aborting beta-scap-eqiad. Has been stuck for 2 hours on "Notifying IRC" after "Connection time out" from scap. [releng]
18:25 <legoktm> running forceRenameUsers.php (SUL finalization) on test* wikis [production]
17:15 <legoktm> running migrateAccount.php --auto (CentralAuth) [production]
16:49 <ottomata> rebooting analyics1020 [production]
16:45 <nuria> restarted eventlogging && deployed d241d75ee2fab554bc47cf8d1ba83f5df2130633 [production]
16:29 <demon> Synchronized php-1.26wmf1/extensions/CentralAuth/: (no message) (duration: 00m 13s) [production]
16:22 <gwicke> running revision render thin-out script on wikipedia HTML [production]
15:35 <bblack> re-enabling puppet on caches, canary nodes were no-op \\o/ [production]
15:27 <bblack> disabling puppet on caches JIC for https://gerrit.wikimedia.org/r/204068 merge [production]
14:54 <legoktm> running deleteEmptyAccounts.php --fix on metawiki (CentralAuth) [production]
13:54 <andrewbogott> purging expired keystone tokens on virt1000 [production]
12:59 <andrewbogott> restarted pdns on virt1000 and labcontrol2001 to recover from the opendj restart [production]
12:58 <akosiaris> restarted keystone, nova services on virt1000 [production]
12:58 <akosiaris> restarted opendj on neptunium [production]
12:58 <andrewbogott> dropping labswiki and labswiki_eqiad from mysql on virt1000 [production]
11:20 <_joe_> restarted apache2 on virt1000, passenger gone to hell [production]
10:31 <godog> bounce jobchron on mw1001 [production]
10:30 <godog> restart keystone on virt1000 (#2) [production]
08:22 <hashar> restarted Jenkins [releng]
08:20 <hashar> Exception in thread "RequestHandlerThread[#2]" java.lang.OutOfMemoryError: Java heap space [releng]
08:16 <hashar> Jenkins process went wild taking all CPU busy on gallium [releng]
08:12 <godog> bounce keystone on virt1000 [production]
04:16 <ebernhardson> Synchronized wmf-config/CommonSettings.php: Bump flow cache version (duration: 00m 11s) [production]
03:11 <LocalisationUpdate> completed (1.26wmf1) at 2015-04-15 03:10:08+00:00 [production]
03:04 <l10nupdate> Synchronized php-1.26wmf1/cache/l10n: (no message) (duration: 09m 22s) [production]
02:39 <LocalisationUpdate> completed (1.25wmf24) at 2015-04-15 02:37:59+00:00 [production]
02:32 <l10nupdate> Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 09m 03s) [production]
2015-04-14 §
23:54 <catrope> Synchronized php-1.25wmf24/extensions/VisualEditor: SWAT (duration: 00m 12s) [production]
23:54 <catrope> Synchronized php-1.25wmf24/extensions/CentralAuth: SWAT (duration: 00m 13s) [production]
23:54 <catrope> Synchronized php-1.25wmf24/extensions/Gather: SWAT (duration: 00m 12s) [production]
23:54 <catrope> Synchronized php-1.25wmf24/extensions/WikiEditor: SWAT (duration: 00m 11s) [production]
23:52 <catrope> Synchronized php-1.26wmf1/extensions/VisualEditor: SWAT (duration: 00m 12s) [production]
23:52 <catrope> Synchronized php-1.26wmf1/extensions/CentralAuth: SWAT (duration: 00m 12s) [production]
23:51 <catrope> Synchronized php-1.26wmf1/extensions/Flow: SWAT (duration: 00m 13s) [production]
23:51 <catrope> Synchronized php-1.26wmf1/extensions/WikiEditor: SWAT (duration: 00m 11s) [production]
22:06 <rmoen> Synchronized php-1.26wmf1/extensions/Gather/: Update gather with cherry picks (duration: 00m 11s) [production]
22:05 <andrewbogott> rebooting all labvirt100x hosts to enable virtualization in the bios [production]
21:16 <rmoen> Synchronized wmf-config/CirrusSearch-production.php: enable cirrus search eventlogging in production (duration: 00m 13s) [production]
20:43 <legoktm> starting SULF on beta cluster [releng]
20:42 <marktraceur> stopping all beta jobs, aborting running (and stuck) beta DB update, kicking bastion, to try and get beta to update [releng]
20:24 <ottomata> powercycled analytics1020 [production]