9001-9050 of 10000 results (30ms)
2013-11-19 §
11:56 <apergos> powercycled ms-be1002, couldn't get in (lots of BUG: soft lockup - CPU#18 stuck for 22s etc on console) [production]
11:25 <akosiaris> remove puppetmaster and apache packages from sockpuppet. It no longer is a puppetmaster [production]
11:18 <akosiaris> remove puppetmaster and apache packages from stafford. It no longer is a puppetmaster [production]
05:00 <Tim> updating DNS to I3adffd88 [production]
02:50 <LocalisationUpdate> ResourceLoader cache refresh completed at Tue Nov 19 02:50:34 UTC 2013 [production]
02:19 <LocalisationUpdate> completed (1.23wmf4) at Tue Nov 19 02:19:11 UTC 2013 [production]
02:12 <springle> synchronized wmf-config/db-pmtpa.php [production]
02:10 <LocalisationUpdate> completed (1.23wmf3) at Tue Nov 19 02:10:34 UTC 2013 [production]
02:06 <springle> synchronized wmf-config/db-pmtpa.php [production]
2013-11-18 §
21:04 <gwicke> deployed Parsoid fd3d6dc [production]
20:39 <RobH> disabled a bunch of rt users who no longer login and dont work here anymore (so if you do work here and i messed up, opps ;) [production]
19:57 <demon> synchronized wmf-config/InitialiseSettings.php 'Disabling Cirrus on all wikis but test2wiki' [production]
19:36 <mwalker> updating civi from 326854db84becacf808add24a75ca27d1358c5bc for new thank you templates [production]
19:31 <mutante> pybal - eqiad/search: disabling testsearch100[13], adding elastic10[12] [production]
19:24 <reedy> synchronized wmf-config/CommonSettings.php 'I0921b9e853fdacd554de40f138644d6187ae7efe' [production]
17:36 <demon> synchronized wmf-config/CirrusSearch-common.php 'If920e0e9' [production]
17:31 <demon> synchronized wmf-config/CirrusSearch-common.php 'I593f62a5' [production]
17:31 <demon> synchronized wmf-config/InitialiseSettings.php 'I593f62a5' [production]
17:25 <demon> synchronized wmf-config/InitialiseSettings.php 'I114677d0' [production]
17:24 <demon> synchronized wmf-config/CirrusSearch-common.php 'I114677d0' [production]
17:23 <demon> updated /a/common to {{Gerrit|I3b5eac7fb}}: depool db74 for move to S6 [production]
16:07 <hashar> gallium: running puppet to apply ferm firewall configuration from {{gerrit|95162}} [production]
10:00 <springle> xtrabackup db50 to db74 [production]
09:46 <springle> synchronized wmf-config/db-pmtpa.php 'depool db74 for move to S6' [production]
06:28 <apergos> powercycled ms-be9, it was unresponsive; doing xfs_repair on /dev/sde1 after 'corruption detected' messages on reboot [production]
03:20 <paravoid> adding mw1045 mw1070 mw1085 mw1165 back to the apache pool; no evidence (RT tickets, pybal comments) as why they are not pooled [production]
02:49 <springle> for index cardinality on nullable fields, testing innodb_stats_method=nulls_ignored on db1049, db1002, db1003, db1004, db1026, db1040, db1041 + reanalyzing tag_summary and change_tag [production]
02:38 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Nov 18 02:37:59 UTC 2013 [production]
02:36 <springle> synchronized wmf-config/db-eqiad.php 'correct s3 groupLoadsBySection key' [production]
02:23 <springle> synchronized wmf-config/db-pmtpa.php 'pmtpa replication reconfig for decomissioning' [production]
02:15 <LocalisationUpdate> completed (1.23wmf4) at Mon Nov 18 02:15:51 UTC 2013 [production]
02:08 <LocalisationUpdate> completed (1.23wmf3) at Mon Nov 18 02:07:57 UTC 2013 [production]
2013-11-17 §
19:13 <Nemo_bis> 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org [production]
19:11 <Nemo_bis> The 5xx errors mega-spikes every ten minutes are back [production]
02:08 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013 [production]
02:03 <LocalisationUpdate> completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013 [production]
02:02 <LocalisationUpdate> completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013 [production]
01:35 <Reedy> Killed updateSpecialPages and related processes on terbium [production]
01:19 <MaxSem> Killed a few long queries on db1007 [production]
01:08 <MaxSem> db1007 is having tough times due to special page updates [production]
2013-11-16 §
02:33 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Nov 16 02:33:35 UTC 2013 [production]
02:12 <LocalisationUpdate> completed (1.23wmf4) at Sat Nov 16 02:12:51 UTC 2013 [production]
02:07 <LocalisationUpdate> completed (1.23wmf3) at Sat Nov 16 02:07:15 UTC 2013 [production]
2013-11-15 §
23:13 <ori> synchronized wmf-config/CommonSettings.php 'Ia9e64253d: Configure tungsten as $wgUDPProfilerHost' [production]
23:11 <ori> updated /a/common to {{Gerrit|Ia9e64253d}}: Configure tungsten as wgUDPProfilerHost [production]
21:05 <mutante> install unzip on tin (needs puppetization) [production]
19:06 <awight> crm updated from 8d9240fea273faf434881a16d894ef0ad6c616c0 to 326854db84becacf808add24a75ca27d1358c5bc [production]
18:30 <ori> updated /a/common to {{Gerrit|I132d97598}}: Beta Labs: enable UDP profiler [production]
17:58 <mutante> uploaded mw1.22.0rc2 from hexmode http://dumps.wikimedia.org/mediawiki/1.22/ [production]
15:02 <akosiaris> update cr{1,2}-eqiad ACLs with new puppetmaster frontend address [production]