9301-9350 of 10000 results (28ms)
2013-11-19 §
19:22 <reedy> synchronized wmf-config/ 'Enable MassMessage Ib29cb042e739e26fc5ef56f41317454ea690d0cb' [production]
19:08 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: https://gerrit.wikimedia.org/r/96297 [production]
19:07 <reedy> updated /a/common to {{Gerrit|I9b48be4c0}}: try to reduce "no working slave" notice flood in pmtpa dberror.log (attempt 2) [production]
18:53 <Nemo_bis> but now ok [production]
18:48 <Nemo_bis> kaulen overloaded at 100 % CPU since 5 min ago, bugzilla almost unusable [production]
18:32 <ori> synchronized php-1.23wmf4/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers' [production]
18:31 <ori> synchronized php-1.23wmf3/resources/mediawiki/mediawiki.js 'Id2835eca4: Enable module storage for 0.05% of visitors w/storage-capable browsers' [production]
17:35 <LeslieCarr> added palladium as a salt master for common-infrastructure4 firewall [production]
17:34 <LeslieCarr> CV71028 flapped again [production]
17:32 <LeslieCarr> FPL link CV71028 flapped 20 minutes ago [production]
11:56 <apergos> powercycled ms-be1002, couldn't get in (lots of BUG: soft lockup - CPU#18 stuck for 22s etc on console) [production]
11:25 <akosiaris> remove puppetmaster and apache packages from sockpuppet. It no longer is a puppetmaster [production]
11:18 <akosiaris> remove puppetmaster and apache packages from stafford. It no longer is a puppetmaster [production]
05:00 <Tim> updating DNS to I3adffd88 [production]
02:50 <LocalisationUpdate> ResourceLoader cache refresh completed at Tue Nov 19 02:50:34 UTC 2013 [production]
02:19 <LocalisationUpdate> completed (1.23wmf4) at Tue Nov 19 02:19:11 UTC 2013 [production]
02:12 <springle> synchronized wmf-config/db-pmtpa.php [production]
02:10 <LocalisationUpdate> completed (1.23wmf3) at Tue Nov 19 02:10:34 UTC 2013 [production]
02:06 <springle> synchronized wmf-config/db-pmtpa.php [production]
2013-11-18 §
21:04 <gwicke> deployed Parsoid fd3d6dc [production]
20:39 <RobH> disabled a bunch of rt users who no longer login and dont work here anymore (so if you do work here and i messed up, opps ;) [production]
19:57 <demon> synchronized wmf-config/InitialiseSettings.php 'Disabling Cirrus on all wikis but test2wiki' [production]
19:36 <mwalker> updating civi from 326854db84becacf808add24a75ca27d1358c5bc for new thank you templates [production]
19:31 <mutante> pybal - eqiad/search: disabling testsearch100[13], adding elastic10[12] [production]
19:24 <reedy> synchronized wmf-config/CommonSettings.php 'I0921b9e853fdacd554de40f138644d6187ae7efe' [production]
17:36 <demon> synchronized wmf-config/CirrusSearch-common.php 'If920e0e9' [production]
17:31 <demon> synchronized wmf-config/CirrusSearch-common.php 'I593f62a5' [production]
17:31 <demon> synchronized wmf-config/InitialiseSettings.php 'I593f62a5' [production]
17:25 <demon> synchronized wmf-config/InitialiseSettings.php 'I114677d0' [production]
17:24 <demon> synchronized wmf-config/CirrusSearch-common.php 'I114677d0' [production]
17:23 <demon> updated /a/common to {{Gerrit|I3b5eac7fb}}: depool db74 for move to S6 [production]
16:07 <hashar> gallium: running puppet to apply ferm firewall configuration from {{gerrit|95162}} [production]
10:00 <springle> xtrabackup db50 to db74 [production]
09:46 <springle> synchronized wmf-config/db-pmtpa.php 'depool db74 for move to S6' [production]
06:28 <apergos> powercycled ms-be9, it was unresponsive; doing xfs_repair on /dev/sde1 after 'corruption detected' messages on reboot [production]
03:20 <paravoid> adding mw1045 mw1070 mw1085 mw1165 back to the apache pool; no evidence (RT tickets, pybal comments) as why they are not pooled [production]
02:49 <springle> for index cardinality on nullable fields, testing innodb_stats_method=nulls_ignored on db1049, db1002, db1003, db1004, db1026, db1040, db1041 + reanalyzing tag_summary and change_tag [production]
02:38 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Nov 18 02:37:59 UTC 2013 [production]
02:36 <springle> synchronized wmf-config/db-eqiad.php 'correct s3 groupLoadsBySection key' [production]
02:23 <springle> synchronized wmf-config/db-pmtpa.php 'pmtpa replication reconfig for decomissioning' [production]
02:15 <LocalisationUpdate> completed (1.23wmf4) at Mon Nov 18 02:15:51 UTC 2013 [production]
02:08 <LocalisationUpdate> completed (1.23wmf3) at Mon Nov 18 02:07:57 UTC 2013 [production]
2013-11-17 §
19:13 <Nemo_bis> 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org [production]
19:11 <Nemo_bis> The 5xx errors mega-spikes every ten minutes are back [production]
02:08 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013 [production]
02:03 <LocalisationUpdate> completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013 [production]
02:02 <LocalisationUpdate> completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013 [production]
01:35 <Reedy> Killed updateSpecialPages and related processes on terbium [production]
01:19 <MaxSem> Killed a few long queries on db1007 [production]
01:08 <MaxSem> db1007 is having tough times due to special page updates [production]