3301-3350 of 10000 results (20ms)
2013-11-18 §
17:31 <demon> synchronized wmf-config/CirrusSearch-common.php 'I593f62a5' [production]
17:31 <demon> synchronized wmf-config/InitialiseSettings.php 'I593f62a5' [production]
17:25 <demon> synchronized wmf-config/InitialiseSettings.php 'I114677d0' [production]
17:24 <demon> synchronized wmf-config/CirrusSearch-common.php 'I114677d0' [production]
17:23 <demon> updated /a/common to {{Gerrit|I3b5eac7fb}}: depool db74 for move to S6 [production]
16:07 <hashar> gallium: running puppet to apply ferm firewall configuration from {{gerrit|95162}} [production]
10:00 <springle> xtrabackup db50 to db74 [production]
09:46 <springle> synchronized wmf-config/db-pmtpa.php 'depool db74 for move to S6' [production]
06:28 <apergos> powercycled ms-be9, it was unresponsive; doing xfs_repair on /dev/sde1 after 'corruption detected' messages on reboot [production]
03:20 <paravoid> adding mw1045 mw1070 mw1085 mw1165 back to the apache pool; no evidence (RT tickets, pybal comments) as why they are not pooled [production]
02:49 <springle> for index cardinality on nullable fields, testing innodb_stats_method=nulls_ignored on db1049, db1002, db1003, db1004, db1026, db1040, db1041 + reanalyzing tag_summary and change_tag [production]
02:38 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Nov 18 02:37:59 UTC 2013 [production]
02:36 <springle> synchronized wmf-config/db-eqiad.php 'correct s3 groupLoadsBySection key' [production]
02:23 <springle> synchronized wmf-config/db-pmtpa.php 'pmtpa replication reconfig for decomissioning' [production]
02:15 <LocalisationUpdate> completed (1.23wmf4) at Mon Nov 18 02:15:51 UTC 2013 [production]
02:08 <LocalisationUpdate> completed (1.23wmf3) at Mon Nov 18 02:07:57 UTC 2013 [production]
2013-11-17 §
19:13 <Nemo_bis> 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org [production]
19:11 <Nemo_bis> The 5xx errors mega-spikes every ten minutes are back [production]
02:08 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013 [production]
02:03 <LocalisationUpdate> completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013 [production]
02:02 <LocalisationUpdate> completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013 [production]
01:35 <Reedy> Killed updateSpecialPages and related processes on terbium [production]
01:19 <MaxSem> Killed a few long queries on db1007 [production]
01:08 <MaxSem> db1007 is having tough times due to special page updates [production]
2013-11-16 §
02:33 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Nov 16 02:33:35 UTC 2013 [production]
02:12 <LocalisationUpdate> completed (1.23wmf4) at Sat Nov 16 02:12:51 UTC 2013 [production]
02:07 <LocalisationUpdate> completed (1.23wmf3) at Sat Nov 16 02:07:15 UTC 2013 [production]
2013-11-15 §
23:13 <ori> synchronized wmf-config/CommonSettings.php 'Ia9e64253d: Configure tungsten as $wgUDPProfilerHost' [production]
23:11 <ori> updated /a/common to {{Gerrit|Ia9e64253d}}: Configure tungsten as wgUDPProfilerHost [production]
21:05 <mutante> install unzip on tin (needs puppetization) [production]
19:06 <awight> crm updated from 8d9240fea273faf434881a16d894ef0ad6c616c0 to 326854db84becacf808add24a75ca27d1358c5bc [production]
18:30 <ori> updated /a/common to {{Gerrit|I132d97598}}: Beta Labs: enable UDP profiler [production]
17:58 <mutante> uploaded mw1.22.0rc2 from hexmode http://dumps.wikimedia.org/mediawiki/1.22/ [production]
15:02 <akosiaris> update cr{1,2}-eqiad ACLs with new puppetmaster frontend address [production]
13:55 <akosiaris> switched *eqiad.wmnet to using new puppet infra. triggering puppet run [production]
12:35 <hashar> github: added zeljkofilipin to the new 'qa' team. Granted rights on wikimedia/mediawiki-selenium and wikimedia/qa-browsertests [production]
12:33 <akosiaris> disabled puppet on *eqiad.wmnet for change [production]
12:32 <akosiaris> updated sockpuppet term on cr1-sdtpa , cr2-tmtpa to include palladium [production]
09:45 <hashar> restarting zuul, leaked file descriptors pointing to non existent git pack files :/ [production]
09:15 <akosiaris> disabled commiting in sockpuppet private for evah (hopefully) through a git hook [production]
09:07 <akosiaris2> disabled puppet an all *.pmtpa.wmnet servers to facilitate auditing the change to new puppetmaster infra [production]
07:52 <apergos> sq44 won't come back up, hung after "Remote Access Controller detected", rt #6312 [production]
06:48 <Tim> also fixed mw1141, mw1193 [production]
06:42 <Tim> fixing broken cgroups on mw1038, mw1044, mw1091 [production]
05:15 <ori> updated /a/common to {{Gerrit|I67a6d49d4}}: Beta Labs: set deployment-fluoride as $wgUDPProfilerHost [production]
03:20 <Tim> on mc1001-1016: set timeout=0 in the redis soft state, as for rdb* [production]
03:15 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Nov 15 03:15:35 UTC 2013 [production]
03:14 <Tim> on rdb1001-1004, set timeout=0 in the soft state, to match I8c9b13c1 [production]
02:34 <LocalisationUpdate> completed (1.23wmf4) at Fri Nov 15 02:34:41 UTC 2013 [production]
02:15 <LocalisationUpdate> completed (1.23wmf3) at Fri Nov 15 02:15:18 UTC 2013 [production]