3301-3350 of 10000 results (26ms)
2011-08-30 §
19:56 <hashar:> previous sync was for [[rev:2238|r2238]]: removing old commented out parts [production]
19:55 <hashar> synchronized wmf-config/InitialiseSettings.php [production]
19:18 <hashar> synchronized wmf-config/InitialiseSettings.php '30523 - Portal namespace for id.wikisource' [production]
19:10 <hashar> synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia' [production]
19:09 <hashar> synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia' [production]
18:38 <Ryan_Lane> wiping the LDAP database on virt1, re-creating using same domain as svn LDAP. [production]
18:26 <aaron> synchronizing Wikimedia installation... : [production]
17:37 <aaron> synchronizing Wikimedia installation... : [production]
17:34 <aaron> synchronizing Wikimedia installation... : [production]
17:30 <aaron> synchronizing Wikimedia installation... : [production]
17:21 <Ryan_Lane> removed duplicate ip in badbadip list in squid [production]
16:31 <mutante> srv266 - power cycle [production]
16:22 <mutante> amssq38 - power up,dist-upgrade/kernel,squid clean,reboot [production]
12:31 <Tim> on db9 and db10, raised max_connect_errors from 10 to 1 billion so that it will stop blocking kaulen. These servers are not publically accessible, nobody can SYN flood them [production]
12:25 <Tim> ran "flush hosts" on db9 to temporarily fix bugzilla connection error [production]
09:22 <RoanKattouw> Reverted yesterday's changes on srv162 (disabled core dumps and suid_dumpable) [production]
02:22 <LocalisationUpdate> completed (1.17) at Tue Aug 30 02:25:02 UTC 2011 [production]
01:16 <binasher> upgrading varnish on mobile caching servers to 3.0.0-1wmf5 [production]
00:19 <binasher> added new udplog-1.7 pkg to lucid-wikimedia repo, will be upgraded everywhere via puppet on next run [production]
2011-08-29 §
22:01 <awjr> updating payflowpro_gateway.i18n on payments1-4 - [[rev:95708|r95708]] [production]
19:41 <catrope> synchronizing Wikimedia installation... Revision: 95690: [production]
19:41 <RoanKattouw> Running scap to deploy Narayam changes [production]
19:03 <neilk> synchronized php/includes/Exception.php 'adding these subclasses in an attempt to stop logged errors in prod - [[rev:95683|r95683]]' [production]
18:54 <RoanKattouw> Re-enabling Apache core dumps on srv162, this time with suid_dumpable enabled [production]
18:18 <Jeff_Green> taking payments3 out of production to test a mediawiki config change [production]
18:03 <neilk> synchronizing Wikimedia installation... Revision: 95681: [production]
15:36 <mutante> srv207 -stop NTP/ntpdate dobson.wm/start NTP (fixes Nagios CRIT), start apache | sq35 -start squid [production]
15:15 <mutante> srv278, srv281 - started apache [production]
15:09 <mutante> srv207 - was unusable due to overload/freeze - powercycle, dist-upgrade/kernel, puppet run, reboot (log entry from July 31st (RAID issues) not confirmed) [production]
14:57 <mutante> srv266 started apache [production]
14:51 <mutante> srv281 - power up, dist-upgrade/kernel, puppet run, reboot (note: see 'srv281' in comments of RT#22 and Server_admin_log) [production]
14:39 <mutante> srv278 - power up, dist-upgrade/kernel, puppet run, reboot [production]
14:28 <mutante> srv266 - power up, dist-upgrade/kernel, puppet run, reboot [production]
14:08 <mutante> srv217 - power up, dist-upgrade/kernel, puppet run, reboot [production]
14:06 <mutante> nagios-wm - ok, just needed restart to talk again [production]
13:54 <mutante> srv188 - power up, dist-upgrade/kernel, puppet run, reboot [production]
13:45 <mutante> nagios-wm is on channel but does not speak!? (not ignoring it) [production]
13:45 <mutante> srv174 - confirmed hardware failure, new RT#1379, acked in Nagios [production]
13:29 <mutante> srv156 - power up, dist-upgrade/kernel, puppet run, reboot [production]
12:57 <RoanKattouw> Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later [production]
09:26 <RoanKattouw> ... on srv162 [production]
09:26 <RoanKattouw> Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does [production]
09:23 <RoanKattouw> Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf [production]
09:03 <RoanKattouw> Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings [production]
07:39 <RoanKattouw> Reverted my changes on srv163 and started puppet [production]
07:38 <RoanKattouw> Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2 [production]
07:34 <RoanKattouw> Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons [production]
07:32 <RoanKattouw> Stopped puppet on srv163 to prevent it from reverting my hacks [production]
07:26 <RoanKattouw> Restarting Apache on srv163 so these changes take effect [production]
07:26 <RoanKattouw> Enabled core dumps for Apache on srv163 by editing /etc/default/apache2 [production]