|
2011-08-30
§
|
| 16:31 |
<mutante> |
srv266 - power cycle |
[production] |
| 16:22 |
<mutante> |
amssq38 - power up,dist-upgrade/kernel,squid clean,reboot |
[production] |
| 12:31 |
<Tim> |
on db9 and db10, raised max_connect_errors from 10 to 1 billion so that it will stop blocking kaulen. These servers are not publically accessible, nobody can SYN flood them |
[production] |
| 12:25 |
<Tim> |
ran "flush hosts" on db9 to temporarily fix bugzilla connection error |
[production] |
| 09:22 |
<RoanKattouw> |
Reverted yesterday's changes on srv162 (disabled core dumps and suid_dumpable) |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.17) at Tue Aug 30 02:25:02 UTC 2011 |
[production] |
| 01:16 |
<binasher> |
upgrading varnish on mobile caching servers to 3.0.0-1wmf5 |
[production] |
| 00:19 |
<binasher> |
added new udplog-1.7 pkg to lucid-wikimedia repo, will be upgraded everywhere via puppet on next run |
[production] |
|
2011-08-29
§
|
| 22:01 |
<awjr> |
updating payflowpro_gateway.i18n on payments1-4 - [[rev:95708|r95708]] |
[production] |
| 19:41 |
<catrope> |
synchronizing Wikimedia installation... Revision: 95690: |
[production] |
| 19:41 |
<RoanKattouw> |
Running scap to deploy Narayam changes |
[production] |
| 19:03 |
<neilk> |
synchronized php/includes/Exception.php 'adding these subclasses in an attempt to stop logged errors in prod - [[rev:95683|r95683]]' |
[production] |
| 18:54 |
<RoanKattouw> |
Re-enabling Apache core dumps on srv162, this time with suid_dumpable enabled |
[production] |
| 18:18 |
<Jeff_Green> |
taking payments3 out of production to test a mediawiki config change |
[production] |
| 18:03 |
<neilk> |
synchronizing Wikimedia installation... Revision: 95681: |
[production] |
| 15:36 |
<mutante> |
srv207 -stop NTP/ntpdate dobson.wm/start NTP (fixes Nagios CRIT), start apache | sq35 -start squid |
[production] |
| 15:15 |
<mutante> |
srv278, srv281 - started apache |
[production] |
| 15:09 |
<mutante> |
srv207 - was unusable due to overload/freeze - powercycle, dist-upgrade/kernel, puppet run, reboot (log entry from July 31st (RAID issues) not confirmed) |
[production] |
| 14:57 |
<mutante> |
srv266 started apache |
[production] |
| 14:51 |
<mutante> |
srv281 - power up, dist-upgrade/kernel, puppet run, reboot (note: see 'srv281' in comments of RT#22 and Server_admin_log) |
[production] |
| 14:39 |
<mutante> |
srv278 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:28 |
<mutante> |
srv266 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:08 |
<mutante> |
srv217 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:06 |
<mutante> |
nagios-wm - ok, just needed restart to talk again |
[production] |
| 13:54 |
<mutante> |
srv188 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 13:45 |
<mutante> |
nagios-wm is on channel but does not speak!? (not ignoring it) |
[production] |
| 13:45 |
<mutante> |
srv174 - confirmed hardware failure, new RT#1379, acked in Nagios |
[production] |
| 13:29 |
<mutante> |
srv156 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 12:57 |
<RoanKattouw> |
Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later |
[production] |
| 09:26 |
<RoanKattouw> |
... on srv162 |
[production] |
| 09:26 |
<RoanKattouw> |
Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does |
[production] |
| 09:23 |
<RoanKattouw> |
Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf |
[production] |
| 09:03 |
<RoanKattouw> |
Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings |
[production] |
| 07:39 |
<RoanKattouw> |
Reverted my changes on srv163 and started puppet |
[production] |
| 07:38 |
<RoanKattouw> |
Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2 |
[production] |
| 07:34 |
<RoanKattouw> |
Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons |
[production] |
| 07:32 |
<RoanKattouw> |
Stopped puppet on srv163 to prevent it from reverting my hacks |
[production] |
| 07:26 |
<RoanKattouw> |
Restarting Apache on srv163 so these changes take effect |
[production] |
| 07:26 |
<RoanKattouw> |
Enabled core dumps for Apache on srv163 by editing /etc/default/apache2 |
[production] |
| 07:19 |
<RoanKattouw> |
Changing Apache's cwd on srv163 by editing /etc/apache2/envvars |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.17) at Mon Aug 29 02:20:17 UTC 2011 |
[production] |