|
2011-08-29
§
|
| 19:41 |
<catrope> |
synchronizing Wikimedia installation... Revision: 95690: |
[production] |
| 19:41 |
<RoanKattouw> |
Running scap to deploy Narayam changes |
[production] |
| 19:03 |
<neilk> |
synchronized php/includes/Exception.php 'adding these subclasses in an attempt to stop logged errors in prod - [[rev:95683|r95683]]' |
[production] |
| 18:54 |
<RoanKattouw> |
Re-enabling Apache core dumps on srv162, this time with suid_dumpable enabled |
[production] |
| 18:18 |
<Jeff_Green> |
taking payments3 out of production to test a mediawiki config change |
[production] |
| 18:03 |
<neilk> |
synchronizing Wikimedia installation... Revision: 95681: |
[production] |
| 15:36 |
<mutante> |
srv207 -stop NTP/ntpdate dobson.wm/start NTP (fixes Nagios CRIT), start apache | sq35 -start squid |
[production] |
| 15:15 |
<mutante> |
srv278, srv281 - started apache |
[production] |
| 15:09 |
<mutante> |
srv207 - was unusable due to overload/freeze - powercycle, dist-upgrade/kernel, puppet run, reboot (log entry from July 31st (RAID issues) not confirmed) |
[production] |
| 14:57 |
<mutante> |
srv266 started apache |
[production] |
| 14:51 |
<mutante> |
srv281 - power up, dist-upgrade/kernel, puppet run, reboot (note: see 'srv281' in comments of RT#22 and Server_admin_log) |
[production] |
| 14:39 |
<mutante> |
srv278 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:28 |
<mutante> |
srv266 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:08 |
<mutante> |
srv217 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 14:06 |
<mutante> |
nagios-wm - ok, just needed restart to talk again |
[production] |
| 13:54 |
<mutante> |
srv188 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 13:45 |
<mutante> |
nagios-wm is on channel but does not speak!? (not ignoring it) |
[production] |
| 13:45 |
<mutante> |
srv174 - confirmed hardware failure, new RT#1379, acked in Nagios |
[production] |
| 13:29 |
<mutante> |
srv156 - power up, dist-upgrade/kernel, puppet run, reboot |
[production] |
| 12:57 |
<RoanKattouw> |
Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later |
[production] |
| 09:26 |
<RoanKattouw> |
... on srv162 |
[production] |
| 09:26 |
<RoanKattouw> |
Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does |
[production] |
| 09:23 |
<RoanKattouw> |
Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf |
[production] |
| 09:03 |
<RoanKattouw> |
Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings |
[production] |
| 07:39 |
<RoanKattouw> |
Reverted my changes on srv163 and started puppet |
[production] |
| 07:38 |
<RoanKattouw> |
Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2 |
[production] |
| 07:34 |
<RoanKattouw> |
Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons |
[production] |
| 07:32 |
<RoanKattouw> |
Stopped puppet on srv163 to prevent it from reverting my hacks |
[production] |
| 07:26 |
<RoanKattouw> |
Restarting Apache on srv163 so these changes take effect |
[production] |
| 07:26 |
<RoanKattouw> |
Enabled core dumps for Apache on srv163 by editing /etc/default/apache2 |
[production] |
| 07:19 |
<RoanKattouw> |
Changing Apache's cwd on srv163 by editing /etc/apache2/envvars |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.17) at Mon Aug 29 02:20:17 UTC 2011 |
[production] |
|
2011-08-26
§
|
| 21:06 |
<mutante> |
amssq48 - power back up, clean squid, dist-upgrade |
[production] |
| 16:37 |
<robh> |
updating text-settings to move sq36 into the squid api cluster. puppet updated already for the same, and pybal updated to remove sq36 frontend from normal text service |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.17) at Fri Aug 26 02:29:15 UTC 2011 |
[production] |
| 00:11 |
<robh> |
change reverted, nothing bad, but undesired result. hooper back to normal |
[production] |
| 00:09 |
<robh> |
hooper apache config change for https redirection on etherpad |
[production] |
| 00:09 |
<robh> |
i meant to paste the rt link |
[production] |
| 00:09 |
<robh> |
testing something in hooper apache config, should result in nothing noticeable to users, unless i did it wrong. |
[production] |
| 00:08 |
<maplebed> |
changed puppet client run interval from the default (30m) to 2hrs to reduce load on the master. |
[production] |