|
2012-05-28
§
|
| 14:12 |
<Reedy> |
Copying php-1.20wmf4 from /tmp to /h/w/c on Fenari |
[production] |
| 13:12 |
<apergos> |
doing security updates for a batch of mws in eqiad |
[production] |
| 11:22 |
<apergos> |
updated kernel etc on mw1133, reboot |
[production] |
| 11:08 |
<apergos> |
powercycled mw1133 |
[production] |
| 09:05 |
<apergos> |
rebooted snapshot4, 3 for security updates |
[production] |
| 08:43 |
<apergos> |
rebooted snapshot1002, security updates (will do the same for 1003, 1004 shortly) |
[production] |
| 08:37 |
<apergos> |
rebooted snapshot1001, security updates |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.20wmf3) at Mon May 28 02:24:05 UTC 2012 |
[production] |
|
2012-05-27
§
|
| 22:23 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 37134 - s:cs: site settings' |
[production] |
| 14:45 |
<asher> |
synchronized wmf-config/db.php 'returning db51 to prod as an s4 slave' |
[production] |
| 14:34 |
<maplebed> |
dns and puppet changes for s4 master rotation done. |
[production] |
| 14:18 |
<binasher> |
rebooted db51, reslaved |
[production] |
| 14:09 |
<maplebed> |
new s4 master position post-rotation is master_log_file="db31-bin.000334", master_log_pos=583315125 |
[production] |
| 14:02 |
<ben> |
synchronized wmf-config/db.php 's4 master switch complete; db31 is the new master. turning off read only on s4' |
[production] |
| 13:25 |
<Tim> |
on db31 set global read_only=1 |
[production] |
| 13:17 |
<tstarling> |
synchronized wmf-config/db.php 's4 read-only and taking out db51' |
[production] |
| 04:27 |
<binasher> |
kaulen - temporarily disabled swap and set oom_adj score to 15 for apache |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.20wmf3) at Sun May 27 02:24:13 UTC 2012 |
[production] |
|
2012-05-25
§
|
| 22:57 |
<maplebed> |
powercycled kaulen on the mgmt interface |
[production] |
| 19:09 |
<maplebed> |
disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead. |
[production] |
| 18:57 |
<RobH> |
bugzilla appears back online |
[production] |
| 18:57 |
<RobH> |
kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access |
[production] |
| 18:55 |
<RobH> |
kaulen serial console unresponsive, rebooting |
[production] |
| 17:42 |
<paravoid> |
rebooting gurvin & yvon with new kernel |
[production] |
| 17:25 |
<paravoid> |
resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime |
[production] |
| 15:34 |
<mark> |
Power cycled kaulen |
[production] |
| 15:23 |
<hashar> |
kaulen (bugzilla) unreacheable :-( |
[production] |
| 13:56 |
<RobH> |
palladium disk replaced |
[production] |
| 13:52 |
<cmjohnson1> |
replacing ps2 on mw1017 |
[production] |
| 13:50 |
<RobH> |
palladium has a bad disk, goign to replace it |
[production] |
| 13:37 |
<RobH> |
updating drac on search18, shouldnt cause system reboot. |
[production] |
| 10:42 |
<apergos> |
restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?) |
[production] |