|
2009-01-27
§
|
| 19:04 |
<brion> |
scapping update to AbuseFilter for test. updated its schema... |
[production] |
| 18:44 |
<brion> |
db16 lagged 2188s |
[production] |
| 18:44 |
<brion> |
restarting slave thread on db16. it got stopped with a lock wait timeout on a page_touched update (wtf?!) |
[production] |
| 18:43 |
<brion> |
slave stopped on db16 |
[production] |
| 17:41 |
<mark> |
knsq1 Up and serving requests with squid 2.7.5 |
[production] |
| 17:25 |
<mark> |
Trying squid 2.7.5 on knsq1 - might be unstable in the mean time |
[production] |
| 17:22 |
<mark> |
Reduced cache_mem on backend esams text squids from 3000 to 2500 |
[production] |
| 16:23 |
<RobH> |
srv76 had a failed hdd, replaced, reinstalled, and bringing back into rotation |
[production] |
| 16:18 |
<RobH> |
srv146 was powered down (heat issue?), powered back up, synced and now in rotation. |
[production] |
| 16:09 |
<RobH> |
srv139 didnt have apache running, synced and started |
[production] |
| 16:01 |
<RobH> |
srv129 didnt have apache running, synced and started |
[production] |
| 15:59 |
<RobH> |
sq11 back online, cleaned |
[production] |
| 15:40 |
<RobH> |
srv126 back online. possible bad disk, if it crashes again, the disk needs replacement. (it went read only before, which seems to sometimes happen even when the disks are not bad.) |
[production] |
| 15:25 |
<RobH> |
srv76 wont boot up, reinstalling. |
[production] |
| 15:12 |
<RobH> |
srv130 coming back online, updated fstab, synced, putting it back in rotation. |
[production] |
| 15:05 |
<RobH> |
moved ts-array4 to its dedicated ports, now its kate's problem ;] |
[production] |
| 14:49 |
<Tim> |
restarted recompressTracked.php |
[production] |
| 14:33 |
<Tim> |
henbane's disk has been full for 8 days due to donate-campaign.log, starting cleanup |
[production] |
| 14:18 |
<Tim> |
killed recompressTracked.php |
[production] |
| 13:44 |
<mark> |
CARP weight redistribution caused large load spike in upload backend request, causing ms1 overload, probably causing issues on apaches via NFS, etc etc... |
[production] |
| 13:29 |
<mark> |
Lowered CARP weight from 10 to 5 for sq1-10.wikimedia.org, from 15 to 10 for sq11-15 |
[production] |
| 06:03 |
<bot broken> |
retrolog |
[production] |
| 02:13 |
<brion> |
updating extensions/AbuseFilter/Views/AbuseFilterViewList.php (mysql 4 compat issue) |
[production] |
| 02:04 |
<brion> |
installed release versions of mwlib on erzurumi and restarted. these should have updated localizations |
[production] |
| 01:48 |
<brion> |
turning AbuseFilter on on test.... having some mysql 4.0 compat issues. poking |
[production] |
| 01:47 |
<brion> |
srv31 seems very sad; slow/borked login? |
[production] |
| 01:39 |
<brion> |
scapping to update AbuseFilter to current |
[production] |
| 01:27 |
<brion> |
prepping testing of AbuseFilter on test.wikipedia |
[production] |
| 00:46 |
<brion> |
enabling Collection also for de.wikisource per frank's req passed on from community |
[production] |
| 00:36 |
<brion> |
adding NS_HELP to $wgCollectionArticleNamespaces |
[production] |
| 00:12 |
<brion> |
Collection extension being enabled on dewiki |
[production] |