|
2009-07-02
§
|
| 21:38 |
<Rob> |
sq24 wont accept ssh, depooling. |
[production] |
| 21:34 |
<Rob> |
rebooting sq21 |
[production] |
| 21:26 |
<Rob> |
ran changes to push dns back to normal scenario |
[production] |
| 19:52 |
<mark> |
Power outage at esams, moving traffic |
[production] |
| 19:44 |
<Andrew> |
Knams down, Rob is looking into it |
[production] |
| 19:41 |
<Andrew> |
Reports of problems from Europe |
[production] |
| 19:25 |
<Andrew> |
running sync-common-all to deploy mobileRedirect.php to fix hcatlin's mobile redirect/cookie bug |
[production] |
| 19:22 |
<andrew> |
synchronized live-1.5/mobileRedirect.php |
[production] |
| 17:15 |
<mark> |
Rebooted srv159 |
[production] |
| 16:13 |
<Fred> |
shutting 217 back down as it is not supposed to be up due to faulty timer causing issues. |
[production] |
| 16:12 |
<Fred> |
rebooted srv217. Was unpingable. |
[production] |
| 14:09 |
<Andrew> |
Started sending updates of spam.log to Project Honeypot folks every 5 minutes, in my crontab on hume. |
[production] |
| 11:20 |
<andrew> |
synchronized php-1.5/skins/common/shared.css 'Live-merging r52669, r52684 at rainman's request, search fixes.' |
[production] |
| 11:18 |
<andrew> |
synchronized php-1.5/includes/specials/SpecialSearch.php 'Live-merging r52669, r52684 at rainman's request, search fixes.' |
[production] |
| 00:03 |
<brion> |
synchronized php-1.5/CommonSettings.php |
[production] |
| 00:02 |
<brion> |
synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'de-merge broken r52664' |
[production] |
|
2009-07-01
§
|
| 23:40 |
<brion> |
poking in tweaks to search and updates to vector |
[production] |
| 23:22 |
<brion> |
synchronized php-1.5/CommonSettings.php 'bump wgStyleVersion' |
[production] |
| 23:21 |
<brion> |
synchronized php-1.5/skins/vector/main-rtl.css |
[production] |
| 23:21 |
<brion> |
synchronized php-1.5/skins/vector/main-ltr.css |
[production] |
| 23:10 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'set vector skin, new toolbar on for usability wiki' |
[production] |
| 23:07 |
<mark> |
Kicked pascal |
[production] |
| 23:05 |
<brion> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.php 'bumping the js ver no' |
[production] |
| 23:01 |
<brion> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js |
[production] |
| 22:59 |
<brion> |
synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'to 52659' |
[production] |
| 22:57 |
<brion> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.i18n.php |
[production] |
| 22:45 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'enabling UsabilityInitiative (for optional EditToolbar)' |
[production] |
| 22:43 |
<brion> |
synchronized php-1.5/CommonSettings.php 'disabling EditWarning pending addl talk' |
[production] |
| 22:40 |
<brion-codereview> |
updating UsabilityInitiative ext to r52657 in prep for enabling new toolbar option |
[production] |
| 22:10 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'Enabling new search UI formatting sitewide' |
[production] |
| 22:02 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'fixing the RTL disable for vector' |
[production] |
| 21:58 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'Vector should now be available in prefs for non-RTL sites' |
[production] |
| 21:57 |
<brion> |
synchronized php-1.5/CommonSettings.php 'vector config tweak' |
[production] |
| 21:42 |
<brion-codereview> |
updating Vector to current |
[production] |
| 19:38 |
<midom> |
synchronized php-1.5/db.php |
[production] |
| 16:13 |
<Fred> |
bayes is running out of memory on a regular basis. Enabled process accounting / sar to gather more data. |
[production] |
| 15:48 |
<Fred> |
rebooting Bayes as it locked up again. |
[production] |
| 11:48 |
<tstarling> |
synchronized php-1.5/InitialiseSettings.php 'trying a lower value for $wgMaxMsgCacheEntrySize' |
[production] |
| 11:19 |
<domas> |
cleaned up srv100 |
[production] |
| 11:18 |
<domas> |
noticed that imagemagick tempfiles are currently created in /u/l/a/c-l/p/ :) |
[production] |
| 09:24 |
<domas> |
pinned mysqlds on half of cores on 8-core boxes: for i in {11..30}; do ssh db$i 'taskset -pc 0-15:2 $(pidof mysqld)' ; done |
[production] |
|
2009-06-30
§
|
| 23:58 |
<Tim> |
killed gearmanWorker.php instances on hume running with the old master conf, re-ran them running as apache |
[production] |
| 23:20 |
<Andrew> |
23:19 <@brion> !log s1 switched to db16, enwiki back online! (thanks tim!) tomasz rebooted wikitech linode which also borked. :P |
[production] |
| 22:58 |
<brion> |
synchronized php-1.5/CommonSettings.php 'put non-enwiki back to read/write' |
[production] |
| 22:57 |
<brion> |
db14 (s1 master) is in some kind of borked state. slaves seem up to date; gonna try a master switch. |
[production] |
| 22:49 |
<brion> |
synchronized php-1.5/CommonSettings.php |
[production] |
| 22:49 |
<brion> |
putting site to read-only |
[production] |
| 22:44 |
<Andrew> |
Rolling back updates, first vector skin, then config changes. If that doesn't help, will roll back UsabilityInitiative changes |
[production] |