|
2010-08-12
§
|
| 13:08 |
<mark> |
Recovered backend squid on knsq11 |
[production] |
| 12:53 |
<mark> |
Reassembling RAID arrays md0 and md1 on knsq11 |
[production] |
| 12:40 |
<mark> |
Running apt-get upgrade && reboot on amssq31 |
[production] |
| 11:17 |
<mark> |
Shutdown knsq1 and knsq11 for swapping drives |
[production] |
| 09:34 |
<catrope> |
synchronized php-1.5/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'r70933' |
[production] |
| 09:08 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24710: Enable $wgVectorShowVariantName on srwiki' |
[production] |
| 06:00 |
<jeluf> |
synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 05:42 |
<JeLuF> |
Bug 24736 - Update wikimania.wikimedia.org to point to wm2011 |
[production] |
| 02:43 |
<RobH> |
dataset1 back online, serving http, any jobs in process are borked (sorry ariel!) |
[production] |
| 02:39 |
<RobH> |
dataset1 unresponsive to physical console, serial console, had to do a hard reset |
[production] |
| 02:05 |
<RobH> |
dataset1 crashed while querying its raid controller about a bad disk, in route to dc to fix. |
[production] |
|
2010-08-11
§
|
| 21:56 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24561 - Quiz on Polish Wikibooks' |
[production] |
| 21:53 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '21375 - new wiki as internal working space for the fiwiki arbcom' |
[production] |
| 21:48 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24486 - Create Appendix namespace on the Luxembourgish Wiktionary' |
[production] |
| 21:46 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24374 - Create new usergroups at commons' |
[production] |
| 21:28 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24460 - Set up transwiki import for lb.wiktionary' |
[production] |
| 21:10 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24626 - Add an "autopatrolled" status for frwiktionary' |
[production] |
| 21:06 |
<RobH> |
db16 will remain offline until replacement parts arrive from Sun. rt#54 |
[production] |
| 21:03 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24570 - Request' |
[production] |
| 20:42 |
<mark> |
Fixed ganglia mess on ms1 |
[production] |
| 20:37 |
<mark> |
Started rsync of ms2:/a to ms1:/a |
[production] |
| 20:32 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24685 - Author, Index, Page namespace for id.wikisource' |
[production] |
| 20:30 |
<mark> |
FLUSH TABLES WITH READ LOCK on ms2 |
[production] |
| 20:27 |
<mark> |
Readded spare /dev/mdak1 to /dev/md1 on ms1. Why do spares go missing all the time... |
[production] |
| 20:26 |
<mark> |
Upgraded ms1 to Lucid, rebooted it |
[production] |
| 20:26 |
<RobH> |
working on db16 |
[production] |
| 20:24 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24719 - Extension' |
[production] |
| 20:07 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 20:07 |
<domas> |
promoted db5 to slave on s4 |
[production] |
| 19:53 |
<mark> |
Upgrading ms1 to Lucid |
[production] |
| 19:44 |
<mark> |
Readded missing spare drive to /dev/md1 on ms1 |
[production] |
| 17:25 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'removed srv95 as it has temp warnings and is going to go away soon.' |
[production] |
| 16:41 |
<RobH_dc> |
pulled network on srv110 and started wipe, byebye |
[production] |
| 16:35 |
<RobH_dc> |
db19 eth1 disconnected per dc tasks |
[production] |
| 16:02 |
<RobH_dc> |
working on ms1 |
[production] |
| 15:22 |
<RobH_dc> |
bad disk replaced in db7, raid is currently rebuilding, system still online. |
[production] |
| 15:10 |
<RobH_dc> |
pulled hdd5 from db7 for replacement |
[production] |
| 15:02 |
<mark> |
Shutdown clematis for decommissioning |
[production] |
| 14:18 |
<RobH> |
knsq11,knsq12,knsq13 are post os reinstall, pre squid deployment config, will finish them in a bit |
[production] |
| 14:07 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 24441 - Enable Rollback in Quechua Wikipedia' |
[production] |
| 13:33 |
<RobH> |
knsq11-knsq13 coming down for reinstallation |
[production] |
| 12:23 |
<Tim> |
deployed non-threaded version of imagemagick on all image scalers |
[production] |
| 11:44 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'OMP_NUM_THREADS=1' |
[production] |
| 11:21 |
<mark> |
Reconfigured wikimedia-lvs-realserver on hume, so wikimedia-task-appserver install succeeds |
[production] |
| 11:19 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'reduced magick memory limit from 100M to 50M to stop hanging with vsize limit 300M' |
[production] |
| 10:46 |
<mark> |
Removed pattern check from nagios check_http |
[production] |
| 09:42 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 09:38 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 09:35 |
<Tim> |
rebooting srv223, went OOM and mostly died |
[production] |
| 09:32 |
<tstarling> |
synchronized php-1.5/includes/media/Bitmap.php 'temporary patch to stop scalers going OOM' |
[production] |