|
2010-08-12
§
|
| 18:53 |
<RobH> |
srv230 coming down for memory testing |
[production] |
| 18:49 |
<RobH> |
set srv230 to false in lvs, need to test memory |
[production] |
| 18:04 |
<RobH> |
reinstalling srv281 |
[production] |
| 17:59 |
<RobH> |
nix that, srv125 was ex-es, leaving those for now. |
[production] |
| 17:58 |
<RobH> |
pulling srv103 & srv125 for wipe (pulling stuff with temp warnings first) |
[production] |
| 17:53 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'removed srv103, replacing it with srv244' |
[production] |
| 17:47 |
<RobH> |
pulling srv95 for wipe |
[production] |
| 17:38 |
<RobH> |
srv110 removed from lvs3 config |
[production] |
| 17:36 |
<mark> |
Removed all apaches up to srv150 from the appserver LVS pool on lvs3 |
[production] |
| 17:21 |
<Fred> |
restarting apache on webservers (220,221,222,224) |
[production] |
| 16:45 |
<RobH> |
wipe running on adler and amane, and they have been removed from puppet and dsh node groups |
[production] |
| 16:12 |
<jeluf> |
synchronized docroot/bits/index.html |
[production] |
| 15:41 |
<mark> |
Setup ports ge-2/0/0 to ge-2/0/20 for search servers on asw-b-sdtpa |
[production] |
| 15:03 |
<mark> |
Shutdown BGP session to AS1257 130.244.6.249 on port 2/5 of br1-knams, preparing for cable move |
[production] |
| 13:08 |
<mark> |
Recovered backend squid on knsq11 |
[production] |
| 12:53 |
<mark> |
Reassembling RAID arrays md0 and md1 on knsq11 |
[production] |
| 12:40 |
<mark> |
Running apt-get upgrade && reboot on amssq31 |
[production] |
| 11:17 |
<mark> |
Shutdown knsq1 and knsq11 for swapping drives |
[production] |
| 09:34 |
<catrope> |
synchronized php-1.5/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'r70933' |
[production] |
| 09:08 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24710: Enable $wgVectorShowVariantName on srwiki' |
[production] |
| 06:00 |
<jeluf> |
synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 05:42 |
<JeLuF> |
Bug 24736 - Update wikimania.wikimedia.org to point to wm2011 |
[production] |
| 02:43 |
<RobH> |
dataset1 back online, serving http, any jobs in process are borked (sorry ariel!) |
[production] |
| 02:39 |
<RobH> |
dataset1 unresponsive to physical console, serial console, had to do a hard reset |
[production] |
| 02:05 |
<RobH> |
dataset1 crashed while querying its raid controller about a bad disk, in route to dc to fix. |
[production] |
|
2010-08-11
§
|
| 21:56 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24561 - Quiz on Polish Wikibooks' |
[production] |
| 21:53 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '21375 - new wiki as internal working space for the fiwiki arbcom' |
[production] |
| 21:48 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24486 - Create Appendix namespace on the Luxembourgish Wiktionary' |
[production] |
| 21:46 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24374 - Create new usergroups at commons' |
[production] |
| 21:28 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24460 - Set up transwiki import for lb.wiktionary' |
[production] |
| 21:10 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24626 - Add an "autopatrolled" status for frwiktionary' |
[production] |
| 21:06 |
<RobH> |
db16 will remain offline until replacement parts arrive from Sun. rt#54 |
[production] |
| 21:03 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24570 - Request' |
[production] |
| 20:42 |
<mark> |
Fixed ganglia mess on ms1 |
[production] |
| 20:37 |
<mark> |
Started rsync of ms2:/a to ms1:/a |
[production] |
| 20:32 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24685 - Author, Index, Page namespace for id.wikisource' |
[production] |
| 20:30 |
<mark> |
FLUSH TABLES WITH READ LOCK on ms2 |
[production] |
| 20:27 |
<mark> |
Readded spare /dev/mdak1 to /dev/md1 on ms1. Why do spares go missing all the time... |
[production] |
| 20:26 |
<mark> |
Upgraded ms1 to Lucid, rebooted it |
[production] |
| 20:26 |
<RobH> |
working on db16 |
[production] |
| 20:24 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24719 - Extension' |
[production] |
| 20:07 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 20:07 |
<domas> |
promoted db5 to slave on s4 |
[production] |
| 19:53 |
<mark> |
Upgrading ms1 to Lucid |
[production] |
| 19:44 |
<mark> |
Readded missing spare drive to /dev/md1 on ms1 |
[production] |
| 17:25 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'removed srv95 as it has temp warnings and is going to go away soon.' |
[production] |
| 16:41 |
<RobH_dc> |
pulled network on srv110 and started wipe, byebye |
[production] |
| 16:35 |
<RobH_dc> |
db19 eth1 disconnected per dc tasks |
[production] |
| 16:02 |
<RobH_dc> |
working on ms1 |
[production] |
| 15:22 |
<RobH_dc> |
bad disk replaced in db7, raid is currently rebuilding, system still online. |
[production] |