|
2011-06-29
§
|
| 20:23 |
<RobH> |
working on db1033-db1048 (setting up hardware raid) |
[production] |
| 20:10 |
<RobH> |
db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting) |
[production] |
| 19:54 |
<RobH> |
os install on db1017-db1031 done, running puppet updates |
[production] |
| 19:45 |
<mark> |
Started rsync of ms5 commons thumb dir '0' to ms1004 |
[production] |
| 19:36 |
<mark> |
Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally |
[production] |
| 19:35 |
<RobH> |
db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite |
[production] |
| 19:23 |
<RobH> |
rob is working installs on db1017-db1032 |
[production] |
| 19:08 |
<RobH> |
updated netboot.cfg for private1-b-eqiad broadcast |
[production] |
| 18:46 |
<RobH> |
raid10 setup on db1017-db1027 |
[production] |
| 18:30 |
<catrope> |
synchronizing Wikimedia installation... Revision: 91090: |
[production] |
| 18:27 |
<RoanKattouw> |
Deploying r91089 (jQuery in the head and position=top support) with scap |
[production] |
| 18:11 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on testwiki' |
[production] |
| 17:58 |
<mark> |
OS-installed ms1004 |
[production] |
| 17:33 |
<RobH> |
os install running on db1017-db1021 |
[production] |
| 17:22 |
<mark> |
Deployed DNS entries for ms1001-1004 |
[production] |
| 17:20 |
<RobH> |
db1015 and db1016 os installed, puppet updated, ready to go into replication service |
[production] |
| 17:20 |
<RobH> |
corrected dns, pushed db1017 and upwards into eqiad-b-private ip range |
[production] |
| 17:18 |
<mark> |
Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action |
[production] |
| 16:43 |
<RobH> |
updating dns for the all eqiad dbs |
[production] |
| 16:36 |
<demon> |
synchronized php-1.17/wmf-config/CommonSettings.php 'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo' |
[production] |
| 15:49 |
<RobH> |
starting install on db1015-db1018 |
[production] |
| 15:45 |
<RobH> |
db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service |
[production] |
| 14:49 |
<RobH> |
installing db1011, db1012, db1013, & db1014 |
[production] |
| 14:40 |
<RobH> |
db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service |
[production] |
| 14:32 |
<RobH> |
stat1.wikimedia.org server online for Erik Z |
[production] |
| 13:46 |
<RobH> |
db1007, db1008, db1009, db1010 OS installed, needs puppet run |
[production] |
| 13:42 |
<RobH> |
starting os install on stat1, db1010 and db1009. mislogged before, was finishing install on db1007 and db1008 |
[production] |
| 13:39 |
<RobH> |
updated dns with stat1 server info |
[production] |
| 13:31 |
<RobH> |
db1008 & db1009 finishing OS install |
[production] |
| 01:50 |
<RobH> |
db1005 & db1006 os installed, puppet updated, ready to go into replication service |
[production] |
| 00:42 |
<demon> |
synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes |
[production] |
|
2011-06-28
§
|
| 23:29 |
<RobH> |
db1003 and db1004 os installed, puppet updated, ready to be put into replication service |
[production] |
| 22:46 |
<RobH> |
db1001 installed and updated with puppet, ready to be put into replication service |
[production] |
| 22:45 |
<RobH> |
db1002 installed and updated with puppet, ready to be put into replication service |
[production] |
| 22:44 |
<RobH> |
db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs |
[production] |
| 22:24 |
<RobH> |
carbon host online as install tftp host in eqiad. |
[production] |
| 22:10 |
<RobH> |
syncing docroot and gracefully restarting apaches to take the update |
[production] |
| 22:01 |
<RobH> |
carbon rebooting |
[production] |
| 21:43 |
<RobH> |
testing tftp function of carbon from cp1001 as its on the same public vlan network |
[production] |
| 21:13 |
<RobH> |
updated puppet to point tftp hosts in eqiad to carbon |
[production] |
| 21:13 |
<RobH> |
rsyncing over the tftpboot directory from brewster to carbon |
[production] |
| 21:00 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. Change requested by jamesofur on irc' |
[production] |
| 20:51 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. This time without breaking the world' |
[production] |
| 20:33 |
<RobH> |
reinstalling carbon |
[production] |
| 20:27 |
<RobH> |
updated misc::install-server and brewster/carbon entires in puppet, install server roles are now called on a service by service bais. brewster also needs reboot for kernel update, so doing that now as well. |
[production] |
| 17:47 |
<RobH> |
carbon host powered down, the misc::install-server class in puppet needs to be broken out into more specialized roles, carbon reinstalled and updated to just host the tftp roles of the install server |
[production] |
| 17:36 |
<RobH> |
new install host carbon os installed, running puppet updates |
[production] |
| 17:11 |
<RobH> |
copied db41 sudoers file to db40 since it was overwritten by old raid utils package. now maybe the log spam will stop |
[production] |
| 15:20 |
<RoanKattouw> |
NTP has synced clock in prototype, it's got the correct time now |
[production] |
| 15:18 |
<RoanKattouw> |
Installed ntp on prototype, its clock was 10 minutes behind. It seems to be slowly catching up now |
[production] |