|
2011-06-29
§
|
| 17:20 |
<RobH> |
corrected dns, pushed db1017 and upwards into eqiad-b-private ip range |
[production] |
| 17:18 |
<mark> |
Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action |
[production] |
| 16:43 |
<RobH> |
updating dns for the all eqiad dbs |
[production] |
| 16:36 |
<demon> |
synchronized php-1.17/wmf-config/CommonSettings.php 'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo' |
[production] |
| 15:49 |
<RobH> |
starting install on db1015-db1018 |
[production] |
| 15:45 |
<RobH> |
db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service |
[production] |
| 14:49 |
<RobH> |
installing db1011, db1012, db1013, & db1014 |
[production] |
| 14:40 |
<RobH> |
db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service |
[production] |
| 14:32 |
<RobH> |
stat1.wikimedia.org server online for Erik Z |
[production] |
| 13:46 |
<RobH> |
db1007, db1008, db1009, db1010 OS installed, needs puppet run |
[production] |
| 13:42 |
<RobH> |
starting os install on stat1, db1010 and db1009. mislogged before, was finishing install on db1007 and db1008 |
[production] |
| 13:39 |
<RobH> |
updated dns with stat1 server info |
[production] |
| 13:31 |
<RobH> |
db1008 & db1009 finishing OS install |
[production] |
| 01:50 |
<RobH> |
db1005 & db1006 os installed, puppet updated, ready to go into replication service |
[production] |
| 00:42 |
<demon> |
synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes |
[production] |
|
2011-06-28
§
|
| 23:29 |
<RobH> |
db1003 and db1004 os installed, puppet updated, ready to be put into replication service |
[production] |
| 22:46 |
<RobH> |
db1001 installed and updated with puppet, ready to be put into replication service |
[production] |
| 22:45 |
<RobH> |
db1002 installed and updated with puppet, ready to be put into replication service |
[production] |
| 22:44 |
<RobH> |
db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs |
[production] |
| 22:24 |
<RobH> |
carbon host online as install tftp host in eqiad. |
[production] |
| 22:10 |
<RobH> |
syncing docroot and gracefully restarting apaches to take the update |
[production] |
| 22:01 |
<RobH> |
carbon rebooting |
[production] |
| 21:43 |
<RobH> |
testing tftp function of carbon from cp1001 as its on the same public vlan network |
[production] |
| 21:13 |
<RobH> |
updated puppet to point tftp hosts in eqiad to carbon |
[production] |
| 21:13 |
<RobH> |
rsyncing over the tftpboot directory from brewster to carbon |
[production] |
| 21:00 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. Change requested by jamesofur on irc' |
[production] |
| 20:51 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. This time without breaking the world' |
[production] |
| 20:33 |
<RobH> |
reinstalling carbon |
[production] |
| 20:27 |
<RobH> |
updated misc::install-server and brewster/carbon entires in puppet, install server roles are now called on a service by service bais. brewster also needs reboot for kernel update, so doing that now as well. |
[production] |
| 17:47 |
<RobH> |
carbon host powered down, the misc::install-server class in puppet needs to be broken out into more specialized roles, carbon reinstalled and updated to just host the tftp roles of the install server |
[production] |
| 17:36 |
<RobH> |
new install host carbon os installed, running puppet updates |
[production] |
| 17:11 |
<RobH> |
copied db41 sudoers file to db40 since it was overwritten by old raid utils package. now maybe the log spam will stop |
[production] |
| 15:20 |
<RoanKattouw> |
NTP has synced clock in prototype, it's got the correct time now |
[production] |
| 15:18 |
<RoanKattouw> |
Installed ntp on prototype, its clock was 10 minutes behind. It seems to be slowly catching up now |
[production] |
| 14:58 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling while my eyes are on other stuff' |
[production] |
| 14:52 |
<RobH> |
stat1 racked and remote accessible. has public vlan setup, needs IP allocated and os installed. (also needs to be added to racktables) |
[production] |
| 13:59 |
<mark> |
Partly implemented European routing policy on AS14907 as well, and tuned a bit for routing southern routes via Tampa |
[production] |
| 13:27 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 13:20 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round' |
[production] |
| 13:19 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round' |
[production] |
| 13:13 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round ended' |
[production] |
| 13:12 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round' |
[production] |
| 11:36 |
<kudu> |
testing |
[production] |
| 11:35 |
<Nemo_bis> |
!Wikipedia and other !Wikimedia sites problems at 6:15 and 9:30 UTC due to database overload fixed in 30 min, config now tweaked. |
[production] |
| 10:41 |
<mark> |
Set 'send-community' on the iBGP peer groups on csw1-sdtpa and csw5-pmtpa |
[production] |
| 10:41 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 10:39 |
<domas> |
reenabled full mysql pcache, watching with half of my right eye |
[production] |
| 10:39 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 10:38 |
<domas> |
reduced max_connections on db40 to 1000, enabled "connection recycling" (disconnects longest idling connections when max_connections gets reached), set wait_timeout=2 |
[production] |
| 10:30 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix' |
[production] |