| 2011-06-29
      
      § | 
    
  | 23:38 | <awjr> | running scap to deploy CentralNotice changes (r91118) | [production] | 
            
  | 23:25 | <awjrichards> | synchronized php-1.17/extensions/CentralNotice/centralnotice.css  'r91118' | [production] | 
            
  | 23:24 | <awjrichards> | synchronized php-1.17/extensions/CentralNotice/centralnotice.js  'r91118' | [production] | 
            
  | 23:03 | <RobH> | db1001-db1048 (minus a few hosts mentioned in rt#1105) are installed, puppetized, and ready to be placed into replication service. | [production] | 
            
  | 22:53 | <awjrichards> | synchronized php-1.17/wmf-config/CommonSettings.php  'Making CentralNotice config changes for test wiki (page path and CentralDBname)' | [production] | 
            
  | 21:03 | <RobH> | OS install running on db1033-db1048 | [production] | 
            
  | 20:53 | <RobH> | db1033-db1048 raid setup, ready for OS installs | [production] | 
            
  | 20:44 | <mark> | Started rsync of ms5 commons thumb dir '1' to ms1004 | [production] | 
            
  | 20:23 | <RobH> | working on db1033-db1048 (setting up hardware raid) | [production] | 
            
  | 20:10 | <RobH> | db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting) | [production] | 
            
  | 19:54 | <RobH> | os install on db1017-db1031 done, running puppet updates | [production] | 
            
  | 19:45 | <mark> | Started rsync of ms5 commons thumb dir '0' to ms1004 | [production] | 
            
  | 19:36 | <mark> | Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally | [production] | 
            
  | 19:35 | <RobH> | db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite | [production] | 
            
  | 19:23 | <RobH> | rob is working installs on db1017-db1032 | [production] | 
            
  | 19:08 | <RobH> | updated netboot.cfg for private1-b-eqiad broadcast | [production] | 
            
  | 18:46 | <RobH> | raid10 setup on db1017-db1027 | [production] | 
            
  | 18:30 | <catrope> | synchronizing Wikimedia installation... Revision: 91090: | [production] | 
            
  | 18:27 | <RoanKattouw> | Deploying r91089 (jQuery in the head and position=top support) with scap | [production] | 
            
  | 18:11 | <catrope> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Enable WikiLove on testwiki' | [production] | 
            
  | 17:58 | <mark> | OS-installed ms1004 | [production] | 
            
  | 17:33 | <RobH> | os install running on db1017-db1021 | [production] | 
            
  | 17:22 | <mark> | Deployed DNS entries for ms1001-1004 | [production] | 
            
  | 17:20 | <RobH> | db1015 and db1016 os installed, puppet updated, ready to go into replication service | [production] | 
            
  | 17:20 | <RobH> | corrected dns, pushed db1017 and upwards into eqiad-b-private ip range | [production] | 
            
  | 17:18 | <mark> | Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action | [production] | 
            
  | 16:43 | <RobH> | updating dns for the all eqiad dbs | [production] | 
            
  | 16:36 | <demon> | synchronized php-1.17/wmf-config/CommonSettings.php  'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo' | [production] | 
            
  | 15:49 | <RobH> | starting install on db1015-db1018 | [production] | 
            
  | 15:45 | <RobH> | db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service | [production] | 
            
  | 14:49 | <RobH> | installing db1011, db1012, db1013, & db1014 | [production] | 
            
  | 14:40 | <RobH> | db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service | [production] | 
            
  | 14:32 | <RobH> | stat1.wikimedia.org server online for Erik Z | [production] | 
            
  | 13:46 | <RobH> | db1007, db1008, db1009, db1010 OS installed, needs puppet run | [production] | 
            
  | 13:42 | <RobH> | starting os install on stat1, db1010 and db1009.  mislogged before, was finishing install on db1007 and db1008 | [production] | 
            
  | 13:39 | <RobH> | updated dns with stat1 server info | [production] | 
            
  | 13:31 | <RobH> | db1008 & db1009 finishing OS install | [production] | 
            
  | 01:50 | <RobH> | db1005 & db1006 os installed, puppet updated, ready to go into replication service | [production] | 
            
  | 00:42 | <demon> | synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes | [production] | 
            
  
    | 2011-06-28
      
      § | 
    
  | 23:29 | <RobH> | db1003 and db1004 os installed, puppet updated, ready to be put into replication service | [production] | 
            
  | 22:46 | <RobH> | db1001 installed and updated with puppet, ready to be put into replication service | [production] | 
            
  | 22:45 | <RobH> | db1002 installed and updated with puppet, ready to be put into replication service | [production] | 
            
  | 22:44 | <RobH> | db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs | [production] | 
            
  | 22:24 | <RobH> | carbon host online as install tftp host in eqiad. | [production] | 
            
  | 22:10 | <RobH> | syncing docroot and gracefully restarting apaches to take the update | [production] | 
            
  | 22:01 | <RobH> | carbon rebooting | [production] | 
            
  | 21:43 | <RobH> | testing tftp function of carbon from cp1001 as its on the same public vlan network | [production] | 
            
  | 21:13 | <RobH> | updated puppet to point tftp hosts in eqiad to carbon | [production] | 
            
  | 21:13 | <RobH> | rsyncing over the tftpboot directory from brewster to carbon | [production] | 
            
  | 21:00 | <pdhanda> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Bug 28781. Change requested by jamesofur on irc' | [production] |