| 2012-04-06
      
      § | 
    
  | 22:13 | <LeslieCarr> | deploying new squid config to amssq35 | [production] | 
            
  | 21:55 | <LeslieCarr> | restarted puppet on spence | [production] | 
            
  | 21:35 | <LeslieCarr> | moved jenkins_1.458_all.deb to /srv/wikimedia/incoming/ on brewster | [production] | 
            
  | 21:32 | <LeslieCarr> | restarted squid on brewster | [production] | 
            
  | 18:27 | <Ryan_Lane> | updating OpenStackManager to [[rev:114758|r114758]] on virt0 | [production] | 
            
  | 17:33 | <mark> | Sending Japanese upload traffic to varnish in eqiad | [production] | 
            
  | 17:15 | <mark> | Power cycled down host lvs5 | [production] | 
            
  | 16:43 | <mutante> | changed master and started slave on es1004 | [production] | 
            
  | 15:55 | <mutante> | used gerrit create-project to create operations/debs/wikistats.git | [production] | 
            
  | 14:13 | <mutante> | manganese (gerrit) now sends SSL CA certificate on https, (curl -vvv says verify ok), should resolve [[RT:2777]] and [[BZ:35709]] | [production] | 
            
  | 11:51 | <mutante> | es1004 - rsync was finished, deleted all binlogs from old host, mysqld_safe& , but did not "change master.." and "start slave" (see mail) | [production] | 
            
  | 11:39 | <notpeter> | restarting lsearchd on search3... again... | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.19) at Fri Apr  6 02:17:37 UTC 2012 | [production] | 
            
  | 01:21 | <Ryan_Lane> | updating OpenStackManager to [[rev:114757|r114757]] on virt0 | [production] | 
            
  | 00:18 | <Ryan_Lane> | updating OpenStackManager to [[rev:114754|r114754]] on virt0 | [production] | 
            
  
    | 2012-04-05
      
      § | 
    
  | 23:49 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Change guwikisource logo to point to the unscaled file instead' | [production] | 
            
  | 21:46 | <notpeter> | halting db15 for it to await decom | [production] | 
            
  | 21:39 | <binasher> | started enwiki.revision sha1 migration on db12 | [production] | 
            
  | 21:32 | <notpeter> | restarting lsearchd on search18 | [production] | 
            
  | 21:22 | <asher> | synchronized wmf-config/db.php  'pulling db12, moving enwiki watchlist,recentchange,etc to db53' | [production] | 
            
  | 21:19 | <asher> | synchronized wmf-config/db.php  'returning db53' | [production] | 
            
  | 21:17 | <py> | synchronized wmf-config/lucene.php  'pushing all search traffic back to pmtpa' | [production] | 
            
  | 18:34 | <Ryan_Lane> | updating OpenStackManager to [[rev:114746|r114746]] on virt0 | [production] | 
            
  | 18:19 | <Ryan_Lane> | updating OpenStackManager to [[rev:114744|r114744]] on virt0 | [production] | 
            
  | 16:49 | <RobH> | brewster puppet running again, cisco installs wont work again until i finish puppetizing the files later today | [production] | 
            
  | 15:41 | <py> | synchronized wmf-config/lucene.php  'pushing search pool4 to eqiad. this is the smaller wikis shard' | [production] | 
            
  | 15:40 | <notpeter> | pointing search pool4 to eqiad (this is the "smaller languages" shard) | [production] | 
            
  | 15:14 | <Rob_H> | puppet daemon being halted on brewster, i need to make local test changes to dhcp | [production] | 
            
  | 14:52 | <py> | synchronized wmf-config/lucene.php  'pushing search prefix pool live in eqiad' | [production] | 
            
  | 14:51 | <notpeter> | pushing search prefix pool live in eqiad | [production] | 
            
  | 14:51 | <mutante> | gallium - disabled incompatible GitTool plugin on jenkins and restarted it | [production] | 
            
  | 14:34 | <mutante> | importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium | [production] | 
            
  | 14:08 | <apergos> | started rsync in screen session as root on es1003 copying snapshot from es1001 to /a/ | [production] | 
            
  | 14:04 | <andrewbogott> | created labs account for cneubauer | [production] | 
            
  | 14:02 | <py> | synchronized wmf-config/lucene.php  'pointing enwiki search and enwiki.prefix at eqiad' | [production] | 
            
  | 14:00 | <notpeter> | pointing enwiki and enwiki.prefix at eqiad search cluster | [production] | 
            
  | 13:48 | <mutante> | gallium - upgraded all pear packages | [production] | 
            
  | 13:45 | <mutante> | gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated) | [production] | 
            
  | 13:43 | <mutante> | gallium - upgrading pear | [production] | 
            
  | 13:33 | <mutante> | installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs | [production] | 
            
  | 13:24 | <py> | synchronized wmf-config/lucene.php  'pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad' | [production] | 
            
  | 13:21 | <notpeter> | pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad | [production] | 
            
  | 12:27 | <notpeter> | search1 and search4 seem to be dead. restarting lsearchd | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.19) at Thu Apr  5 02:16:52 UTC 2012 | [production] | 
            
  | 00:33 | <Ryan_Lane> | updating OpenStackManager to [[rev:114730|r114730]] on virt0 | [production] | 
            
  | 00:24 | <Ryan_Lane> | updating OpenStackManager to [[rev:114729|r114729]] on virt0 | [production] | 
            
  | 00:19 | <Ryan_Lane> | updating OpenStackManager to [[rev:114728|r114728]] on virt0 | [production] | 
            
  | 00:12 | <Ryan_Lane> | updating OpenStackManager to [[rev:114726|r114726]] on virt0 | [production] | 
            
  | 00:00 | <Ryan_Lane> | updating OpenStackManager to [[rev:114724|r114724]] on virt0 | [production] |