| 2013-03-15
      
      § | 
    
  | 23:24 | <laner> | synchronized wmf-config/CommonSettings.php  'Disabling aft5' | [production] | 
            
  | 22:45 | <asher> | synchronized wmf-config/db-eqiad.php  'returning es1007,10 at lower weights' | [production] | 
            
  | 22:35 | <binasher> | rebooted es1007,1010 for kernel upgrades | [production] | 
            
  | 22:33 | <asher> | synchronized wmf-config/db-eqiad.php  'temporarily pulling es1007, es1010' | [production] | 
            
  | 22:23 | <Ryan_Lane> | hid domain dropdown for account creation and login on wikitech | [production] | 
            
  | 22:23 | <Ryan_Lane> | changed wikitech configuration to require email address on account creation | [production] | 
            
  | 22:22 | <Ryan_Lane> | deployed change 54093 to wikitech | [production] | 
            
  | 22:03 | <hashar> | Zuul : opened a bug about this week slowness, breakage:  https://bugzilla.wikimedia.org/show_bug.cgi?id=46176 | [production] | 
            
  | 21:51 | <aaron> | synchronized php-1.21wmf11/includes  'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc' | [production] | 
            
  | 21:36 | <binasher> | db59 -> mariadb 5.5.30 | [production] | 
            
  | 21:29 | <hashar> | restarted Zuul with the safe version ff79197 | [production] | 
            
  | 21:27 | <hashar> | downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events) | [production] | 
            
  | 21:25 | <binasher> | added new build of mariadb 5.5.30 to precise-wikimedia on brewster | [production] | 
            
  | 21:20 | <hashar> | hard restarted Zuul. It was locked down again :/ | [production] | 
            
  | 21:18 | <mutante> | installing package upgrades on gallium | [production] | 
            
  | 21:14 | <hashar> | restarting Zuul, it is locked somehow. Definitely need to update it :( | [production] | 
            
  | 21:09 | <RobH> | professor services seem back up | [production] | 
            
  | 20:51 | <RobH> | professor shutting down for fan swap, short downtime (hopefully) | [production] | 
            
  | 20:47 | <hashar> | integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in {{gerrit|54068}} | [production] | 
            
  | 20:33 | <mutante> | puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513 | [production] | 
            
  | 20:31 | <mutante> | restarting pdns on ns0 | [production] | 
            
  | 20:30 | <mutante> | DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm | [production] | 
            
  | 19:37 | <mutante> | maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx | [production] | 
            
  | 19:22 | <sbernardin> | taking down raskin for decommissioning | [production] | 
            
  | 19:13 | <mutante> | maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443 | [production] | 
            
  | 18:50 | <mutante> | installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...) | [production] | 
            
  | 18:46 | <mutante> | installing package upgrades on formey | [production] | 
            
  | 18:44 | <mutante> | fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades | [production] | 
            
  | 18:33 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ | [production] | 
            
  | 18:19 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ | [production] | 
            
  | 18:19 | <notpeter> | ran puppetd --enable on all mw hosts | [production] | 
            
  | 18:15 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ | [production] | 
            
  | 18:07 | <mlitn> | Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master | [production] | 
            
  | 17:52 | <mlitn> | Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master | [production] | 
            
  | 17:36 | <hashar> | jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org | [production] | 
            
  | 17:22 | <mlitn> | synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/  'Update ArticleFeedbackv5 to master' | [production] | 
            
  | 17:21 | <hashar> | gallium : reloading Zuul so it reports the status url with the wikimedia.org domain | [production] | 
            
  | 17:19 | <hashar> | integration website going down while we deploy a new layout. | [production] | 
            
  | 17:03 | <hashar> | gallium : restarted puppet | [production] | 
            
  | 17:03 | <hashar> | gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and  moved all content from /srv/org/mediawiki/integration  to the new /srv/org/wikimedia/integration | [production] | 
            
  | 16:59 | <hashar> | stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied. | [production] | 
            
  | 05:09 | <Tim> | restarting squid on cp1010 for squid redirector test | [production] | 
            
  | 05:06 | <Tim> | testing new squid redirector | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013 | [production] | 
            
  | 02:16 | <reedy> | synchronized docroot | [production] | 
            
  | 02:13 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 01:53 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 01:52 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 01:52 | <reedy> | gracefulled all apaches | [production] |