| 2011-11-22
      
      § | 
    
  | 23:04 | <Tim> | on kaulen: reverted the firewall rule and createaccount.cgi deny rule from yesterday, replaced the latter by temporarily disabling account creation with BZ "createemailregexp" | [production] | 
            
  | 22:15 | <Ryan_Lane> | started gerrit back up | [production] | 
            
  | 22:10 | <Ryan_Lane> | temporarily shutting down labsconsole | [production] | 
            
  | 22:10 | <Ryan_Lane> | temporarily stopping gerrit | [production] | 
            
  | 21:45 | <catrope> | synchronized php-1.18/resources/mediawiki/mediawiki.util.js  '[[rev:103962|r103962]]' | [production] | 
            
  | 21:33 | <Ryan_Lane> | upgrading the nova database | [production] | 
            
  | 21:31 | <reedy> | synchronizing Wikimedia installation... : Stylize. AaronSchulz made me do it | [production] | 
            
  | 21:29 | <Ryan_Lane> | upgrading packages on virt4 | [production] | 
            
  | 21:28 | <Ryan_Lane> | upgrading packages on virt3 | [production] | 
            
  | 21:26 | <Ryan_Lane> | upgrading packages on virt2 | [production] | 
            
  | 21:25 | <Ryan_Lane> | upgrading packages on virt1 | [production] | 
            
  | 21:17 | <Ryan_Lane> | stopping all nova services | [production] | 
            
  | 21:10 | <reedy> | synchronized wmf-config/abusefilter.php  'Remove lines of sites setting abusefilter-private to anything' | [production] | 
            
  | 20:19 | <catrope> | synchronized php-1.18/includes/Export.php  '[[rev:103954|r103954]]' | [production] | 
            
  | 20:04 | <Ryan_Lane> | upgrading nova from cactus to diablo. during this time period, instances may be occasionally inaccessible, and labsconsole may also be occasionally inaccessible | [production] | 
            
  | 19:45 | <binasher> | increased udp recv buffer on locke to 512M. i think lp-filter needs some work. | [production] | 
            
  | 19:13 | <binasher> | udp2log on emery was broken post upgrade due to files under /var/log/squid being owned by human users. fixed + restarted | [production] | 
            
  | 18:03 | <midom> | synchronized wmf-config/db.php  'readding db19' | [production] | 
            
  | 17:52 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32521 - Request for new name space on the Japanese Wiktionary' | [production] | 
            
  | 17:48 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32553 - Rollbacker user group on sk.wikipedia.org' | [production] | 
            
  | 17:42 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32547 - Add pagetitle-view-mainpage to for zhwiki' | [production] | 
            
  | 17:37 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32546 - Cleanup interface_editor/interface_editors inconsistency' | [production] | 
            
  | 17:34 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32587 - Using localized section headings for uploads to Commons through upload-form' | [production] | 
            
  | 17:32 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 32587 - Using localized section headings for uploads to Commons through upload-form' | [production] | 
            
  | 15:35 | <mark> | Ran apt-get dist-upgrade && halt on db17 | [production] | 
            
  | 12:34 | <mutante> | sq38 - broken - giving up on it as it now hangs after RAC detection however you boot it | [production] | 
            
  | 12:05 | <mutante> | sq38 - Attempting PXE boot to reinstall (RT 2017) | [production] | 
            
  | 06:40 | <Ryan_Lane> | added a default sudo policy for labs instances. project members automatically have sudo privileges, excluding specific projects. | [production] | 
            
  | 05:03 | <Tim> | on bugzilla: allow reopened -> new transition to make dealing with malicious mass bug closures easier | [production] | 
            
  | 02:54 | <Tim> | fixed resolution and status fields of the rocketmail bugs | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.18) at Tue Nov 22 02:28:06 UTC 2011 | [production] | 
            
  | 01:57 | <LocalisationUpdate> | failed | [production] | 
            
  | 01:57 | <K4-713> | synchronized payments1, 2, and 3 to [[rev:103880|r103880]] | [production] | 
            
  | 01:30 | <awjr> | synchronizing thank_you module in CiviCRM on Grosley and Aluminium to [[rev:848|r848]] | [production] | 
            
  | 01:04 | <notpeter> | restzaring apache on formey | [production] | 
            
  | 00:59 | <awjr> | synchronizing thank_you module in CiviCRM on Grosley and Aluminium to [[rev:847|r847]] | [production] | 
            
  | 00:35 | <Tim> | blocked all account creation on bugzilla due to tor abuse | [production] | 
            
  | 00:30 | <asher> | synchronized wmf-config/CommonSettings.php  'changed wgUDPProfilerHost to professor.pmtpa.wmnet' | [production] | 
            
  | 00:24 | <K4-713> | Synchronized payments cluster to [[rev:103869|r103869]] | [production] | 
            
  | 00:23 | <domas> | nfs home is fucked up, 1.5s response times on simple ops | [production] | 
            
  | 00:21 | <midom> | synchronized wmf-config/db.php  'taking out db19, was out for past week or so' | [production] | 
            
  | 00:20 | <Tim> | on kaulen: firewalled 46.165.196.182 due to bugzilla trolling | [production] | 
            
  | 00:12 | <notpeter> | upgrading udp2log on emery to udplog_1.8-2 and restarting udp2log | [production] |