| 
      
        2011-11-22
      
      §
     | 
  
    
  | 21:28 | 
  <Ryan_Lane> | 
  upgrading packages on virt3 | 
  [production] | 
            
  | 21:26 | 
  <Ryan_Lane> | 
  upgrading packages on virt2 | 
  [production] | 
            
  | 21:25 | 
  <Ryan_Lane> | 
  upgrading packages on virt1 | 
  [production] | 
            
  | 21:17 | 
  <Ryan_Lane> | 
  stopping all nova services | 
  [production] | 
            
  | 21:10 | 
  <reedy> | 
  synchronized wmf-config/abusefilter.php  'Remove lines of sites setting abusefilter-private to anything' | 
  [production] | 
            
  | 20:19 | 
  <catrope> | 
  synchronized php-1.18/includes/Export.php  '[[rev:103954|r103954]]' | 
  [production] | 
            
  | 20:04 | 
  <Ryan_Lane> | 
  upgrading nova from cactus to diablo. during this time period, instances may be occasionally inaccessible, and labsconsole may also be occasionally inaccessible | 
  [production] | 
            
  | 19:45 | 
  <binasher> | 
  increased udp recv buffer on locke to 512M. i think lp-filter needs some work. | 
  [production] | 
            
  | 19:13 | 
  <binasher> | 
  udp2log on emery was broken post upgrade due to files under /var/log/squid being owned by human users. fixed + restarted | 
  [production] | 
            
  | 18:03 | 
  <midom> | 
  synchronized wmf-config/db.php  'readding db19' | 
  [production] | 
            
  | 17:52 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32521 - Request for new name space on the Japanese Wiktionary' | 
  [production] | 
            
  | 17:48 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32553 - Rollbacker user group on sk.wikipedia.org' | 
  [production] | 
            
  | 17:42 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32547 - Add pagetitle-view-mainpage to for zhwiki' | 
  [production] | 
            
  | 17:37 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32546 - Cleanup interface_editor/interface_editors inconsistency' | 
  [production] | 
            
  | 17:34 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32587 - Using localized section headings for uploads to Commons through upload-form' | 
  [production] | 
            
  | 17:32 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32587 - Using localized section headings for uploads to Commons through upload-form' | 
  [production] | 
            
  | 15:35 | 
  <mark> | 
  Ran apt-get dist-upgrade && halt on db17 | 
  [production] | 
            
  | 12:34 | 
  <mutante> | 
  sq38 - broken - giving up on it as it now hangs after RAC detection however you boot it | 
  [production] | 
            
  | 12:05 | 
  <mutante> | 
  sq38 - Attempting PXE boot to reinstall (RT 2017) | 
  [production] | 
            
  | 06:40 | 
  <Ryan_Lane> | 
  added a default sudo policy for labs instances. project members automatically have sudo privileges, excluding specific projects. | 
  [production] | 
            
  | 05:03 | 
  <Tim> | 
  on bugzilla: allow reopened -> new transition to make dealing with malicious mass bug closures easier | 
  [production] | 
            
  | 02:54 | 
  <Tim> | 
  fixed resolution and status fields of the rocketmail bugs | 
  [production] | 
            
  | 02:25 | 
  <LocalisationUpdate> | 
  completed (1.18) at Tue Nov 22 02:28:06 UTC 2011 | 
  [production] | 
            
  | 01:57 | 
  <LocalisationUpdate> | 
  failed | 
  [production] | 
            
  | 01:57 | 
  <K4-713> | 
  synchronized payments1, 2, and 3 to [[rev:103880|r103880]] | 
  [production] | 
            
  | 01:30 | 
  <awjr> | 
  synchronizing thank_you module in CiviCRM on Grosley and Aluminium to [[rev:848|r848]] | 
  [production] | 
            
  | 01:04 | 
  <notpeter> | 
  restzaring apache on formey | 
  [production] | 
            
  | 00:59 | 
  <awjr> | 
  synchronizing thank_you module in CiviCRM on Grosley and Aluminium to [[rev:847|r847]] | 
  [production] | 
            
  | 00:35 | 
  <Tim> | 
  blocked all account creation on bugzilla due to tor abuse | 
  [production] | 
            
  | 00:30 | 
  <asher> | 
  synchronized wmf-config/CommonSettings.php  'changed wgUDPProfilerHost to professor.pmtpa.wmnet' | 
  [production] | 
            
  | 00:24 | 
  <K4-713> | 
  Synchronized payments cluster to [[rev:103869|r103869]] | 
  [production] | 
            
  | 00:23 | 
  <domas> | 
  nfs home is fucked up, 1.5s response times on simple ops | 
  [production] | 
            
  | 00:21 | 
  <midom> | 
  synchronized wmf-config/db.php  'taking out db19, was out for past week or so' | 
  [production] | 
            
  | 00:20 | 
  <Tim> | 
  on kaulen: firewalled 46.165.196.182 due to bugzilla trolling | 
  [production] | 
            
  | 00:12 | 
  <notpeter> | 
  upgrading udp2log on emery to udplog_1.8-2 and restarting udp2log | 
  [production] | 
            
  
    | 
      
        2011-11-21
      
      §
     | 
  
    
  | 23:35 | 
  <awjrichards> | 
  synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.body.php  '[[rev:103849|r103849]]' | 
  [production] | 
            
  | 23:34 | 
  <awjrichards> | 
  synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.php  '[[rev:103849|r103849]]' | 
  [production] | 
            
  | 23:00 | 
  <LeslieCarr> | 
  updating dns with bast1001 | 
  [production] | 
            
  | 22:25 | 
  <binasher> | 
  mediawiki sync scripts under fenari:/usr/local/bin now log deploys to graphite  | 
  [production] | 
            
  | 22:14 | 
  <LeslieCarr> | 
  taking scandium for new bastion host | 
  [production] | 
            
  | 21:39 | 
  <LeslieCarr> | 
  repreffing the tinet route | 
  [production] | 
            
  | 20:01 | 
  <Ryan_Lane> | 
  gerrit is back up | 
  [production] | 
            
  | 19:57 | 
  <Ryan_Lane> | 
  bringing gerrit down again, to move all of /var/lib/gerrit2 to the partition | 
  [production] | 
            
  | 19:49 | 
  <notpeter> | 
  upgrading apache version on stafford to fix sec vuln | 
  [production] | 
            
  | 18:27 | 
  <mutante> | 
  changed default text for new etherpads (bz #29496) | 
  [production] | 
            
  | 17:36 | 
  <Ryan_Lane> | 
  bringing gerrit down, to move its data to a new partition | 
  [production] | 
            
  | 10:42 | 
  <apergos> | 
  shutting down snapshot3 for move to internal network and OS upgrade | 
  [production] | 
            
  | 02:21 | 
  <LocalisationUpdate> | 
  completed (1.18) at Mon Nov 21 02:24:05 UTC 2011 | 
  [production] | 
            
  | 01:57 | 
  <LocalisationUpdate> | 
  failed | 
  [production] |