| 2012-01-20
      
      § | 
    
  | 23:45 | <LeslieCarr> | ms6 sdc is undergoing fsck due to wrong fs type, bad option, bad superblock, or other on /dev/sdc1, | [production] | 
            
  | 23:44 | <binasher> | moving north america bits back to eqiad | [production] | 
            
  | 23:34 | <binasher> | moved bits eqiad to pmtpa (via scenarios/normal/bits-geo.wikimedia.org) | [production] | 
            
  | 23:32 | <LeslieCarr> | killed carnish on niobium , cpu load seems to be going down | [production] | 
            
  | 23:30 | <LeslieCarr> | reloading arsenic | [production] | 
            
  | 23:10 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Wrap some stupidly long lines' | [production] | 
            
  | 22:52 | <LeslieCarr> | rebooting cp3001 | [production] | 
            
  | 22:30 | <LeslieCarr> | reloading cp3001 | [production] | 
            
  | 22:24 | <LeslieCarr> | restarting networking on cp3001 | [production] | 
            
  | 22:03 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33789 - Enable botadmin usergroup on ml.wikipedia' | [production] | 
            
  | 22:00 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33789 - Enable botadmin usergroup on ml.wikipedia' | [production] | 
            
  | 21:58 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 33789 - Enable botadmin usergroup on ml.wikipedia' | [production] | 
            
  | 20:40 | <RobH> | dns servers all still online after update | [production] | 
            
  | 20:40 | <RobH> | dns update for dataset1001 | [production] | 
            
  | 18:33 | <LeslieCarr> | knsq9 has recovered post-reboot | [production] | 
            
  | 18:21 | <LeslieCarr> | knsq9 will be rebooted as it is dead, dead, dead | [production] | 
            
  | 17:48 | <LeslieCarr> | knsq9 is dead/overloaded | [production] | 
            
  | 15:11 | <mutante> | knsq30 still has bad disk, powering down again | [production] | 
            
  | 15:07 | <mutante> | powercycling knsq30 after replacing cable | [production] | 
            
  | 15:04 | <^demon> | fixed post-commit hook on formey email notifs to point to correct smtp server | [production] | 
            
  | 14:41 | <reedy> | synchronized php/cache/interwiki.cdb  'Updating interwiki cache' | [production] | 
            
  | 14:27 | <reedy> | synchronized php/cache/interwiki.cdb  'Updating interwiki cache' | [production] | 
            
  | 13:03 | <mutante> | reinstalling srv199 | [production] | 
            
  | 02:05 | <LocalisationUpdate> | completed (1.18) at Fri Jan 20 02:05:21 UTC 2012 | [production] | 
            
  | 01:17 | <awjr> | updated representative/zipcode mapping and some contact info for a handful of reps/senators for CongressLookup [[rev:109598|r109598]] | [production] | 
            
  | 01:12 | <binasher> | started another hotbackup of db38 to db52 | [production] | 
            
  | 01:08 | <asher> | synchronized wmf-config/db.php  'pulling db52' | [production] | 
            
  | 00:45 | <asher> | synchronized wmf-config/db.php  'doubling db52 weight' | [production] | 
            
  | 00:38 | <asher> | synchronized wmf-config/db.php  'lowering db52 weight' | [production] | 
            
  | 00:32 | <binasher> | deployed new enwiki slave, db52 | [production] | 
            
  | 00:32 | <asher> | synchronized wmf-config/db.php  'setting db52 to full weight' | [production] | 
            
  | 00:19 | <asher> | synchronized wmf-config/db.php  'adding new enwiki slave db52, with a low weight' | [production] | 
            
  | 00:08 | <preilly> | push weekly mobile frontend update | [production] | 
            
  | 00:08 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php  'weekly update to Mobile Frontend' | [production] | 
            
  
    | 2012-01-19
      
      § | 
    
  | 23:56 | <Ryan_Lane> | changed global roles netadmins and sysadmins to be virtual static groups in ldap that autopopulate with any user that has objectclass=novauser | [production] | 
            
  | 23:15 | <Tim> | rebuilt wikidiff2 with package name php-wikidiff2, removed lucid package php5-wikidiff2 from apt using "reprepro remove" | [production] | 
            
  | 22:52 | <Tim> | recompiled wikidiff2 and put the new version up on apt.wikimedia.org | [production] | 
            
  | 21:51 | <Jeff_Green> | starting conversion of fundraisingdb 'faulkner' tables from myisam to innodb, expect replication delays | [production] | 
            
  | 21:12 | <binasher> | starting slaving db52 from db36, running hotbackup of db32 to db53 | [production] | 
            
  | 20:36 | <RobH> | dataset1001 shut down for later use | [production] | 
            
  | 20:27 | <RobH> | dataset1001 mgmt online | [production] | 
            
  | 20:15 | <RobH> | dataset1001.mgmt even | [production] | 
            
  | 20:15 | <RobH> | updating dns for dataset1mgmt | [production] | 
            
  | 20:03 | <LeslieCarr> | testing | [production] | 
            
  | 20:00 | <LeslieCarr> | testing | [production] | 
            
  | 05:16 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'new sopa banner' | [production] | 
            
  | 05:14 | <laner> | synchronized wmf-config/InitialiseSettings.php  'Enabling anon editing for enwiki' | [production] | 
            
  | 05:12 | <laner> | synchronized wmf-config/InitialiseSettings.php  'Enabling page creation for users' | [production] | 
            
  | 05:08 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'new sopa banner' | [production] | 
            
  | 05:06 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'new sopa banner' | [production] |