| 2014-01-29
      
      § | 
    
  | 23:58 | <RobH> | all lists.w.o updates done and now on individual certificate | [production] | 
            
  | 23:26 | <awight> | rollback payments to 3cda9052104d90a746ba5ca243c22ab8d36e0274 | [production] | 
            
  | 23:14 | <awight> | updated payments from 65d6fa8d44a483a3789dbdee2c4926143c3a2338 to bcf5ec47bd995f89f000f4e57ffeee527eb29e11 | [production] | 
            
  | 22:41 | <awight> | updating payments from 3cda9052104d90a746ba5ca243c22ab8d36e0274 to 65d6fa8d44a483a3789dbdee2c4926143c3a2338 | [production] | 
            
  | 22:13 | <RobH> | otrs now using its own cert per rt 6702, confirmed working chain | [production] | 
            
  | 21:48 | <RobH> | svn.w.o on own cert, confirmed chain is properly functioning | [production] | 
            
  | 21:45 | <RobH> | updating svn to use own cert, service disruption may (but shouldnt) occur | [production] | 
            
  | 19:49 | <cmjohnson1> | tungsten replacing failing disk at slot 10 | [production] | 
            
  | 19:42 | <cmjohnson1> | db1002 replacing disk at slot 1 | [production] | 
            
  | 18:53 | <manybubbles> | shutting down searchidx1001 for hardware fix | [production] | 
            
  | 17:27 | <demon> | synchronized wmf-config/db-eqiad.php  'Removing underscores from class names' | [production] | 
            
  | 17:27 | <demon> | synchronized wmf-config/db-labs.php  'Removing underscores from class names' | [production] | 
            
  | 17:26 | <demon> | synchronized wmf-config/db-pmtpa.php  'Removing underscores from class names' | [production] | 
            
  | 17:06 | <demon> | synchronized wmf-config/InitialiseSettings.php  'Enable Cirrus for huwiki + some shard config' | [production] | 
            
  | 17:05 | <demon> | synchronized wmf-config/CirrusSearch-common.php  'Turn commons file searching back on' | [production] | 
            
  | 16:53 | <LeslieCarr> | aggregated labnet1001 secondary port | [production] | 
            
  | 14:50 | <springle> | synchronized wmf-config/db-eqiad.php  'db1040 LB full steam' | [production] | 
            
  | 14:14 | <springle> | synchronized wmf-config/db-eqiad.php  'repool db1040, warm up' | [production] | 
            
  | 12:06 | <andrewbogott> | deployed (broken!) havana/nova on virt1000, 1001, 1002, 1003, labnet1001.  Should be safe, but any recent breakage on virt1000 is most likely a side-effect. | [production] | 
            
  | 09:15 | <paravoid> | reenable ospfv3 on the eqiad/esams link | [production] | 
            
  | 08:54 | <paravoid> | applying NTP access lists on cr{1,2}-{esams,knams,eqiad,pmtpa,sdtpa,ulsfo}, csw2-esams, pfw1-eqiad | [production] | 
            
  | 07:46 | <springle> | powercycled unresponsive db1042, /a tank data mount failed on boot, vgchange -a y + mount + xfs_check. still investigating | [production] | 
            
  | 06:51 | <springle> | synchronized wmf-config/db-eqiad.php  'depool db1042 for schema changes' | [production] | 
            
  | 02:58 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-01-29 02:58:21+00:00 | [production] | 
            
  | 02:31 | <LocalisationUpdate> | completed (1.23wmf10) at 2014-01-29 02:31:30+00:00 | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.23wmf11) at 2014-01-29 02:16:57+00:00 | [production] | 
            
  | 00:34 | <reedy> | updated /a/common to {{Gerrit|I2294bac73}}: Throttle now handles IP ranges. | [production] | 
            
  | 00:34 | <ori> | another recurrent error in antimony:/var/log/upstart/gitblit.log : "org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed." repeats for each repository. traces: <http://p.defau.lt/?VKPpRReIO7aGDxmHfSJeWw> | [production] | 
            
  | 00:34 | <reedy> | synchronized wmf-config/missing.php | [production] | 
            
  | 00:29 | <ori> | gitblit on antimony crashed with org.eclipse.jetty.io.EofException. trace: <http://p.defau.lt/?1E8Wwj7_089XS6dtu6nqfA>. lots of java.lang.NullPointerException due to malformed URLs, but these appear to happen continuously. | [production] | 
            
  | 00:20 | <reedy> | synchronized wmf-config/ | [production] | 
            
  
    | 2014-01-28
      
      § | 
    
  | 23:36 | <Reedy> | Created FlaggedRevs tables on cewiki | [production] | 
            
  | 22:27 | <reedy> | updated /a/common to {{Gerrit|I6b16562d0}}: Rename phase1 dblist to group0 | [production] | 
            
  | 22:19 | <reedy> | synchronized database lists files: | [production] | 
            
  | 22:19 | <reedy> | synchronized docroot and w | [production] | 
            
  | 20:31 | <RobH> | fixing virt0 wikitech cert, wikitech may restart | [production] | 
            
  | 19:46 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'touch' | [production] | 
            
  | 19:46 | <reedy> | synchronized database lists files: | [production] | 
            
  | 19:29 | <reedy> | synchronized php-1.23wmf11/extensions/Wikibase | [production] | 
            
  | 19:23 | <reedy> | updated /a/common to {{Gerrit|I4abcbb866}}: Non Wikipedias to 1.23wmf11 | [production] | 
            
  | 19:22 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf11 | [production] | 
            
  | 18:40 | <reedy> | updated /a/common to {{Gerrit|Ic2ea727b4}}: Fixup default file permissions | [production] |