| 
      
        2014-07-09
      
      §
     | 
  
    
  | 22:05 | 
  <mutante> | 
  restarted apache on zirconium for config change | 
  [production] | 
            
  | 20:07 | 
  <subbu> | 
  deployed parsoid 1632288d | 
  [production] | 
            
  | 18:36 | 
  <yurik> | 
  Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 22s) | 
  [production] | 
            
  | 18:29 | 
  <yurik> | 
  Synchronized php-1.24wmf11/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 24s) | 
  [production] | 
            
  | 17:17 | 
  <demon> | 
  Synchronized wmf-config/InitialiseSettings.php: eswiki cirrus (duration: 00m 04s) | 
  [production] | 
            
  | 16:51 | 
  <csteipp> | 
  Finished scap: Update CentralAuth for Global Rename (duration: 28m 46s) | 
  [production] | 
            
  | 16:22 | 
  <csteipp> | 
  Started scap: Update CentralAuth for Global Rename | 
  [production] | 
            
  | 16:17 | 
  <mark> | 
  ulsfo is now offline | 
  [production] | 
            
  | 16:17 | 
  <mark> | 
  Shutdown NTT BGP sessions on cr2-ulsfo | 
  [production] | 
            
  | 16:13 | 
  <mark> | 
  Shutdown TiNet BGP sessions on cr1-ulsfo | 
  [production] | 
            
  | 16:10 | 
  <mark> | 
  Shutdown IXP BGP sessions on cr2-ulsfo | 
  [production] | 
            
  | 16:10 | 
  <mark> | 
  Shutdown WMF HQ BGP sessions on cr2-ulsfo | 
  [production] | 
            
  | 16:09 | 
  <mark> | 
  Shutdown WMF HQ BGP sessions on cr1-ulsfo | 
  [production] | 
            
  | 16:03 | 
  <hoo> | 
  Synchronized php-1.24wmf12/extensions/Wikidata/: Update Wikibase to fix a fatal and various JS things (duration: 00m 14s) | 
  [production] | 
            
  | 14:13 | 
  <hashar> | 
  Jenkins: bringing back  puppet-compiler02.eqiad.wmflabs node online. /tmp get filled when running huge catalog compilations which causes Jenkins to unpool the node :/ | 
  [production] | 
            
  | 13:30 | 
  <godog> | 
  reboot ms-be1005, raid controller confused (?) after disk replacement | 
  [production] | 
            
  | 12:52 | 
  <godog> | 
  umounted sdg1 on ms-be1005, device disappeared, errors in dmesg | 
  [production] | 
            
  | 12:36 | 
  <bblack> | 
  enabled amssq47 text frontend cache in pybal for esams | 
  [production] | 
            
  | 09:30 | 
  <hashar> | 
  restarted Zuul to clear out stalled items in queue | 
  [production] | 
            
  | 09:12 | 
  <hashar> | 
  Jenkins being slow because the mediawiki-core* jobs history cache has been wiped out while updating their configuration. Jenkins is busy processing the history :( | 
  [production] | 
            
  | 09:02 | 
  <hashar> | 
  Jenkins killing slave process on lanthanum. Some job is stalled and unrecoverable. | 
  [production] | 
            
  | 08:53 | 
  <godog> | 
  upgrade ms-be1013/1014/1015 (zone5) to icehouse swift | 
  [production] | 
            
  | 08:51 | 
  <hashar> | 
  Jenkins migrating jobs to use $ZUUL_URL instead of git://zuul.eqiad.wmnet   Preparing to scale out Zuul merger to several nodes | 
  [production] | 
            
  | 08:19 | 
  <godog> | 
  upgrade ms-be1009/1010/1011 (zone4)  to swift icehouse | 
  [production] | 
            
  | 08:04 | 
  <hashar> | 
  Jenkins: granted matanya the ability to manually trigger builds. Use case: the puppet compiler! | 
  [production] | 
            
  | 08:02 | 
  <godog> | 
  upgrade ms-be1005/1006/1007 (zone3) to swift icehouse | 
  [production] | 
            
  | 03:37 | 
  <mutante> | 
  ran puppet on neon - false puppet failure alarms | 
  [production] | 
            
  | 02:55 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Jul  9 02:54:37 UTC 2014 (duration 54m 36s) | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.24wmf12) at 2014-07-09 02:25:33+00:00 | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  completed (1.24wmf11) at 2014-07-09 02:14:38+00:00 | 
  [production] | 
            
  | 01:26 | 
  <mutante> | 
  Bugzilla - enabled https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security | 
  [production] | 
            
  | 00:50 | 
  <mutante> | 
  restarted gitblit service | 
  [production] | 
            
  
    | 
      
        2014-07-08
      
      §
     | 
  
    
  | 22:50 | 
  <mutante> | 
  radon (phab)- package and kernel upgrades, rebooting | 
  [production] | 
            
  | 20:22 | 
  <legoktm> | 
  finished running migrateAccount.php --attachbroken --attachmissing (bug 61876) | 
  [production] | 
            
  | 20:07 | 
  <legoktm> | 
  finished migrateAccount.php --safe, now starting migrateAccount.php --attachbroken | 
  [production] | 
            
  | 20:05 | 
  <mutante> | 
  restarted apache on ytterbium | 
  [production] | 
            
  | 19:47 | 
  <K4-713> | 
  updated payments fraud filters again | 
  [production] | 
            
  | 19:47 | 
  <legoktm> | 
  running migrateAccount.php --safe for accounts only existing on one wiki (bug 39817) | 
  [production] | 
            
  | 19:27 | 
  <mutante> | 
  this should have fixed all the services behind misc. varnish now getting an actual "A" rating on ssllabs | 
  [production] | 
            
  | 19:21 | 
  <mutante> | 
  arr, i meant "nginx", not varnish | 
  [production] | 
            
  | 19:15 | 
  <mutante> | 
  restarting varnish on cp1043/cp1044 (misc cluster) | 
  [production] | 
            
  | 18:56 | 
  <cmjohnson1> | 
  disconnecting serial cable from psw1-c2-eqiad  | 
  [production] | 
            
  | 18:50 | 
  <csteipp> | 
  patch for bug66608 deployed to wmf11/12 | 
  [production] | 
            
  | 18:50 | 
  <K4-713> | 
  updated fraud filters on payments cluster | 
  [production] | 
            
  | 18:28 | 
  <reedy> | 
  Synchronized robots-private.txt: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 18:27 | 
  <reedy> | 
  Synchronized wmf-config/: (no message) (duration: 00m 15s) | 
  [production] | 
            
  | 18:20 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf12 | 
  [production] | 
            
  | 15:22 | 
  <reedy> | 
  Purged l10n cache for 1.24wmf10 | 
  [production] | 
            
  | 15:21 | 
  <reedy> | 
  Purged l10n cache for 1.24wmf9 | 
  [production] | 
            
  | 15:15 | 
  <anomie> | 
  Synchronized php-1.24wmf11/extensions/Scribunto/: SWAT: Fix regression in os.date and os.time at module scope [[gerrit:144559]] (duration: 00m 10s) | 
  [production] |