| 2015-02-03
      
      § | 
    
  | 22:54 | <bd808> | restarted elasticsearch on logstash1003 | [production] | 
            
  | 22:53 | <bd808> | starting rolling restart of logstash elasticsearch cluster to pick up index.merge.scheduler.max_thread_count puppet change | [production] | 
            
  | 22:52 | <robh> | magnesium apache reload for rt cert replacement | [production] | 
            
  | 22:43 | <robh> | replacing etherpad sha1 with sha256 cert | [production] | 
            
  | 22:13 | <reedy> | Synchronized php-1.25wmf15/extensions/WikimediaMaintenance: tmp script (duration: 00m 07s) | [production] | 
            
  | 21:38 | <maxsem> | Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/188300/ (duration: 00m 07s) | [production] | 
            
  | 21:32 | <bblack> | repooled cp10[67]0 | [production] | 
            
  | 21:29 | <mutante> | restarted gitblit | [production] | 
            
  | 21:17 | <andrewbogott> | increased opendj  lookthrough-limit to 12000 on both ldap hosts.  We just hit lucky 5000 users and some queries stopped working. | [production] | 
            
  | 20:59 | <bblack> | depooling cp1060, cp1070 (1 each bits + mobile) for reinstall | [production] | 
            
  | 20:59 | <robh> | updating gerrit.wikimedia.org cert | [production] | 
            
  | 19:30 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf15 | [production] | 
            
  | 18:17 | <YuviPanda> | set up new images for ubuntu trusty / precise on labs, for https://phabricator.wikimedia.org/T87003 | [production] | 
            
  | 17:23 | <bblack> | repooled cp1064 | [production] | 
            
  | 17:07 | <bd808> | restarted elasticsearch on logstash1002; OOM | [production] | 
            
  | 16:55 | <bblack> | cp1064 (eqiad upload cache) depooled in pybal | [production] | 
            
  | 15:39 | <_joe_> | installing a new package to canary servers | [production] | 
            
  | 14:55 | <_joe_> | uploaded a new hhvm package version, deploying to testwiki and beta | [production] | 
            
  | 08:50 | <reedy> | Synchronized wmf-config/: Bye bye Solarium (duration: 00m 06s) | [production] | 
            
  | 08:20 | <ori> | Synchronized wmf-config/InitialiseSettings.php: Ie7b32e3d8: Add log group for T87645 (duration: 00m 05s) | [production] | 
            
  | 08:19 | <ori> | Synchronized php-1.25wmf14/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 07s) | [production] | 
            
  | 08:17 | <ori> | Synchronized php-1.25wmf15/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 05s) | [production] | 
            
  | 08:14 | <paravoid> | radium: upgrade tor to the latest torproject.org version | [production] | 
            
  | 08:10 | <springle> | wikitech mysql restart to fix novaold errors | [production] | 
            
  | 05:14 | <springle> | wikitech virt1000 test db dump T88311 | [production] | 
            
  | 04:49 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Feb  3 04:48:38 UTC 2015 (duration 48m 37s) | [production] | 
            
  | 02:49 | <ori> | deployed https://gerrit.wikimedia.org/r/#/c/187304/ (php-set X-Analytics header) to both production branches. | [production] | 
            
  | 02:42 | <bblack> | cp1065 re-pooled in pybal | [production] | 
            
  | 02:31 | <LocalisationUpdate> | completed (1.25wmf15) at 2015-02-03 02:30:18+00:00 | [production] | 
            
  | 02:30 | <l10nupdate> | Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s) | [production] | 
            
  | 02:19 | <mutante> | installing package upgrades on radium | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.25wmf14) at 2015-02-03 02:16:37+00:00 | [production] | 
            
  | 02:16 | <l10nupdate> | Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s) | [production] | 
            
  | 02:00 | <bblack> | depool cp1065 (text eqiad in pybal -> jessie) | [production] | 
            
  | 01:59 | <bd808> | Manually created apifeatureusage-2015.02.02 and apifeatureusage-2015.02.03 indices in elasticsearch; clsuter needs rolling restart for autocreate to work for these names | [production] | 
            
  | 01:51 | <bd808> | restarted logstash on logstash1001 | [production] | 
            
  | 01:51 | <bd808> | restarted elasticsearch on logstash1003 | [production] | 
            
  | 00:03 | <mutante> | rbf2002 - error while setting up RAID during installer (rbf2001 did not have this? or did it?) | [production] | 
            
  | 00:02 | <mutante> | rbf2001 - initial puppet run, adding users | [production] | 
            
  
    | 2015-02-02
      
      § | 
    
  | 23:59 | <mutante> | signing puppet cert for rbf2001, PXE booting rbf2002 | [production] | 
            
  | 22:18 | <ori> | Synchronized php-1.25wmf14/extensions/XAnalytics: (no message) (duration: 00m 05s) | [production] | 
            
  | 22:18 | <ori> | Synchronized php-1.25wmf15/extensions/XAnalytics: (no message) (duration: 00m 05s) | [production] | 
            
  | 21:16 | <subbu> | deployed parsoid version e3c9ae99 | [production] | 
            
  | 19:59 | <ejegg> | updated payments from ce73ed11de9775a596c51acdc036503751961bc8 to 1e9b78e9a8bf557a710988620bd6f1a335787173 | [production] | 
            
  | 19:11 | <aude> | Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking on Wikidata (duration: 00m 07s) | [production] | 
            
  | 18:15 | <mutante> | restarted hhvm on mw1207 | [production] | 
            
  | 18:05 | <mutante> | started nginx on ms1001, dataset1001 | [production] | 
            
  | 18:00 | <mutante> | restarted gitblit | [production] | 
            
  | 17:20 | <marktraceur> | Synchronized php-1.25wmf15/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf15] Fix UploadWizard for ogg files (duration: 00m 07s) | [production] | 
            
  | 17:20 | <marktraceur> | Synchronized php-1.25wmf14/extensions/UploadWizard/resources/mw.FormDataTransport.js: [SWAT] [wmf14] Fix UploadWizard for ogg files (duration: 00m 06s) | [production] |