Running sparklyr – RStudio’s R Interface to Spark on Amazon EMR

Useful tutorial.

Bigtop

Apache Bigtop is by far the easiest to install Hadoop distribution. I was able to get it work on an
CentOS VM. Unfortunately the bundled Spark is 1.5.1, which does not work well with the sparklyr package. I have not figured out how to make the yarn manager work with a third-party copy of Spark  yet. Need to wait for the Bigtop distribution to upgrade.

Scalable R on Spark with SparkR, sparklyr and RevoScaleR

Useful tutorial here.

Sparklyr

The new sparklyr package from rstudio provides a convenient interface between R/Rstudio and Spark. It runs well on Linux; it also works on Windows for Spark 1.6.2 and lower. For some reasons, it does not work with Spark 2.0 on Windows. I assume it will get fixed in subsequent releases.

Microsoft Analytics in 2016

Here is a thorough introduction of data science solution offered by Microsoft.

Notebook interface for everything

Zeppelin provides a unified interface for nearly all the major data process engines, including Spark. I quickly set it up on a virtual machine and gave it a test run. It works great.

This one has SparkR support built-in.

I was never really into the ipython/jupyter notebook mainly because there is nothing they can do that a good IDE such as PyCharm or Rodeo cannot. Zeppelin is different because its capability of tightly integrating different Spark front-ends, including Scala, Python, and R is uniquely powerful. I would call this revolutionary.

Getting Started with Spark (in Python)

Using Python, here.