How to use SparkR within Rstudio?

Setting up Spark and SparkR is quite easy (assume you are running v.1.4): just grab one of the pre-built binaries and unzip to a folder. There is also a shell script to start SparkR from command line. The document suggest to put the following lines

Sys.setenv(SPARK_HOME="/home/shige/bin/spark")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
sc <- sparkR.init(master="local")

into the .Rprofile file. This, however, has the undesirable side effect of adding yet another directory to which R packages can be installed.

My solution is:

1. Create a soft link of the SparkR directory in the the directory where other R packages are installed (ln -s /home/shige/bin/spark/R/lib/SparkR /home/shige/R/x86_64-pc-linux-gnu-library/3.2)
2. Add (Sys.setenv(SPARK_HOME="/home/shige/bin/spark")) to the .Rprofile file.
3. Add (Sys.setenv(SPARKR_SUBMIT_ARGS ='"--packages" "com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')) to the .Rprofile.

All set.

SparkR

The SparkR project integrates R and Spark for big data processing. It looks very promising.

SEM DiagrammeR

The "sem" package makes use of the "DiagrammeR" package to draw path diagram, very cool!

IPython Markdown Opportunities in IPython Notebooks and Rstudio

Some interesting ideas here.

A new GTK theme

The paper theme looks really nice.

Rstudio supports Chinese input

Not so long ago, Rstudio did not fully support Chinese characters. The server version seemed fine but the desktop version did not. Now, with the new official version of 0.99.441, the support for Chinese input is complete.

Data science and our magical mind: Scott Mongeau at TEDxRSM

Useful talk (causal inference, scientific methods, etc.)

A Stargazer Cheatsheet

Useful tips here.

CSS templates for tables

I am compiling a list of free CSS templates to make good-looking tables:

1. Stylish CSS Tables Tutorial
2. Custom CSS for HTML generated using RStudio