A useful disk.frame tutorial

Useful package and useful tutorial here.

Building Scalable Data Pipelines with Microsoft R Server and Azure Data Factory

Useful information on big data computation using Microsoft platform.

New package "dplyrr" - Utilities for comfortable use of dplyr with databases

This package is potentially useful for big data analysis.

Analyzing big data with Revolution R Enterprise

A short and yet clear introduction here.

How to use SparkR within Rstudio?

Setting up Spark and SparkR is quite easy (assume you are running v.1.4): just grab one of the pre-built binaries and unzip to a folder. There is also a shell script to start SparkR from command line. The document suggest to put the following lines

Sys.setenv(SPARK_HOME="/home/shige/bin/spark")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
sc <- sparkR.init(master="local")

into the .Rprofile file. This, however, has the undesirable side effect of adding yet another directory to which R packages can be installed.

My solution is:

1. Create a soft link of the SparkR directory in the the directory where other R packages are installed (ln -s /home/shige/bin/spark/R/lib/SparkR /home/shige/R/x86_64-pc-linux-gnu-library/3.2)
2. Add (Sys.setenv(SPARK_HOME="/home/shige/bin/spark")) to the .Rprofile file.
3. Add (Sys.setenv(SPARKR_SUBMIT_ARGS ='"--packages" "com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')) to the .Rprofile.

All set.

SparkR

The SparkR project integrates R and Spark for big data processing. It looks very promising.

Snowdoop

This looks promising.

ffbases2

The "ffbase2" packages aims to combine the "dplyr" and "ff" packages, really cool.

RHadoop

Looks like RHadoop is still the tool of choice for R-based big data analysis.

I found this post helpful to get Hadoop installed on my workstation.

The Big Data Brain Drain: Why Science is in Trouble

Insightful observation and interesting comments here.

Don't use Hadoop - your data isn't that big

Here.

Use the open source R in the similar way to SAS

Here is how. Very good examples!

By the way, learning materials for the ff package are surprisingly sparse.

R and Hadoop

Some useful information here.

R 3.0 supports long vector

It is a very exciting news that the new R 3.0 supports long vector! This is major step toward a viable big data platform.

Replicating the Revolutionary Big Data Analysis using ordinary R

This post shows how.