Using C and C++ for data science

Useful information here.

Arch-Data-Science

Data science toolkit for Arch and Arch-based system here.

Real-World Machine Learning

I find this book very helpful. The introduction chapter is freely available.

Microsoft Analytics in 2016

Here is a thorough introduction of data science solution offered by Microsoft.

Notebook interface for everything

Zeppelin provides a unified interface for nearly all the major data process engines, including Spark. I quickly set it up on a virtual machine and gave it a test run. It works great.

This one has SparkR support built-in.

I was never really into the ipython/jupyter notebook mainly because there is nothing they can do that a good IDE such as PyCharm or Rodeo cannot. Zeppelin is different because its capability of tightly integrating different Spark front-ends, including Scala, Python, and R is uniquely powerful. I would call this revolutionary.

How to use SparkR within Rstudio?

Setting up Spark and SparkR is quite easy (assume you are running v.1.4): just grab one of the pre-built binaries and unzip to a folder. There is also a shell script to start SparkR from command line. The document suggest to put the following lines

Sys.setenv(SPARK_HOME="/home/shige/bin/spark")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
sc <- sparkR.init(master="local")

into the .Rprofile file. This, however, has the undesirable side effect of adding yet another directory to which R packages can be installed.

My solution is:

1. Create a soft link of the SparkR directory in the the directory where other R packages are installed (ln -s /home/shige/bin/spark/R/lib/SparkR /home/shige/R/x86_64-pc-linux-gnu-library/3.2)
2. Add (Sys.setenv(SPARK_HOME="/home/shige/bin/spark")) to the .Rprofile file.
3. Add (Sys.setenv(SPARKR_SUBMIT_ARGS ='"--packages" "com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')) to the .Rprofile.

All set.

Great R packages for data import, wrangling & visualization

Nice summary of useful R packages, a few of which are unfamiliar to me before.

Shiny, ggvis, and rmarkdown

I have been using rmarkdown in my research and really enjoyed it. I also tried to get my head around shiny and ggvis. I can understand what they do but, compared to rmarkdown, dplyr, and ggplot2, they seemed less relevant. Yesterday I went to the R-Day of the Strata+Hadoop meeting and the presentations made by Winston Chang and Garrett Grolemund really changed my opinion. And the timing is perfect: now I have strong incentive to incorporate both technologies into my Advanced Analytics course in the Spring of 2015.

According to Hadley Wichham, the author of dplyr and a number of other excellent packages, the next version of dplyr will have a function for recoding.

Data science toolkit

Here is a list of useful resources for data science.

R, Python, and Julia

Good posting and lots of excellent comments here!

Will it Python?

Will It Python provides Python translation of the R code used in Machine Learning for Hackers.

Python Tools for Machine Learning

Useful resources.