Processing web analysis log using R

webreadr seems to be the right starting place for this.

Computing by groups within data.frames with dplyr and broom

This is quite useful.

Rstudio tips

Very useful information.

Top programming languages that will be most popular in 2017

R is doing well.

Running sparklyr – RStudio’s R Interface to Spark on Amazon EMR

Useful tutorial.

Bigtop

Apache Bigtop is by far the easiest to install Hadoop distribution. I was able to get it work on an
CentOS VM. Unfortunately the bundled Spark is 1.5.1, which does not work well with the sparklyr package. I have not figured out how to make the yarn manager work with a third-party copy of Spark  yet. Need to wait for the Bigtop distribution to upgrade.

The importance of googling ...

Trying to submit a manuscript today. The main manuscript was written in Rmarkdown whereas the supplementary materials in Word. The online submission system kept complaining abut my pdf file. After some googling and tweaking, it turns out the problem was caused by indicator function "\mathbbm{1}". Following the suggestion given here, I replaced it with "\mathds{1}". Problem solved!

Building Scalable Data Pipelines with Microsoft R Server and Azure Data Factory

Useful information on big data computation using Microsoft platform.

RStudio IDE Easy Tricks You Might’ve Missed

I missed a few of the tricks mentioned here. Very neat indeed!

Tidy Text Mining with R

Cool text mining book.

Rmazon

This is a simple but useful package for downloading product information and reviews from Amazon.com.

Scalable R on Spark with SparkR, sparklyr and RevoScaleR

Useful tutorial here.

Interview with J.J. Allaire

Very informative interview here.

Real-World Machine Learning

I find this book very helpful. The introduction chapter is freely available.

Sparklyr

The new sparklyr package from rstudio provides a convenient interface between R/Rstudio and Spark. It runs well on Linux; it also works on Windows for Spark 1.6.2 and lower. For some reasons, it does not work with Spark 2.0 on Windows. I assume it will get fixed in subsequent releases.

LaplacesDemon is back

Looks like LaplacesDemon package is back. Now we have a pure R-based Bayesian computation platform.

Microsoft Analytics in 2016

Here is a thorough introduction of data science solution offered by Microsoft.

Making Causal Impact Analysis Easy

Very helpful blog post regarding the CausalImpact package and MarketMatching package.

Setting Up New R Notebook

This is extremely cool!

Microsoft and Linux: True Romance or Toxic Love?

Insightful article.

tm reading in data frame and keep text's id

Here are some useful discussions.

Multiple imputation using R

R has a long list of packages for multiple imputation. The main problem is integration: statistical procedures in other packages may or may not work with the imputation procedures. I have been using Amelia together with Zelig. Because they were written by the same group, they work well together. However, I have been having trouble with making multiple imputation to work with the plm package. After searching the internet, here comes the solution:

1. Impute the missing data using Amelia or Mice.
2. Estimate the model on each imputed data.
3. Use the mitools package to extract and combine results. 

For example, here is a simple example:

...

imp <- mice(d)

mydata <- imputationList(lapply(1:5, complete, x = imp))

fit <- lapply(mydata$imputations, function(x){

plm(cog3pl ~ oc + grade9 + boy + han + ruralbirth, data = x,

index = c("schids"), model = "pooling")})

betas <- MIextract(fit, fun = coef)

vars <- MIextract(fit, fun = vcov)

summary(MIcombine(betas, vars))

I bet this will work for most, if not all, estimation procedures in R.

Another text analysis package

Quanteda seems to be a serious contender for analyzing textual data using R.

Rstudio 0.99.878 becomes official

Rstudio 0.99.878 becomes official. There is server version in AUR that can be used. However, the pandoc that comes with that version seems to have problems on Arch/Manjora. The discussion here is very helpful. One easy solution is to replace the built-in pandoc with the one that comes with the system:

sudo mv /usr/lib/rstudio-server/bin/pandoc/pandoc /usr/lib/rstudio-server/bin/pandoc/pandoc_old
sudo mv /usr/lib/rstudio-server/bin/pandoc/pandoc-citeproc /usr/lib/rstudio-server/bin/pandoc/pandoc-citeproc_old

sudo ln -s /usr/bin/pandoc /usr/lib/rstudio-server/bin/pandoc/pandoc
sudo ln -s /usr/bin/pandoc-citeproc /usr/lib/rstudio-server/bin/pandoc/pandoc-citeproc

Alternate R Markdown Templates

These alternative R Markdown templates look great.

ggplot2 extensions

Here is a list of ggplot2 extension packages.

Notebook interface for everything

Zeppelin provides a unified interface for nearly all the major data process engines, including Spark. I quickly set it up on a virtual machine and gave it a test run. It works great.

This one has SparkR support built-in.

I was never really into the ipython/jupyter notebook mainly because there is nothing they can do that a good IDE such as PyCharm or Rodeo cannot. Zeppelin is different because its capability of tightly integrating different Spark front-ends, including Scala, Python, and R is uniquely powerful. I would call this revolutionary.

R Users Will Now Inevitably Become Bayesians

Good post here. I would also add that the rethinking package is a third option that helps R users to become a Bayesian.