Pearson’s Rho (aka Pearson’s Product-Moment Correlation Coefficient)

Pearson’s coefficient is the default and most commonly used, but its application is limited to situations where a specific set of assumptions is met. A detailed list can be found here but the main ones are: - Both variables are measured on a continuous scale - Both variables are normally distributed - Observations are independent - Variables are linearly related

The last assumption might seem odd - isn’t the whole purpose of correlation coefficients to tell us whether there is a linear correlation??? The answer is no. You must always visually inspect your data first. If you see a linear relationship, you can then quantify it using a correlation coefficient. Note, also, that the variables must be linearly correlated. The emphasis on “linearly” correlated means that if your variables have a parabolic, exponential, sinusoidal, etc. relationship, a correlation coefficient is about as useful as a glass hammer.

The population correlation between two random variables X and Y is defined as: ρXY=Cov(X,Y)σXσY

and the sample correlation is defined as: rxy=Σni=1(xi−ˉx)(yi−ˉy)√Σni=1(xi−ˉx)2√Σni=1(yi−ˉy)2

Spearman’s Rho (aka Spearman’s Rank Correlation Coefficient)

In contrast to Pearson’s rho, Spearman’s rho is a non-parametric statistic, meaning it can be applied to non normally-distributed data (Puth, Neuhäuser, & Ruxton, 2015).

Spearman’s coefficient also does not stipulate that the two variables must be linearly correlated - rather, they must have a monotonic relationship, meaning that when one variable increases, so does the other (not necessarily in a linear fashion.)² This picture is all you need to understand monotonicity:

Finally, Spearman’s coefficient does not require that the variables be continuous, so you can use categorical variables, as long as they are ordinal.

The trade-off for this flexibility is that the Spearman coefficient is slightly less interpretable. Essentially, we are not measuring the direct correlation using our exact data values. Instead, we are correlating the rank of each value within one variable to the rank of each value in another.

For example, here are the top 6 values of the healthy_life_expectancy vector:

c(76.80458, 76.77171, 75.00097, 74.40271, 74.10245, 73.80193)

## [1] 76.80458 76.77171 75.00097 74.40271 74.10245 73.80193

When we compute Spearman’s coefficient, these values are instead ranked, so Spearman’s test would interpret 76.80 as 1, 76.77 as 2, and so on, and then correlate these ranks to the ranks of the second variable.

Because of this adjustment, Spearman’s coefficient is completely tolerant to outliers. But you have to bear in mind that when you compute and interpret a Spearman coefficient, you’re dealing not with direct correlation between two variables but the rank correlation, so ensure you’re interpreting accordingly.

The formula for computing Spearman’s coefficient is the same as Pearson’s, but rather than dealing with X and Y, we’re dealing with the rank of X and Y:

ρrankX,rankY=Cov(rankX,rankY)σrankXσrankY

If there are no ties, Spearman’s coefficient is computed using this formula:

rs=1−6∗Σd2in(n2−1) where d=rankxi−rankyi

Kendall’s Tau (aka Kendall’s Rank Correlation Coefficient)

Kendall’s tau is another rank-based, non-parametric statistic. It follows the same assumptions as Spearman’s rank-order correlation coefficient, but it produces more robust standard errors and is therefore [thought]((https://statistics.laerd.com/spss-tutorials/kendalls-tau-b-using-spss-statistics.php) to be more suitable for small sample sizes.

The calculation for Kendall’s Tau is actually pretty simple. You (or your software) will simply rank all the x’s and all the y’s in order, pair them, and count the number of concordant vs discordant pairs. Concordant pairs are ones where xi>xi+1 and yi>yi+1. Discordant pairs are ones where one of those conditions is not met. \tau = \frac {number\,concordant\,pairs - number\,discordant\,pairs} {\binom n 2}

The trade-off for the increased precision we get with Kendall’s tau is that the estimated correlation coefficients are generally smaller in magnitude than those computed using Spearman’s rho (Fredricks & Nelsen, 2007).

Hypothesis Testing

Presumably, you want to use the correlation coefficient to make some inferences about the population of interest. To do so, you need to conduct a hypothesis test and report either a p-value or a confidence interval. We will look at several functions for doing this in R below, but all have the following null and alternative hypotheses:

H_0: \rho = 0 there is no linear (or monotonic, in the case of Spearman and Kendall) association between the two variables in the population H_a: \rho \ne 0 there is a linear/monotonic association between the two variables in the population.

You can also conduct a one-sided test, where the alternative hypothesis would be one of the following: H_a: \rho > 0 H_a: \rho < 0

Data Import

The Kaggle dataset loads in a zip file containing a separate .csv for every year from 2015 to 2020. We will use the 2020 data here.

happy_df = read.csv(file = "../data/2020.csv") %>% 
  janitor::clean_names() 

head(happy_df)

##   country_name regional_indicator ladder_score standard_error_of_ladder_score
## 1      Finland     Western Europe       7.8087                     0.03115630
## 2      Denmark     Western Europe       7.6456                     0.03349229
## 3  Switzerland     Western Europe       7.5599                     0.03501417
## 4      Iceland     Western Europe       7.5045                     0.05961586
## 5       Norway     Western Europe       7.4880                     0.03483738
## 6  Netherlands     Western Europe       7.4489                     0.02779175
##   upperwhisker lowerwhisker logged_gdp_per_capita social_support
## 1     7.869766     7.747634              10.63927      0.9543297
## 2     7.711245     7.579955              10.77400      0.9559908
## 3     7.628528     7.491272              10.97993      0.9428466
## 4     7.621347     7.387653              10.77256      0.9746696
## 5     7.556281     7.419719              11.08780      0.9524866
## 6     7.503372     7.394428              10.81271      0.9391388
##   healthy_life_expectancy freedom_to_make_life_choices  generosity
## 1                71.90083                    0.9491722 -0.05948202
## 2                72.40250                    0.9514443  0.06620178
## 3                74.10245                    0.9213367  0.10591104
## 4                73.00000                    0.9488919  0.24694422
## 5                73.20078                    0.9557503  0.13453263
## 6                72.30092                    0.9085478  0.20761244
##   perceptions_of_corruption ladder_score_in_dystopia
## 1                 0.1954446                 1.972317
## 2                 0.1684895                 1.972317
## 3                 0.3037284                 1.972317
## 4                 0.7117097                 1.972317
## 5                 0.2632182                 1.972317
## 6                 0.3647171                 1.972317
##   explained_by_log_gdp_per_capita explained_by_social_support
## 1                        1.285190                    1.499526
## 2                        1.326949                    1.503449
## 3                        1.390774                    1.472403
## 4                        1.326502                    1.547567
## 5                        1.424207                    1.495173
## 6                        1.338946                    1.463646
##   explained_by_healthy_life_expectancy
## 1                            0.9612714
## 2                            0.9793326
## 3                            1.0405332
## 4                            1.0008434
## 5                            1.0080719
## 6                            0.9756753
##   explained_by_freedom_to_make_life_choices explained_by_generosity
## 1                                 0.6623167               0.1596704
## 2                                 0.6650399               0.2427934
## 3                                 0.6289545               0.2690558
## 4                                 0.6619807               0.3623302
## 5                                 0.6702009               0.2879851
## 6                                 0.6136265               0.3363176
##   explained_by_perceptions_of_corruption dystopia_residual
## 1                              0.4778573          2.762835
## 2                              0.4952603          2.432741
## 3                              0.4079459          2.350267
## 4                              0.1445408          2.460688
## 5                              0.4341006          2.168266
## 6                              0.3685698          2.352117

cor

There are several options in R for computing correlation coefficients. The first is base R’s cor function, which allows selection of Pearson, Spearman, or Kendall using a method argument. We will first specify Spearman based on our conclusions above.

spearman = cor(happy_df$ladder_score, happy_df[, c(8:12)], method = "spearman")
spearman

##      social_support healthy_life_expectancy freedom_to_make_life_choices
## [1,]      0.8031141               0.7830361                    0.5919578
##      generosity perceptions_of_corruption
## [1,] 0.07451114                -0.2774517

Happiness has a strong positive correlation with social support (r = 0.80), a strong positive correlation with life expectancy (r = 0.78), and a moderate positive correlation with freedom to make life choices (r = 0.59). Happiness has no correlation with generosity (r = 0.07), and a weak negative correlation with perceptions of corruption (r = -0.28).

Now, let’s compare to Kendall:

kendall = cor(happy_df$ladder_score, happy_df[, c(8:12)], method = "kendall")
kendall

##      social_support healthy_life_expectancy freedom_to_make_life_choices
## [1,]      0.6081871               0.5775102                    0.4248366
##      generosity perceptions_of_corruption
## [1,] 0.04317165                -0.1854145

Just for demonstration, let’s also compute the Pearson coefficients and compare them to Spearman’s rho and Kendall’s tau.

pearson = cor(happy_df$ladder_score, happy_df[, c(8:12)], method = "pearson")
pearson

##      social_support healthy_life_expectancy freedom_to_make_life_choices
## [1,]      0.7650008               0.7703163                    0.5905968
##      generosity perceptions_of_corruption
## [1,] 0.06904313                -0.4183051

Putting everything in a quick table:

pearson = as.data.frame(pearson, row.names = "pearson")
spearman = as.data.frame(spearman, row.names = "spearman")
kendall = as.data.frame(kendall, row.names = "kendall")

knitr::kable(bind_rows(pearson, spearman, kendall))

Comparing Pearson vs Spearman, the only big difference with respect to happiness is perception of corruption (Pearson -0.42 vs Spearman -0.28). This isn’t surprising, given that perception_of_corruption appeared to be the most skewed (i.e. the most deviant from normal distribution) in the chart.Correlation matrix we looked at above.

Comparing Spearman to Kendall, the magnitude of our coefficients dropped to the null pretty sizably, which aligns with the general behavior of Kendall’s tau vs Spearman’s rho (Fredricks & Nelsen, 2007).

cor.test and rcorr

While cor is quick and easy, it does not compute any statistical significance measures. For that, we need cor.test (base) or rcorr (HMisc package).

The disadvantage of cor.test is that it only computes one coefficient at a time, meaning you can’t create a matrix in one line. rcorr, on the other hand, will give you a matrix of correlation coefficients and their associated p-values. The disadvantages are a) it doesn’t compute Kendall’s tau and b) you won’t get confidence intervals.

When we run cor.test, we see that it actually doesn’t give a confidence interval for Spearman coefficients either (although it does for Pearson):

cor.test(happy_df$ladder_score, happy_df$social_support, method = "spearman", conf.level = 0.95)

## 
##  Spearman's rank correlation rho
## 
## data:  happy_df$ladder_score and happy_df$social_support
## S = 117522, p-value < 2.2e-16
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##       rho 
## 0.8031141

cor.test(happy_df$ladder_score, happy_df$social_support, method = "pearson", conf.level = 0.95)

## 
##  Pearson's product-moment correlation
## 
## data:  happy_df$ladder_score and happy_df$social_support
## t = 14.596, df = 151, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.6900921 0.8236918
## sample estimates:
##       cor 
## 0.7650008

cor.test(happy_df$ladder_score, happy_df$social_support, method = "kendall", conf.level = 0.95)

## 
##  Kendall's rank correlation tau
## 
## data:  happy_df$ladder_score and happy_df$social_support
## z = 11.157, p-value < 2.2e-16
## alternative hypothesis: true tau is not equal to 0
## sample estimates:
##       tau 
## 0.6081871

So basically, in the context of this analysis, cor.test is weak sauce. Let’s move on to rcorr, looking only at Spearman. rcorr does not play with Kendall and won’t give you confidence intervals for any type of coefficient. But unlike cor.test, rcorr will give you a matrix of coefficients and a matrix of p-values:

happy_df_corr = 
  happy_df %>% 
  select(c(3,8:12)) 

happy_df_corr = as.matrix(happy_df_corr)
rcorr(happy_df_corr, type = "spearman")

##                              ladder_score social_support
## ladder_score                         1.00           0.80
## social_support                       0.80           1.00
## healthy_life_expectancy              0.78           0.73
## freedom_to_make_life_choices         0.59           0.49
## generosity                           0.07           0.00
## perceptions_of_corruption           -0.28          -0.14
##                              healthy_life_expectancy
## ladder_score                                    0.78
## social_support                                  0.73
## healthy_life_expectancy                         1.00
## freedom_to_make_life_choices                    0.45
## generosity                                     -0.04
## perceptions_of_corruption                      -0.23
##                              freedom_to_make_life_choices generosity
## ladder_score                                         0.59       0.07
## social_support                                       0.49       0.00
## healthy_life_expectancy                              0.45      -0.04
## freedom_to_make_life_choices                         1.00       0.30
## generosity                                           0.30       1.00
## perceptions_of_corruption                           -0.37      -0.25
##                              perceptions_of_corruption
## ladder_score                                     -0.28
## social_support                                   -0.14
## healthy_life_expectancy                          -0.23
## freedom_to_make_life_choices                     -0.37
## generosity                                       -0.25
## perceptions_of_corruption                         1.00
## 
## n= 153 
## 
## 
## P
##                              ladder_score social_support
## ladder_score                              0.0000        
## social_support               0.0000                     
## healthy_life_expectancy      0.0000       0.0000        
## freedom_to_make_life_choices 0.0000       0.0000        
## generosity                   0.3600       0.9617        
## perceptions_of_corruption    0.0005       0.0781        
##                              healthy_life_expectancy
## ladder_score                 0.0000                 
## social_support               0.0000                 
## healthy_life_expectancy                             
## freedom_to_make_life_choices 0.0000                 
## generosity                   0.6244                 
## perceptions_of_corruption    0.0051                 
##                              freedom_to_make_life_choices generosity
## ladder_score                 0.0000                       0.3600    
## social_support               0.0000                       0.9617    
## healthy_life_expectancy      0.0000                       0.6244    
## freedom_to_make_life_choices                              0.0002    
## generosity                   0.0002                                 
## perceptions_of_corruption    0.0000                       0.0022    
##                              perceptions_of_corruption
## ladder_score                 0.0005                   
## social_support               0.0781                   
## healthy_life_expectancy      0.0051                   
## freedom_to_make_life_choices 0.0000                   
## generosity                   0.0022                   
## perceptions_of_corruption

So we’ve re-computed the Spearman coefficients from cor and we’ve learned that, with respect to happiness, all correlations except ladder_score vs generoisty are statistically significant at an alpha level of 0.05.

We’ve also learned that all of the correlations functions we computed have gaps and inconveniences. Most likely, if you’re doing a correlation analysis in R, you’ll need to use some combination of these functions.

Here’s a quick summary of the capabilities of cor vs cor.test vs rcorr: