Ames Housing - Part 2 - Building Models

Mon, 26 Dec 2016 00:00:00 +0000

In a previous post in this series, we did an exploratory data analysis of the Ames Housing dataset.

In this post, we will build both linear and non-linear models and see how well they predict the SalePrice of properties.

Evaluation Criteria

Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed SalePrice will be our evaluation criteria. Taking the log ensures that errors in predicting expensive and cheap houses will affect the result equally.

Steps for Building Models

Here are the steps for building models and determining the best hyperparameter combinations by K-fold cross validation:

Partition the training dataset into model training and validation sets. Use stratified sampling such that each partition has a similar distribution of the target variable - SalePrice.
Define linear and non-linear models.
For each model, create a space-filling (maximum entropy) grid of hyperparameter combinations.
For each hyperparameter combination, fit a model on the training set and make predictions on the validation set. Repeat the process for all folds.
Determine root mean squared errors (RMSE) and choose the best hyperparameter combination that corresponds to the minimum RMSE.
Train each model with its best hyperparameter combination on the entire training set.
Calculate RMSE of the each finalized model on the testing set.
Finally, choose the best model that gives the least RMSE.

Partitioning Training Data

We split the training data into 5 folds. Within each fold, 80% of the data is used for training models and 20% for validating the predicted values against the actual values.

Let’s look at the distribution of the target variable across all folds:

cv_folds %>% 
  mutate(target = map(splits, function(Z) bind_rows(training(Z) %>% select(target) %>% mutate(type = "training"), 
                                                    testing(Z) %>% select(target) %>% mutate(type = "validation")))) %>% 
  select(-splits) %>% 
  unnest(cols = c(target)) %>% 
  ggplot(aes(x = target)) +
  geom_histogram(bins = 30, color = "white", fill = "#F37735") +
  labs(x = "log(SalePrice)") +
  facet_grid(type ~ id)

By using stratified sampling, we ensure that the training and validation distributions of the target variable are similar.

Linear Models

Ordinary Least Squares Regression

Before creating any new features or indulging in more complex modelling methods, we will cross validate a simple linear model on the training data to establish a benchmark. If more complex approaches do not have a significant improvement in the model validation metrics, then they are not worthwhile to be pursued.

model_spec_lm

Linear Regression Model Specification (regression)

Computational engine: lm

cv_results_lm

# A tibble: 1 × 1
  mean_rmse
      <dbl>
1     0.147

rmse_val <- cv_results_lm %>% pull(mean_rmse)

What’s notable?

After training a linear model on all predictors, we get an RMSE of 0.1468.
This is the simplest and fastest model with no hyperparameters to tune.

Regularized Linear Model

We will use glmnet that uses LASSO and Ridge Regression with regularization. We will do a grid search of the following hyperparameters that minimize RMSE:

penalty: The total amount of regularization in the model.
mixture: The proportion of L1 regularization in the model.

model_spec_glmnet

Linear Regression Model Specification (regression)

Main Arguments:
  penalty = tune()
  mixture = tune()

Computational engine: glmnet

Let’s take a look at the top 10 RMSE values and hyperparameter combinations:

cv_results_glmnet %>% head(10)

# A tibble: 10 × 3
    penalty mixture mean_rmse
      <dbl>   <dbl>     <dbl>
 1 1.46e- 2  0.297      0.128
 2 1.22e- 2  0.657      0.129
 3 1.22e- 3  0.490      0.133
 4 1.09e- 2  0.0439     0.133
 5 3.47e- 4  0.992      0.134
 6 8.18e- 4  0.0856     0.137
 7 1.15e- 4  0.798      0.138
 8 7.37e- 7  0.0221     0.138
 9 1.23e-10  0.333      0.138
10 2.07e- 9  0.950      0.138

What’s notable?

After hyperparameter tuning with cross validation, the best glmnet model gives an RMSE of 0.128 with penalty = 0.0146 and mixture = 0.2974.
It is a significant improvement over Ordinary Least Squares regression that had an RMSE of 0.1468.
glmnet cross validation is fast, taking about 13 seconds to execute.
But the presence of outliers can significantly affect its performance.

Here a plot of the glmnet hyperparameter grid along with the best hyperparameter combination:

eda_plot_scatter(param_grid_glmnet, "mixture", "penalty", alpha = 1, add_lm = F, yLog = T, yLabelsScale = scales::scientific) +
  geom_point(data = param_best_glmnet, aes(x = mixture, y = penalty), color = "red") +
  ggrepel::geom_label_repel(data = param_best_glmnet %>% map_df(round, digits = 4), 
                            aes(x = mixture, y = penalty, label = glue::glue("penalty: {penalty}, mixture:{mixture}")),
                            family = "Roboto Condensed")

Non-linear Models

Next, we will train a couple of tree-based algorithms, which are not very sensitive to outliers and skewed data.

ranger (Random Forest)

We use the ranger implementation of random forests, which gives the same results as the randomForest package but runs about 3 times faster. In each ensemble, we have 500 trees (random forest error levels off well before that on a dataset of this size) and do a grid search of the following hyperparameters:

mtry: The number of predictors to randomly sample at each split.
min_n: The minimum number of data points in a node required to further split the node.

model_spec_rf

Random Forest Model Specification (regression)

Main Arguments:
  mtry = tune()
  trees = 500
  min_n = tune()

Engine-Specific Arguments:
  num.threads = 1

Computational engine: ranger

Let’s take a look at the top 10 RMSE values and hyperparameter combinations:

cv_results_rf %>% head(10)

# A tibble: 10 × 3
   min_n  mtry mean_rmse
   <int> <int>     <dbl>
 1     2    84     0.134
 2     5   134     0.136
 3     9    36     0.136
 4    12   110     0.136
 5    15    55     0.136
 6    18   136     0.137
 7     8   199     0.138
 8    22    69     0.138
 9    19   179     0.139
10    29   123     0.139

What’s notable?

After cross validation, we get the best RMSE of 0.134 with mtry = 84 and min_n = 2.
This is no improvement in RMSE compared to glmnet, and ranger cross validation takes much longer to execute (about 371 seconds, versus 13 seconds for glmnet).

Here a plot of the ranger hyperparameter grid along with the best hyperparameter combination:

eda_plot_scatter(param_grid_rf, "min_n", "mtry", alpha = 1, add_lm = F) +
  geom_point(data = param_best_rf, aes(x = min_n, y = mtry), color = "red") +
  ggrepel::geom_label_repel(data = param_best_rf, aes(x = min_n, y = mtry, label = glue::glue("mtry: {mtry}, min_n:{min_n}")), family = "Roboto Condensed")

xgboost

In each ensemble we have 1000 trees and do a grid search of the following hyperparameters:

min_n: The minimum number of data points in a node required to further split the node.
tree_depth: The maximum depth or the number of splits of the tree.
learn_rate: The rate at which the boosting algorithm adapts from one iteration to another.

model_spec_xgboost

Boosted Tree Model Specification (regression)

Main Arguments:
  trees = 1000
  min_n = tune()
  tree_depth = tune()
  learn_rate = tune()

Engine-Specific Arguments:
  objective = reg:squarederror
  nthread = 1

Computational engine: xgboost

Let’s take a look at the top 10 RMSE values and hyperparameter combinations:

cv_results_xgboost %>% head(10)

# A tibble: 10 × 4
   min_n tree_depth learn_rate mean_rmse
   <int>      <int>      <dbl>     <dbl>
 1    33          2    0.0289      0.126
 2    23          4    0.0254      0.127
 3     3          2    0.0171      0.127
 4    24          1    0.190       0.128
 5    38          9    0.0270      0.130
 6     4          1    0.0872      0.131
 7    23          9    0.0405      0.131
 8    32          9    0.00605     0.131
 9    19         11    0.00482     0.131
10    11          5    0.0549      0.131

What’s notable?

After cross validation, we get the best RMSE of 0.126 with min_n = 33, tree_depth = 2 and learn_rate = 0.0289.
This is the best cross validated RMSE of all the models, slightly ahead of glmnet (0.128) and ranger (0.134).
xgboost cross validation takes longer to execute than that of glmnet, but is faster than that of ranger (about 168 seconds).

Finalizing Models

For each model, we found the combination of hyperparameters that minimize RMSE. Using those parameters, we can now train the same models on the entire training dataset. Finally, we can use the trained models to predict log(SalePrice) on the entire training set to see the actual v/s predicted log(SalePrice) results.

# Get predicted estimates
get_predicted <- function(model_final, dfNew) {

  # Predict on testing set
  vEstimates <- model_final %>%
    predict(new_data = dfNew) %>%
    pull(.pred)
  
  return(vEstimates)
}

dfResults_train <- dfTrain %>% 
  mutate(actual = log(SalePrice)) %>% 
  select(Id, actual) %>% 
  mutate(glmnet = get_predicted(model_final = model_final_glmnet, dfTrain_Proc),
         ranger = get_predicted(model_final = model_final_rf, dfTrain_Proc),
         xgboost = get_predicted(model_final = model_final_xgboost, dfTrain_Proc))

dfPlot <- dfResults_train %>% 
  gather(model, predicted, -actual, -Id) %>% 
  mutate(model = ordered(model, levels = c("glmnet","ranger","xgboost")))

dfPlot %>%
  rename(`log(SalePrice)` = actual) %>%
  eda_plot_actual_vs_predicted("log(SalePrice)", "predicted", feature_model = "model", feature_id = "Id", nLabel = 3) +
  labs(title = "Actual v/s predicted log(SalePrice) on training set")

# properties among the 3 largest errors of every model
vWorst_train <- dfPlot %>% 
  group_by(model) %>% 
  slice_max(abs(predicted - actual), n = 3) %>% 
  ungroup() %>% 
  count(Id) %>% 
  filter(n == n_distinct(dfPlot$model)) %>% 
  pull(Id)

What’s notable?

Both ranger and xgboost models fit the training data closely with the tuned parameters, as the predictions lie close to the straight line drawn at 45 degrees. ranger fits it most closely, with a training RMSE of 0.051 versus 0.094 for xgboost.
The glmnet model shows a couple of outliers with Ids 524 and 1299 whose predicted values are far in excess of their actual values. Even properties whose SalePrice is at the lower end, show a wide dispersion in prediced values.
The plots label the three largest errors of each model. The property with Id 633 is among them for all three models, so it is worth checking for errors in its data.
But the true performance can only be measured on unseen testing data.

Performance on Test Data

dfResults_test <- dfTest %>% 
  mutate(actual = log(SalePrice)) %>% 
  select(Id, actual) %>% 
  mutate(glmnet = get_predicted(model_final = model_final_glmnet, dfTest_Proc),
         ranger = get_predicted(model_final = model_final_rf, dfTest_Proc),
         xgboost = get_predicted(model_final = model_final_xgboost, dfTest_Proc))

dfPlot <- dfResults_test %>% 
  gather(model, predicted, -actual, -Id) %>% 
  mutate(model = ordered(model, levels = c("glmnet","ranger","xgboost")))

dfPlot %>%
  rename(`log(SalePrice)` = actual) %>%
  eda_plot_actual_vs_predicted("log(SalePrice)", "predicted", feature_model = "model", feature_id = "Id", nLabel = 3) +
  labs(title = "Actual v/s predicted log(SalePrice) on testing set")

dfRMSE <- map(list("glmnet","ranger","xgboost"), ~ rmse(dfResults_test, actual, !!sym(.x)) %>% mutate(model = .x)) %>%
  bind_rows() %>%
  mutate(cv_rmse = c(cv_results_glmnet$mean_rmse[1], cv_results_rf$mean_rmse[1], cv_results_xgboost$mean_rmse[1])) %>%
  select(-.estimator, -.metric) %>%
  rename(test_rmse = .estimate) %>%
  select(model, everything())

dfRMSE

# A tibble: 3 × 3
  model   test_rmse cv_rmse
  <chr>       <dbl>   <dbl>
1 glmnet      0.130   0.128
2 ranger      0.138   0.134
3 xgboost     0.131   0.126

What’s notable?

All models have similar RMSE on the unseen testing set as their cross validated RMSE, which shows the cross validation process and hyperparameters worked very well.
Records with Ids 1537 and 2217 are outliers, as none of the models are able to predict close to actual values.
glmnet has the lowest test RMSE of 0.130, with xgboost close behind at 0.131.
However, glmnet is sensitive to outliers, as we saw with Ids 524 and 1299 in the training set. Since xgboost had the best cross validated RMSE and the difference in test RMSE is small, we could finalize xgboost, which is robust to outliers and captures non-linear relationships, as the model that generalizes well on this dataset.

model_final <- model_final_xgboost

Feature Importance

Even though xgboost is not as easily interpretable as a linear model, we could use variable importance plots to determine the most important features selected by the model.

Let’s take a look at the top 10 most important features of our finalized xgboost model:

library(vip)

p1 <- vip(extract_fit_parsnip(model_final), aesthetics = list(fill = "#00AED8")) +
  labs(title = "xgboost model Feature Importance in Training") +
  theme_quant()

vFeaturesImp <- vip::vi(extract_fit_parsnip(model_final)) %>% top_n(10, Importance) %>% pull(Variable)

p2 <- eda_plot_correlation_matrix(dfTrain %>% select(SalePrice, one_of(vFeaturesImp)), 
                            outlineVar = "SalePrice", plotTitle = "Correlations with SalePrice")

cowplot::plot_grid(plotlist = list(p1, p2), nrow = 1)

Correlations of numerical features are plotted side-by-side. All of them have a correlation of 0.5 or more (in absolute terms) with SalePrice.
All of the top 10 features make sense. To evaluate SalePrice, a buyer would definitely look at total square footage, overall quality, neighborhood, number of bathrooms, kitchen quality, age of property, garage size and finish, central air conditioning, fireplace quality, etc.
This shows, our finalized model generalizes well and makes very reasonable choices in terms of features.

New Property Premium

Among the top 10 features by importance in our final model, most of the features like square footage, neighborhood and number of bathrooms remain the same throughout the life of the property. Quality and condition of property does change but their evaluation is mostly subjective. The only other feature that cannot be disputed to change over time is PropertyAge.

So, how would the predicted SalePrice differ if a property was newly constructed vis-a-vis the same property if it were constructed more than 30 years earlier, and all the times in between?

We could pick a couple of properties at random, change PropertyAge and see its impact on SalePrice.

getSampleProperty <- function(randomSeed) {
  
  set.seed(randomSeed)
  dfSample <- dfTest %>% 
    filter(PropertyAge >= 30) %>% 
    sample_n(1)
  
  dfSample <- tibble(PropertyAge = 1:dfSample$PropertyAge[1]) %>% expand_grid(dfSample %>% select(-PropertyAge))
}

dfSample <- getSampleProperty(1000) %>% bind_rows(getSampleProperty(2000))

# preprocess the sample data and generated predicted values from the final model
get_preprocessing_recipe(dfSample) %>% 
  bake(dfSample) %>% 
  get_predicted(model_final, .) %>% 
  as_tibble() %>% 
  bind_cols(dfSample) %>% 
  mutate(predicted = exp(value)) %>% 
  group_by(Id) %>% 
  ggplot(aes(x = PropertyAge, y = predicted)) +
  geom_point(color = "#00AED8") +
  geom_line() +
  facet_wrap(~Id, scales = "free") +
  scale_y_continuous(labels = scales::label_dollar(accuracy = 1)) +
  labs(y = "Predicted Sale Price",
       title = "New Property Premium")

We can see there’s a small premium for a newly constructed property v/s an older property of the same build, quality and condition. This premium isn’t very much in a place like Ames, IA but we’d reckon it would be much higher in a larger metropolitan city.

Ranger | Nitin Gupta

Ames Housing - Part 2 - Building Models

Evaluation Criteria

Steps for Building Models

Partitioning Training Data

Linear Models

Ordinary Least Squares Regression

What’s notable?

Regularized Linear Model

What’s notable?

Non-linear Models

ranger (Random Forest)

What’s notable?

xgboost

What’s notable?

Finalizing Models

What’s notable?

Performance on Test Data

What’s notable?

Feature Importance

New Property Premium