Diamonds - Part 3 - A polished gem - Building Non-linear Models
Other posts in this series:
- Diamonds - Part 1 - In the rough - An Exploratory Data Analysis
- Diamonds - Part 2 - A cut above - Building Linear Models
In the previous posts, we found out which attributes of a diamond are important for its price. An exploratory data analysis showed that carat, clarity and color are the most important features to predict price. Then we built linear models to predict the prices, and we chose the best model by its metrics.
In this post, we use non-linear models to predict the prices of diamonds, and we compare them with the linear models. A non-linear model can learn curved relationships and interactions between features by itself.
Training Non-linear Models
We use three types of non-linear models:
randomForestbuilds many decision trees and takes the average of their predictions. A decision tree predicts a value by asking a series of yes/no questions about the features, such as “Iscaratmore than 1?”.xgboost(gradient boosting) builds many small decision trees one after the other. Each new tree corrects some of the errors of the trees before it.kerasbuilds a neural network. A neural network passes the features through layers of simple calculations, and it learns the strength of each connection from the data.
We follow some of the same steps as for the linear models, and we change some features:
- Split the dataset into a training set with 75% of the diamonds and a testing set with 25%.
- Keep the same mix of
claritygrades in both sets (stratified sampling). clarity,colorandcuthave ordered categories, from the lowest grade to the highest grade.randomForestcan use these categories directly. Butxgboostandkerasneed all the features as numbers. There are two methods to change categories into numbers:- One-hot encoding: each category of
clarity,colorandcutbecomes a new feature with the values 0 and 1. A disadvantage of this method is that the order of the grades is lost. But the training set is large, so the non-linear models can learn the order by themselves. - Integer scores: the grades become the numbers 1, 2, 3 and so on, from the lowest grade to the highest. But this method assumes that the steps between all the grades are equal, which may not be correct here.
- One-hot encoding: each category of
- Center and scale all the values in the training set, so that all the features have similar ranges. Then put the features in a matrix (a table of numbers).
- Train a non-linear model on the training set.
- Make predictions on the testing set and calculate the metrics.
- Put all the steps above in a function that takes the model formula and a random seed. The seed controls the random split into the training set and the testing set.
- Run each model several times with different seeds, and calculate its average metrics. These averages show how well the model works on new data.
Here are the average metrics for all the models that we trained with keras, randomForest and xgboost:
load("fit_models_keras.Rda")
dfModels_keras <- dfModels %>% mutate(type = "keras")
best_model_results_keras <- best_model_results
dfResults_keras <- best_model_results_keras$dfResults %>% mutate(type = "keras")
load("fit_models_rf.Rda")
dfModels_rf <- dfModels %>% mutate(type = "randomForest")
best_model_results_rf <- best_model_results
dfResults_rf <- best_model_results_rf$dfResults %>% mutate(type = "randomForest")
load("fit_models_xgb.Rda")
dfModels_xgb <- dfModels %>% mutate(type = "xgboost")
best_model_results_xgb <- best_model_results
dfResults_xgb <- best_model_results_xgb$dfResults %>% mutate(type = "xgboost")
# combine the predicted v/s actual results from the best model using the 3 methods
dfResults <- map_df(list(dfResults_keras, dfResults_rf, dfResults_xgb), bind_rows) %>%
select(type, everything()) %>%
mutate_if(is.numeric, ~ round(.x, digits = 2)) %>%
group_by(type)
# combine the average metrics from all the models using the 3 methods
dfModels <- map_df(list(dfModels_keras, dfModels_rf, dfModels_xgb), bind_rows) %>%
select(type, everything()) %>%
mutate_if(is.numeric, ~ round(.x, digits = 2)) %>%
gather(avg_metric, value, rmse, rsq, mae)
# create an HTML table: one column group for each method, with its three average metrics
dfModels %>%
mutate(metric = factor(avg_metric, levels = c("rmse", "rsq", "mae"),
# "$" is a dollar sign: a literal "$" would start a math formula in pandoc
labels = c("RMSE ($)", "R-squared", "MAE ($)"))) %>%
arrange(type, metric) %>%
tidyHtmlTable(value = value, header = metric, cgroup = type, rnames = model,
align = "|rrr|rrr|rrr|")
| keras | randomForest | xgboost | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| RMSE ($) | R-squared | MAE ($) | RMSE ($) | R-squared | MAE ($) | RMSE ($) | R-squared | MAE ($) | |||
| price ~ . | 989.71 | 0.93 | 360.55 | 529.28 | 0.98 | 262.35 | 540.76 | 0.98 | 280.49 | ||
| price ~ carat | 1499.2 | 0.86 | 860.29 | 1427.25 | 0.87 | 816.1 | 1427.35 | 0.87 | 815.76 | ||
| price ~ carat + clarity | 1040.69 | 0.93 | 590.32 | 1006.61 | 0.94 | 548.67 | 992.46 | 0.94 | 544.48 | ||
| price ~ carat + clarity + color | 645.4 | 0.97 | 358.85 | 571.73 | 0.98 | 305.17 | 575.3 | 0.98 | 306.86 | ||
| price ~ carat + clarity + color + cut | 626.78 | 0.98 | 347.99 | 545.02 | 0.98 | 285.96 | 541.63 | 0.98 | 282.38 | ||
The R-squared values show that all the models learn the complex relationship between price and carat very well. For the linear models, we had to change both price and carat to a log scale. The neural network and the decision tree models learn this relationship by themselves. The root mean squared error (RMSE) is in dollars, so it is easy to understand. The average price of a diamond is about $4000, and the prices vary by about $4000 (the standard deviation). So the RMSE values of the models are very low. With carat, clarity and color, the RMSE is between $572 and $645. The best linear model with the same features has an RMSE of about $816.
The exploratory data analysis helps here, because the models with only carat, clarity and color give excellent results. If we add cut, the RMSE goes down only a little.
Even the base models with all the features (price ~ .) explain the variation in the prices very well, although some of their features repeat the same information. Decision trees and neural networks are not affected by such repeated information (multicollinearity) as much as linear models are. To find the most important features of these models, we can explain their predictions one at a time.
Local Interpretable Model-agnostic Explanations
LIME is a method to explain the predictions of a “black box” model, which is a model whose calculations are too complex for a person to follow. LIME explains one prediction at a time. It assumes that, close to a single diamond, the complex model behaves like a simple linear model. So LIME fits a simple model around that one diamond. The simple model then explains how the complex model makes its prediction for that diamond.
LIME uses these steps:
- Take one diamond, and make many copies of it with small random changes to its features.
- Measure how similar each copy is to the original diamond.
- Use the complex model to predict the price of each copy.
- Choose a small number (m) of features that best explain the predicted prices.
- Fit a simple model to the copies, with the m features. Give more weight to the copies that are more similar to the original diamond.
- Use the weights of the simple model to explain the prediction for the original diamond.
Here, we choose the 5 features that best explain the predictions for 6 random diamonds from the testing set.
# Plot Feature Importance by sample
plot_features(explanation_keras, ncol = 3) +
labs(title = "LIME Feature Importance from a keras model",
subtitle = glue::glue("{numSamples} random samples from testing set")) +
theme_quant() +
theme(plot.subtitle = element_text(hjust = 0.5))

plot_features(explanation_xgb, ncol = 3) +
labs(title = "LIME Feature Importance from an xgboost model",
subtitle = glue::glue("{numSamples} random samples from testing set")) +
theme_quant() +
theme(plot.subtitle = element_text(hjust = 0.5))

For these 6 random diamonds, the most important features for the predictions are carat, clarity, color, x and y.
# Plot Feature Importance Heatmap
plot_explanations(explanation_keras) +
labs(title = "LIME Feature Importance Heatmap from a keras model",
subtitle = glue::glue("{numSamples} random samples from testing set")) +
theme_quant() +
theme(plot.subtitle = element_text(hjust = 0.5))

plot_explanations(explanation_xgb) +
labs(title = "LIME Feature Importance Heatmap from an xgboost model",
subtitle = glue::glue("{numSamples} random samples from testing set")) +
theme_quant() +
theme(plot.subtitle = element_text(hjust = 0.5))

We know that x and y have a very strong correlation with carat, so they repeat the same information. That is why it is good practice to remove such repeated features before we train a model. For keras, the model with carat, clarity and color has the best metrics. For the decision tree models, the model with all the features is a little more accurate, but it needs many more features.
Predicted v/s Actual
Finally, here are the scatter plots of the predicted prices against the actual prices, for the best model of each type on the testing set:
p1 <- dfResults %>%
ungroup() %>%
rename(price = actual) %>%
eda_plot_predicted_vs_actual("price", "estimate", feature_model = "type", nLabel = 0,
labelsScale = scales::label_dollar(accuracy = 1)) +
labs(title = paste(formulaString_best, "(linear scale)")) +
theme(panel.spacing.x = unit(2, "lines"), plot.margin = margin(5, 25, 5, 5))
p2 <- dfResults %>%
ungroup() %>%
rename(price = actual) %>%
eda_plot_predicted_vs_actual("price", "estimate", feature_model = "type", nLabel = 0, xLog = T, yLog = T,
labelsScale = scales::label_dollar(accuracy = 1)) +
labs(title = paste(formulaString_best, "(log scale)")) +
theme(panel.spacing.x = unit(2, "lines"), plot.margin = margin(5, 25, 5, 5))
cowplot::plot_grid(plotlist = list(p1, p2), ncol = 1)

The scatter plots use both normal (linear) axes and log axes. The R-squared and RMSE values of the 3 methods are similar. But for expensive diamonds, the predictions of keras are more spread out than the predictions of the two decision tree methods. For cheap diamonds, the decision tree methods seem to predict the prices better, with less spread.
As with the linear models, the spread of the predicted prices increases with price. The decision tree methods keep their predictions close to the range of prices in the data. Their highest predictions are $18,340 (randomForest) and $19,118 (xgboost). The most expensive diamond in the testing set costs $18,823. A randomForest prediction is an average of prices from the training data, so it cannot be higher than those prices. xgboost adds up many small corrections, so its predictions can be only a little higher. A neural network, like a linear model, can predict prices far above the prices in the data (extrapolation). The highest price that keras predicts is $26,591. So the non-linear methods learn the relationships between price and the other features very well, and the decision tree methods also keep their predictions within a reasonable range.
Summary
- All 3 non-linear methods learn the complex relationship between
price,caratand the other features by themselves. They do not need feature engineering. - The exploratory data analysis helps us remove repeated features before we train the models. This makes the training faster. For
keras, it also gives much better metrics. kerasneural networks train the fastest, because they can use a GPU (a graphics processor that does many calculations at the same time).- Of the two decision tree methods,
xgboosttrains much faster thanrandomForest. randomForestandxgboostcan use several CPU cores at the same time. On a local computer, the amount of memory (RAM) is the only limit.