Diamonds - Part 1 - In the rough - An Exploratory Data Analysis

Numerical Features

eda_plot_histograms(dfInput, vFeaturesNum)

The plots show outliers in each feature. Let’s remove the outliers and plot the features again. We use a common rule: a value is an outlier if it is more than 2.5 interquartile ranges from the median. The median is the middle value: half the values are below it and half are above it. The interquartile range is the width of the middle half of the values. The outliers themselves have almost no effect on these two numbers, so the rule is reliable. A rule that uses the average would be less reliable, because the outliers pull the average towards them.

makeOutliersNA <- function(Z) {
  # values more than 2.5 interquartile ranges from the median become NA
  ifelse(flag_outliers(Z, nIQR = 2.5), NA, Z)
}

# convert outliers to NA values
dfInput <- dfInput %>% mutate_at(vFeaturesNum, makeOutliersNA)

eda_plot_histograms(dfInput, vFeaturesNum) + labs(title = "Histograms of numerical features excluding outliers")

Without the outliers, the ranges of the values are more reasonable. carat and price are right skewed: most values are small, and a few values are much larger.

Let’s plot price on a log scale. On a log scale, each step on the axis multiplies the value by the same amount (for example, from $100 to $1,000 to $10,000). This makes it easier to see the spread of skewed values:

eda_plot_histograms(dfInput, "price", nBins = 100, xLog = T, xLabelsScale = scales::label_dollar(accuracy = 1))

The plot has two peaks. This means that two price ranges are the most common: one just below USD 1000 and one near USD 5000. Interestingly, almost no diamonds cost about USD 1500. So the plot shows a big gap at that price.

Categorical Features

cowplot::plot_grid(plotlist = map(vFeaturesCat, ~ eda_plot_column(dfInput, .x)), ncol = 2)

The plots clearly show that the categories of cut and clarity are not balanced.

Numerical-numerical

First, we measure the correlation of price with each numerical feature. A correlation is a number from -1 to 1. It tells how closely two features move together in a straight line. A value near 1 means that when one feature goes up, the other also goes up. A value near -1 means that the other goes down. A value near 0 means that there is no straight-line relationship.

eda_plot_correlation_matrix(dfInput[vFeaturesNum] %>% drop_na(), outlineVar = "price", plotTitle = "Clustered correlations of numerical features")

price has a very strong correlation with carat and with the dimensions x, y and z. These four features are also strongly correlated with each other, because they all measure the size of a diamond. So a linear model cannot easily separate their effects. table and depth have almost no correlation with price. So they are not useful to predict the price.

Now let’s look at the scatter plots. A scatter plot shows one dot for each diamond:

eda_plot_scatter(dfInput, vFeaturesNum, "price", yLimits = c(200, 15000), yLabelsScale = scales::label_dollar(accuracy = 1))

Without the outliers, price grows faster and faster as carat, x, y and z increase. So let’s plot price on a log scale:

eda_plot_scatter(dfInput, vFeaturesNum, "price", yLog = T, yLimits = c(200, 15000), yLabelsScale = scales::label_dollar(accuracy = 1))

On a log scale, log(price) has an approximately straight-line (linear) relationship with x, y and z. But its relationship with carat is still curved. Also, the spread (variance) of price increases as the diamonds get larger.

Can we change carat with a simple formula so that its relationship with log(price) becomes a straight line? We call such a change a transform. Let’s compare some common transforms. Each panel shows the R-squared of a straight line through the points. R-squared is a number from 0 to 1. It tells how much of the variation in log(price) the straight line explains. A higher value means that the line fits the points better:

p_transforms <- eda_plot_transforms(dfInput, "carat", "price")
p_transforms

# R-squared of each transform, in the order of the panels
vR2_carat <- p_transforms$data %>%
  distinct(transform, r_squared) %>%
  pull(r_squared) %>%
  set_names(c("identity", "log", "sqrt", "cuberoot"))

log(carat) gives the best straight-line fit, with an R-squared of 0.926. The cube root is second (0.919) and the square root is third (0.91). carat without a transform has the lowest R-squared, 0.862. So log(price) has an almost straight-line relationship with log(carat). This means that when the weight increases by a given percentage, the price increases by an approximately constant percentage. We will use this result when we create features for the models (Feature Engineering).

Numerical-Categorical

Let’s look at price for each category of the categorical features. A box plot shows the middle half of the prices as a box, with a line at the median:

cowplot::plot_grid(plotlist = map(vFeaturesCat, ~ eda_plot_box(dfInput, .x, "price", yLimits = c(300, 8000), 
                                                               yLabelsScale = scales::label_dollar(accuracy = 1))), ncol = 3)

To show the trends better, the price axis of these box plots does not show the most expensive diamonds. The box plots are surprising: for cut, color and clarity, the median price usually goes down as the grade gets better. This is very strange. We will find the reason in the multivariate analysis.

• The median price goes down steadily from a Fair cut to an Ideal cut.
• For color, the median price goes down from J (worst) to G (middle grade). Then it goes up, and it goes down again for D (best).
• For clarity, the median price goes up from I1 to SI2. Then it goes down steadily to IF.

Numerical-Numerical-Categorical

cowplot::plot_grid(plotlist = map(vFeaturesCat, ~ eda_plot_scatter_cat(dfInput, "carat", "price", .x, yLabelsScale = scales::dollar)), ncol = 2)

The dots overlap a lot. But for diamonds of the same carat weight, price clearly goes up as clarity gets better. color shows the same pattern, but it is weaker. cut does not show any effect.

So, for diamonds of the same weight, both color and clarity explain some of the differences in price.

All Features

Why do the diamonds with better grades seem cheaper in the bivariate analysis? To find out, we measure the association between every pair of features. An association tells how strongly two features are related. Here, we rank the grades of cut, color and clarity from worst to best. Then we can compare them with the numerical features with Spearman’s rank correlation. This correlation uses the ranks of the values instead of the values themselves, so it also works for grades:

eda_plot_association_matrix(dfInput, method = "spearman", ordinalAsNumeric = T, outlineVar = "price",
                            plotTitle = "Spearman correlations, with grades ranked from worst to best")

mAssoc <- association_matrix(dfInput, method = "spearman", ordinalAsNumeric = T)

• carat has a negative correlation with clarity (-0.37) and with color (-0.24). This means that diamonds with better grades are usually smaller.
• price has a very strong correlation with carat (0.96). So price also has a negative correlation with clarity (-0.21) and with color (-0.13).
• carat, x, y and z form a tight group, and they are almost perfectly correlated with each other. So, to predict the price, these features give almost the same information.

Categorical-Numerical-Numerical

In the scatter plots, the dots overlap too much to show how much each grade matters. So let’s compare grades among diamonds of similar weight. We split the diamonds into four groups (bins) of equal size by carat. Then we plot price for each grade in each group:

lBinPlots <- map(c("clarity", "color", "cut"), ~ eda_plot_box_by_bins(dfInput, .x, "price", "carat", yLog = T,
                                                                       yLabelsScale = scales::label_dollar(accuracy = 1)))
cowplot::plot_grid(plotlist = lBinPlots, ncol = 1)

# median price by grade within each carat bin
get_bin_medians <- function(p) {
  p$data %>%
    group_by(bin = as.integer(bin), grade = as.character(x)) %>%
    summarise(median = median(y), .groups = "drop")
}
dfBinMedians_clarity <- get_bin_medians(lBinPlots[[1]])
bin_median <- function(dfMedians, binNum, gradeName) {
  scales::dollar(dfMedians$median[dfMedians$bin == binNum & dfMedians$grade == gradeName], accuracy = 1)
}

# median carat by clarity
dfCarat_clarity <- dfInput %>% group_by(clarity) %>% summarise(carat = median(carat, na.rm = T))

These plots explain the strange box plots of the bivariate analysis:

• In each carat group, the median price usually goes up as clarity gets better. The effect is clearest for the heaviest diamonds: the median price goes from $3,985 for I1 to $10,332 for IF. The prices of the best grades (VVS2 to IF) are close to each other, and they are not always in order.
• In each group, price also goes up as color gets better from J to G. But the best grades (G to D) have about the same price. For the heaviest diamonds, D and E are even a little cheaper than G.
• cut does not show a clear trend, even for diamonds of similar weight.

Diamonds with better grades seem cheaper overall because they are usually smaller. The median weight of an I1 diamond is 1.07 carat. The median weight of an IF diamond is only 0.35 carat. So carat is a confounder: it is related to both price and the grades, and it hides the true effect of the grades. To measure the effect of clarity or color, we must compare diamonds of similar weight.

Next, let’s check whether table and depth affect price together with color and clarity:

dfFeatures <- expand_grid(feat1 = c("depth","table"), feat2 = c("color","clarity")) %>% arrange(desc(feat2))
cowplot::plot_grid(plotlist = map2(dfFeatures$feat1, dfFeatures$feat2, 
                                   ~ eda_plot_scatter_cat(dfInput, .x, "price", .y, yLabelsScale = scales::dollar)), nrow = 2)

These plots do not show any pattern. So table and depth do not help to predict price.

Categorical-Categorical-Numerical

Does clarity together with cut or color tell us more about price? A heatmap answers this question. It shows a grid of tiles, one for each combination of two categories. The color of each tile shows the median price of the diamonds in that combination:

cowplot::plot_grid(plotlist = map(c("cut","color"), ~ eda_plot_heatmap(dfInput, .x, "clarity", "price")), nrow = 2)

The second heatmap is more interesting. From the bottom left to the top right, the grades of color and clarity get better, and the median price usually goes down. Again, this is the opposite of what we expect, because the diamonds with the best color and clarity should be the most expensive. As we saw above, the reason is that diamonds with better grades are usually smaller.

For cut and clarity, the median prices do not show a clear pattern. Each tile shows its number of diamonds. One combination (Fair cut and IF clarity) has fewer than 10 diamonds. Its tile is grey, because a median of so few diamonds is not reliable.