library(tidyverse)
library(mosaic)
library(ggformula)
library(skimr)
library(janitor) # Data cleaning and tidying package
library(visdat) # Visualize whole dataframes for missing data
library(naniar) # Clean missing data
library(DT) # Interactive Tables for our data
library(tinytable) # Elegant Tables for our data
library(ggrepel) # Repel overlapping text labels in ggplot
library(marquee) # Annotations in ggplot
Outliers and other Crazy Things
Qual Variables
Quant Variables
Box Plots
Violin Plots
Abstract
Quant and Qual Variable Graphs and their Siblings
“In keeping silent about evil, in burying it so deep within us that no sign of it appears on the surface, we are implanting it, and it will rise up a thousand fold in the future.”
— Aleksandr Solzhenitsyn
1
Plot Fonts and Theme
Show the Code
library(systemfonts)
library(showtext)
## Clean the slate
systemfonts::clear_local_fonts()
systemfonts::clear_registry()
##
showtext_opts(dpi = 96) # set DPI for showtext
sysfonts::font_add(
family = "Alegreya",
regular = "../../../../../../fonts/Alegreya-Regular.ttf",
bold = "../../../../../../fonts/Alegreya-Bold.ttf",
italic = "../../../../../../fonts/Alegreya-Italic.ttf",
bolditalic = "../../../../../../fonts/Alegreya-BoldItalic.ttf"
)
sysfonts::font_add(
family = "Roboto Condensed",
regular = "../../../../../../fonts/RobotoCondensed-Regular.ttf",
bold = "../../../../../../fonts/RobotoCondensed-Bold.ttf",
italic = "../../../../../../fonts/RobotoCondensed-Italic.ttf",
bolditalic = "../../../../../../fonts/RobotoCondensed-BoldItalic.ttf"
)
showtext_auto(enable = TRUE) # enable showtext
##
theme_custom <- function() {
theme_bw(base_size = 10) +
theme_sub_axis(
title = element_text(
family = "Roboto Condensed",
size = 8
),
text = element_text(
family = "Roboto Condensed",
size = 6
)
) +
theme_sub_legend(
text = element_text(
family = "Roboto Condensed",
size = 6
),
title = element_text(
family = "Alegreya",
size = 8
)
) +
theme_sub_plot(
title = element_text(
family = "Alegreya",
size = 14, face = "bold"
),
title.position = "plot",
subtitle = element_text(
family = "Alegreya",
size = 10
),
caption = element_text(
family = "Alegreya",
size = 6
),
caption.position = "plot"
)
}
## Use available fonts in ggplot text geoms too!
ggplot2::update_geom_defaults(geom = "text", new = list(
family = "Roboto Condensed",
face = "plain",
size = 3.5,
color = "#2b2b2b"
))
ggplot2::update_geom_defaults(geom = "label", new = list(
family = "Roboto Condensed",
face = "plain",
size = 3.5,
color = "#2b2b2b"
))
ggplot2::update_geom_defaults(geom = "marquee", new = list(
family = "Roboto Condensed",
face = "plain",
size = 3.5,
color = "#2b2b2b"
))
ggplot2::update_geom_defaults(geom = "text_repel", new = list(
family = "Roboto Condensed",
face = "plain",
size = 3.5,
color = "#2b2b2b"
))
ggplot2::update_geom_defaults(geom = "label_repel", new = list(
family = "Roboto Condensed",
face = "plain",
size = 3.5,
color = "#2b2b2b"
))
## Set the theme
ggplot2::theme_set(new = theme_custom())
## tinytable options
options("tinytable_tt_digits" = 2)
options("tinytable_format_num_fmt" = "significant_cell")
options(tinytable_html_mathjax = TRUE)
## Set defaults for flextable
flextable::set_flextable_defaults(font.family = "Roboto Condensed")
2
| Variable #1 | Variable #2 | Chart Names | Chart Shape | |
|---|---|---|---|---|
| Quant | Qual | Box Plot |
2.1
| No | Pronoun | Answer | Variable/Scale | Example | What Operations? |
|---|---|---|---|---|---|
| 1 | How Many / Much / Heavy? Few? Seldom? Often? When? | Quantities, with Scale and a Zero Value.Differences and Ratios /Products are meaningful. | Quantitative/Ratio | Length,Height,Temperature in Kelvin,Activity,Dose Amount,Reaction Rate,Flow Rate,Concentration,Pulse,Survival Rate | Correlation |
| 4 | What, Who, Where, Whom, Which | Name, Place, Animal, Thing | Qualitative/Nominal | Name | Count no. of cases,Mode |
3
Alice said, “I say what I mean and I mean what I say!” Are the rest of us so sure? What do we mean when we use any of the phrases above? How definite are we? There is a range of “sureness” and “unsureness”…and this is where we can use box plots like Figure 1 to show that range of opinion.
Maybe it is time for a box plot on uh, shades1 of meaning for Jane Austen Gen-Z phrases! Bah.
4
Box Plots are an extremely useful data visualization that gives us an idea of the distribution of a Quant variable, for each level of another Qual variable.
4.1 How are Boxplots Computed?
The internal process of this plot is as follows:
(Hat tip to student Tanya Michelle Justin for a good question on outlier calculation)
- Make groups of the Quant variable for each level of the Qual
- In each group, rank the Quant variable values in increasing order
- Calculate:
- The values for
median = Q2, Q1, and Q3 based on rank!! -
Values for
min,max, and thenIQR = Q1 - Q3 - Calculate
outlierlimits:- \([Q1 - 1.5*IQR, Q2 + 1.5*IQR]\)
-
Whiskers: All values within \([Q1 - 1.5*IQR, Q2 + 1.5*IQR]\) -
Outliers: All values outside of \([Q1 - 1.5*IQR, Q2 + 1.5*IQR]\)
- The values for
- Plot these as a vertical or horizontal box structure, as shown.
As a result of this, while the box-part of the boxplot always shows 2 full quartiles, the whiskers may not stretch through their quartiles, since some values may be outliers on either side.
4.2 Ranks and Values
The Quant variable is ordered based on the values from min to max. So you could imagine that each value has a rank or sequence number. The min value has \(rank = 1\) and the max value has \(rank = length(var)\).
4.3 Histograms and Box Plots
Note how the histogram that dwells upon the mean and standard deviation, whereas the boxplot focuses on the median and quartiles.
The former uses the values of the Quant variable, whereas the latter uses their sequence number or ranks.
Box plots are often used for example in HR operations to understand Salary distributions across grades of employees. Marks of students in competitive exams are also declared using Quartiles.
4.4 Box Plots and Skewness
Boxplots can be symmetric or show evidence of skew in the data. In such cases the box will typically the two halves of different “sizes”, since these two Quartiles span different ranges in value.
In the Figure 3 (a), we see the difference between boxplots that show symmetric and skewed distributions. The “lid” and the “bottom” of the box are not of similar width in distributions with significant skewness.
Compare these with the corresponding distributions in Figure 3 (b).
4.5 Box Plots and Outliers
Box plots can show the presence of outliers in distributions, with a large number of outliers on one side, as in Figure 4.
5 Plotting Box Plots
We will first look at Wage data from the General Social Survey (1974-2018) conducted in the USA, which is used to illustrate wage discrepancies by gender (while also considering respondent occupation, age, and education). This is available on Vincent Arel-Bundock’s superb repository of datasets. Let us read into R directly from the website.
6 gss_wages dataset
wages <- read_csv("https://vincentarelbundock.github.io/Rdatasets/csv/stevedata/gss_wages.csv")Rows: 61697 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (5): occrecode, wrkstat, gender, educcat, maritalcat
dbl (7): rownames, year, realrinc, age, occ10, prestg10, childs
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.The data has automatically been read into the webr session, so you can continue on to the next code chunk!
6.1
As per our Workflow, we will look at the data using all the three methods we have seen.
glimpse(wages)Rows: 61,697
Columns: 12
$ rownames <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, …
$ year <dbl> 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974,…
$ realrinc <dbl> 4935, 43178, NA, NA, 18505, 22206, 55515, NA, NA, 4935, NA,…
$ age <dbl> 21, 41, 83, 69, 58, 30, 48, 67, 51, 54, 89, 71, 27, 30, 22,…
$ occ10 <dbl> 5620, 2040, NA, NA, 5820, 910, 230, 6355, 4720, 3940, 4810,…
$ occrecode <chr> "Office and Administrative Support", "Professional", NA, NA…
$ prestg10 <dbl> 25, 66, NA, NA, 37, 45, 59, 49, 28, 38, 47, 45, 50, 29, 33,…
$ childs <dbl> 0, 3, 2, 2, 0, 0, 2, 1, 2, 2, 3, 1, 4, 3, 0, 1, 2, 3, 4, 8,…
$ wrkstat <chr> "School", "Full-Time", "Housekeeper", "Housekeeper", "Full-…
$ gender <chr> "Male", "Male", "Female", "Female", "Female", "Male", "Male…
$ educcat <chr> "High School", "Bachelor", "Less Than High School", "Less T…
$ maritalcat <chr> "Married", "Married", "Widowed", "Widowed", "Never Married"…skim(wages)| Name | wages |
| Number of rows | 61697 |
| Number of columns | 12 |
| _______________________ | |
| Column type frequency: | |
| character | 5 |
| numeric | 7 |
| ________________________ | |
| Group variables | None |
Variable type: character
| skim_variable | n_missing | complete_rate | min | max | empty | n_unique | whitespace |
|---|---|---|---|---|---|---|---|
| occrecode | 3561 | 0.94 | 5 | 37 | 0 | 11 | 0 |
| wrkstat | 21 | 1.00 | 5 | 23 | 0 | 8 | 0 |
| gender | 0 | 1.00 | 4 | 6 | 0 | 2 | 0 |
| educcat | 135 | 1.00 | 8 | 21 | 0 | 5 | 0 |
| maritalcat | 27 | 1.00 | 7 | 13 | 0 | 5 | 0 |
Variable type: numeric
| skim_variable | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist |
|---|---|---|---|---|---|---|---|---|---|---|
| rownames | 0 | 1.00 | 30849.00 | 17810.53 | 1 | 15425 | 30849 | 46273 | 61697.0 | ▇▇▇▇▇ |
| year | 0 | 1.00 | 1996.07 | 12.79 | 1974 | 1985 | 1996 | 2006 | 2018.0 | ▆▇▇▇▇ |
| realrinc | 23810 | 0.61 | 22326.36 | 28581.79 | 227 | 8156 | 16563 | 27171 | 480144.5 | ▇▁▁▁▁ |
| age | 219 | 1.00 | 46.18 | 17.56 | 18 | 32 | 44 | 59 | 89.0 | ▇▇▆▅▂ |
| occ10 | 3561 | 0.94 | 4695.77 | 2627.72 | 10 | 2710 | 4720 | 6230 | 9997.0 | ▃▅▇▂▃ |
| prestg10 | 4186 | 0.93 | 43.06 | 12.99 | 16 | 33 | 42 | 50 | 80.0 | ▃▇▇▃▁ |
| childs | 189 | 1.00 | 1.92 | 1.76 | 0 | 0 | 2 | 3 | 8.0 | ▇▇▂▁▁ |
inspect(wages)
categorical variables:
name class levels n missing
1 occrecode character 11 58136 3561
2 wrkstat character 8 61676 21
3 gender character 2 61697 0
4 educcat character 5 61562 135
5 maritalcat character 5 61670 27
distribution
1 Professional (19%), Service (16.9%) ...
2 Full-Time (49.4%), Housekeeper (15.1%) ...
3 Female (56.1%), Male (43.9%)
4 High School (51.5%) ...
5 Married (51.7%), Never Married (21.8%) ...
quantitative variables:
name class min Q1 median Q3 max mean sd
1 rownames numeric 1 15425 30849 46273 61697.0 30849.000000 17810.534116
2 year numeric 1974 1985 1996 2006 2018.0 1996.073715 12.794470
3 realrinc numeric 227 8156 16563 27171 480144.5 22326.359234 28581.794499
4 age numeric 18 32 44 59 89.0 46.176177 17.561065
5 occ10 numeric 10 2710 4720 6230 9997.0 4695.774081 2627.724076
6 prestg10 numeric 16 33 42 50 80.0 43.060701 12.987526
7 childs numeric 0 0 2 3 8.0 1.923457 1.763569
n missing
1 61697 0
2 61697 0
3 37887 23810
4 61478 219
5 58136 3561
6 57511 4186
7 61508 189- Much data is missing in the target variable
realinc(income). - Good mix of Qual and Quant variables
6.2
Since there are so many missing data in the target variable realinc and there is still enough data leftover, we can drop the missing values in that variable. This is not advised at all as a general procedure!! Data is valuable and there are better ways to manage this problem!
Important
It turns out that following our process of munging the data using the naniar package to replace common NA strings and numbers with actual NA makes the code run very slowly. I need to figure out how to optimize this.
Show the Code
wages_clean <- wages %>%
tidyr::drop_na(realrinc) %>% # choose column or leave blank to choose all columns
janitor::clean_names(case = "snake") %>% # clean names
dplyr::mutate(across(where(is.character), as_factor)) %>% # make factors
dplyr::mutate(childs = as_factor(childs)) %>% # make childs a factor too
dplyr::relocate(where(is.factor), .after = rownames) # move factors to the right of rownames
glimpse(wages_clean)Rows: 37,887
Columns: 12
$ rownames <dbl> 1, 2, 5, 6, 7, 10, 15, 16, 21, 26, 27, 30, 31, 33, 34, 36, …
$ occrecode <fct> "Office and Administrative Support", "Professional", "Offic…
$ childs <fct> 0, 3, 0, 0, 2, 2, 0, 1, 0, 1, 4, 3, 2, 0, 4, 2, 4, 0, 1, 0,…
$ wrkstat <fct> "School", "Full-Time", "Full-Time", "School", "Full-Time", …
$ gender <fct> Male, Male, Female, Male, Male, Female, Female, Male, Male,…
$ educcat <fct> High School, Bachelor, High School, Bachelor, Graduate, Les…
$ maritalcat <fct> Married, Married, Never Married, Married, Married, Married,…
$ year <dbl> 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974, 1974,…
$ realrinc <dbl> 4935, 43178, 18505, 22206, 55515, 4935, 4935, 18505, 11103,…
$ age <dbl> 21, 41, 58, 30, 48, 54, 22, 23, 25, 59, 53, 35, 51, 21, 72,…
$ occ10 <dbl> 5620, 2040, 5820, 910, 230, 3940, 4020, 7810, 8640, 4710, 4…
$ prestg10 <dbl> 25, 66, 37, 45, 59, 38, 33, 28, 31, 48, 31, 47, 39, 51, 38,…
6.3
From the dataset documentation page, we note that this is a large dataset (61K rows), with 11 variables:
NoteQualitative Data
-
occrecode(fct): recode of the occupation code into one of 11 main categories
-
wrkstat(fct): the work status of the respondent (full-time, part-time, temporarily not working, unemployed (laid off), retired, school, housekeeper, other). 8 levels. -
gender(fct): respondent’s gender (male or female). 2 levels.
-
educcat(fct): respondent’s degree level (Less Than High School, High School, Junior College, Bachelor, or Graduate). 5 levels.
-
maritalcat(fct): respondent’s marital status (Married, Widowed, Divorced, Separated, Never Married). 5 levels.
childs(dbl): number of children (0-8)
NoteQuantitative Data
-
year(dbl): the survey year -
realrinc(dbl): the respondent’s base income (in constant 1986 USD -
age(dbl): the respondent’s age in years -
occ10(dbl): respondent’s occupation code (2010) -
prestg10(dbl): respondent’s occupational prestige score (2010)
6.4
Show the Code
wages_clean %>%
DT::datatable(
caption = htmltools::tags$caption(
style = "caption-side: top; text-align: left; color: black; font-size: 150%;",
"GSS Wages Dataset (Clean)"
),
options = list(pageLength = 10, autoWidth = TRUE)
) %>%
DT::formatStyle(
columns = names(wages_clean),
fontFamily = "Roboto Condensed",
fontSize = "12px"
)
6.5
- The target variable for an experiment that resulted in this data might be the
realincvariable, the resultant income of the individual. Which is numerical variable. - The predictor variables could be
gender,educcat,maritalcat, andwrkstat(all categorical variables), andageandchilds(numerical variables). - Note that
childsis a coded as numerical variable, but it is actually a count of children, and so it can be treated as a categorical variable with 9 levels (0 to 8). We have done that above.
6.6 Research Questions:
- What is the basic distribution of
realrinc? - Is
realrincaffected bygender? -
By
educcat? Bymaritalcat? - Is
realrincaffected bychild? - Do combinations of these factors have an effect on the target variable?
These should do for now! But we should make more questions when have seen some plots!
Note
See that preposition “by” in the questions above? It is a good idea to use it when you are asking about the effect of a Qual variable on a Quant variable. And we know that Qual variables are the ones that define…groups! So we are asking about the effect of groups on a Quant variable.
7
7.1 realrinc?
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(realrinc ~ "Overall", orientation = "x") %>% # Dummy X-axis "variable"
gf_labs(
title = "Plot 1A: Income has a skewed distribution",
subtitle = "Many outliers on the high side",
x = "", y = "Income"
) # Blank out the X-axis title
- Income is a very skewed distribution, as might be expected.
- Presence of many higher-side outliers is noted.
7.2 realrinc affected by gender?
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(gender ~ realrinc, orientation = "y") %>%
gf_labs(y = "Gender", x = "Income", title = "Plot 2A: Income by Gender")
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(gender ~ realrinc, fill = ~gender, orientation = "y") %>%
gf_refine(scale_x_log10(), scale_fill_brewer(palette = "Set1")) %>%
gf_labs(y = "Gender", x = "Income", title = "Plot 2C: Income filled by Gender, log scale")
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
ggplot() +
geom_boxplot(aes(y = gender, x = realrinc)) +
labs(title = "Plot 2A: Income by Gender")
##
wages_clean %>%
ggplot() +
geom_boxplot(aes(y = gender, x = log10(realrinc))) +
labs(title = "Plot 2B: Log(Income) by Gender")
##
wages_clean %>%
ggplot() +
geom_boxplot(aes(y = gender, x = realrinc, fill = gender)) +
scale_x_log10() +
scale_fill_brewer(palette = "Set1") +
labs(title = "Plot 2C: Income filled by Gender, log scale")
- Even when split by
gender,realincomepresents a skewed set of distributions. - The IQR for
males is smaller than the IQR forfemales. There is less variation in the middle ranges ofrealrincfor men. - log10 transformation helps to view and understand the regions of low
realrinc. - There are outliers on both sides, indicating that there may be many people who make very small amounts of money and large amounts of money in both
genders.
7.3 realrinc affected by educcat?
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(educcat ~ realrinc, orientation = "y") %>%
gf_labs(title = "Plot 3A: Income by Education Category")
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(
reorder(educcat, realrinc, FUN = median) ~ log(realrinc),
fill = ~educcat, alpha = 0.5, orientation = "y"
) %>%
gf_labs(
title = "Plot 3C: Log(Income) by Education Category, sorted",
x = "Log Income",
y = "Education Category"
) %>%
gf_refine(scale_fill_brewer(palette = "Set1"))
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(reorder(educcat, realrinc, FUN = median) ~ realrinc,
fill = ~educcat, orientation = "y",
alpha = 0.5
) %>%
gf_refine(scale_x_log10()) %>%
gf_labs(
title = "Plot 3D: Income by Education Category, sorted",
subtitle = "Log Income",
x = "Income",
y = "Education Category"
) %>%
gf_refine(scale_fill_brewer(palette = "Set1"))
Note that educcat has been sorted in the last two plots, by the median value of realrinc in each category. This makes it easier to see the trend. And that educcat has a “NA” level as well, that indicates people who did not report their education level. This is plotted at one end of the box plot by default. (regardless of how the boxplots are sorted. Try reordering in desc() order to see what happens!)
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
ggplot() +
geom_boxplot(aes(realrinc, educcat)) + # (x,y) format
labs(title = "Plot 3A: Income by Education Category")
##
wages_clean %>%
ggplot() +
geom_boxplot(aes(log10(realrinc), educcat)) +
labs(title = "Plot 3B: Log(Income) by Education Category")
##
wages_clean %>%
ggplot() +
geom_boxplot(
aes(log(realrinc),
reorder(educcat, realrinc, FUN = median),
fill = educcat
),
alpha = 0.5
) +
labs(
title = "Plot 3C: Log(Income) by Education Category, sorted",
x = "Log Income", y = "Education Category"
)
##
wages_clean %>%
ggplot() +
geom_boxplot(
aes(realrinc,
reorder(educcat, realrinc, FUN = median),
fill = educcat
),
alpha = 0.5
) +
scale_x_log10() +
labs(
title = "Plot 3D: Income by Education Category, sorted",
subtitle = "Log Income Scale",
x = "Income", y = "Education Category"
)
-
realrincrises witheduccat, which is to be expected. - However, there are people with very low and very high income in all categories of
educcat - Hence
educcatalone may not be a good predictor forrealrinc.
We can do similar work with the other Qual variables. Let us now see how we can use more than one Qual variable and answer the last hypothesis, Question 4.
7.4 realrinc affected by combinations of Qual factors gender, educcat, maritalcat and childs?
Important
This is a rather complex question and could take us deep into Modelling. Ideally we ought to:
- take each Qual variable, explain its effect on the target variable
-
remove that effect and model the remainder ( i.e. residual) with the next Qual variable
- Proceed in this way until we have a good model.
if we are going to do this manually.
There are more modern Modelling Workflows, that can do things much faster and without such manual tweaking.
So will simply plot box plots showing effects on the target variable of combinations of Qual variables taken two at a time. (We will of course use facetted box plots!)
We will also drop NA values all around this time, to avoid seeing boxplots for undocumented categories.
7.5 Question-4: Is realrinc affected by combinations of factors?
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
gf_boxplot(reorder(educcat, realrinc) ~ log10(realrinc),
fill = ~educcat, orientation = "y",
alpha = 0.5
) %>%
gf_facet_wrap(vars(childs)) %>%
gf_refine(scale_fill_brewer(type = "qual", palette = "Dark2")) %>%
gf_labs(
title = "Plot 4A: Log Income by Education Category and Family Size",
x = "Log income",
y = "No. of Children"
)
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
mutate(childs = as_factor(childs)) %>%
gf_boxplot(childs ~ log10(realrinc),
group = ~childs,
fill = ~childs, orientation = "y",
alpha = 0.5
) %>%
gf_facet_wrap(~gender) %>%
gf_refine(scale_fill_brewer(type = "qual", palette = "Set3")) %>%
gf_labs(
title = "Plot 4B: Log Income by Gender and Family Size",
x = "Log income",
y = "No. of Children"
)
ggplot2::theme_set(new = theme_custom())
wages_clean %>%
ggplot() +
geom_boxplot(
aes(log10(realrinc), reorder(educcat, realrinc),
fill = educcat
), # aes() closes here
alpha = 0.5
) +
facet_wrap(vars(childs)) +
scale_fill_brewer(type = "qual", palette = "Dark2") +
labs(title = "Plot 4A: Log Income by Education Category and Family Size", x = "Log income", y = "No. of Children")
##
wages_clean %>%
mutate(childs = as_factor(childs)) %>%
ggplot() +
geom_boxplot(
aes(log10(realrinc), childs,
group = childs,
fill = childs
), # aes() closes here
alpha = 0.5
) +
facet_wrap(vars(gender)) +
scale_fill_brewer(type = "qual", palette = "Set3") +
labs(
title = "Plot 4B: Log Income by Gender and Family Size",
x = "Log income",
y = "No. of Children"
)
- We see that
realrincincreases witheduccat, across (almost) all family sizeschilds. - However, this trend breaks a little when family sizes
childsis large, say >= 7. Be aware that the data observations for such large families may be sparse and this inference may not be necessarily valid. - We see that the effect of
childsonrealrincis different for eachgender! For females, the income steadily drops with the number of children, whereas for males it actually increases up to a certain family size before decreasing again.
8
ImportantHunches and Hypotheses
In data analysis, we always want to know2, as in life, how important things are, whether they matter. To do this, we make up hunches, or more precisely, Hypotheses. We make two in fact:
\(H_0\): Nothing is happening;
\(H_a\): (“a” for Alternate): Something is happening and it is important enough to pay attention to.
We then pretend that \(H_0\) is true and ask that our data prove us wrong; if it does, we reject \(H_0\) in favour of \(H_a\).
This is a very important idea of Hypothesis Testing which helps you justify your hunch. We will study this when we do Stats Tests for differences between two means(t-tests), and those between more than two means(ANOVA).
9
- Box plots are a powerful statistical graphic that give us a combined view of data ranges, quartiles, medians, and outliers.
- Box plots can compare
groupswithin our Quant variable, based onlevelsof a Qual variable. This is a very common and important task in research! - In your design research, you would have numerical Quant data that is accompanied by categorical Qual data pertaining to groups within your target audience.
- Analyzing for differences in the Quant across levels of the Qual (e.g household expenditure across groups of people) is a vital step in justifying time, effort, and money for further actions in your project. Don’t faff this.
- Box plots are ideal for visualizing statistical tests for difference in mean values across groups (t-test and ANOVA). (Even though they plot medians)
10
- Box Plots “dwell upon” medians and Quartiles
- Box Plots can show distributions of a Quant variable over levels of a Qual variable
- Box Plots should be sorted by median values of the Quant variable. And coloured too.
- Horizontal aligning of multiple Box Plots is preferable! Why?
- This allows a comparison of box plots side by side to visibly detect differences in medians and IQRs across such levels.
11
Here are a couple of datasets that you might want to analyze with box plots:
NoteUFO Encounters