Due: 2019-09-12 at noon
A note on expectations: For each exercise and on your own question you answer include any relevant output (tables, summary statistics, plots) in your answer. Doing this is easy! Just place any relevant R code in a code chunk, and hit Knit HTML.
Some define statistics as the field that focuses on turning information into knowledge. The first step in that process is to summarize and describe the raw information - the data. In this lab we explore data on college majors and earnings, specifically the data behind the FiveThirtyEight story “The Economic Guide To Picking A College Major”.
These data originally come from the American Community Survey (ACS) 2010-2012 Public Use Microdata Series. While this is outside the scope of this lab, if you are curious about how raw data from the ACS were cleaned and prepared, see the code FiveThirtyEight authors used.
We should also note that there are many considerations that go into picking a major. Earnings potential and employment prospects are two of them, and they are important, but they don’t tell the whole story. Keep this in mind as you analyze the data.
In this lab we will work with the tidyverse
and fivethirtyeight
packages. We start by loading the package.
Note that these packages are also loaded in your R Markdown document.
The data frame we will be working with today is called college_recent_grads
and it’s in the fivethirtyeight
package.
To find out more about the dataset, type the following in your Console: ?college_recent_grads
. A question mark before the name of an object will always bring up its help file. This command must be ran in the Console.
college_recent_grads
is a tidy data frame, with each row representing an observation and each column representing a variable.
To view the data, click on the name of the data frame in the Environment tab.
You can also take a quick peek at your data frame and view its dimensions with the glimpse
function.
The description of the variables, i.e. the codebook, is given below.
Header | Description |
---|---|
rank |
Rank by median earnings |
major_code |
Major code, FO1DP in ACS PUMS |
major |
Major description |
major_category |
Category of major from Carnevale et al |
total |
Total number of people with major |
sample_size |
Sample size (unweighted) of full-time, year-round ONLY (used for earnings) |
men |
Male graduates |
women |
Female graduates |
sharewomen |
Women as share of total |
employed |
Number employed (ESR == 1 or 2) |
employed_full_time |
Employed 35 hours or more |
employed_part_time |
Employed less than 35 hours |
employed_full_time_yearround |
Employed at least 50 weeks (WKW == 1) and at least 35 hours (WKHP >= 35) |
unemployed |
Number unemployed (ESR == 3) |
unemployment_rate |
Unemployed / (Unemployed + Employed) |
median |
Median earnings of full-time, year-round workers |
p25th |
25th percentile of earnings |
p75th |
75th percentile of earnings |
college_jobs |
Number with job requiring a college degree |
non_college_jobs |
Number with job not requiring a college degree |
low_wage_jobs |
Number in low-wage service jobs |
The college_recent_grads
data frame is a trove of information. Let’s think about some questions we might want to answer with these data:
In the next section we aim to answer these questions.
In order to answer this question all we need to do is sort the data. We use the arrange
function to do this, and sort it by the unemployment_rate
variable. By default arrange
sorts in ascending order, which is what we want here – we’re interested in the major with the lowest unemployment rate.
This gives us what we wanted, but not in an ideal form. First, the name of the major barely fits on the page. Second, some of the variables are not that useful (e.g. major_code
, major_category
) and some we might want front and center are not easily viewed (e.g. unemployment_rate
).
We can use the select
function to choose which variables to display, and in which order:
Note how easily we expanded our code with adding another step to our pipeline, with the pipe operator: %>%
.
Ok, this is looking better, but do we really need all those decimal places in the unemployment variable? Not really!
There are two ways we can address this problem. One would be to round the unemployment_rate
variable in the dataset or we can change the number of digits displayed, without touching the input data.
Below are instructions for how you would do both of these:
unemployment_rate
: We create a new variable with the mutate
function. In this case, we’re overwriting the existing unemployment_rate
variable, by round
ing it to 4
decimal places.college_recent_grads %>%
arrange(unemployment_rate) %>%
select(rank, major, unemployment_rate) %>%
mutate(unemployment_rate = round(unemployment_rate, digits = 4))
Note that the digits
in options
is scientific digits, and in round
they are decimal places. If you’re thinking “Wouldn’t it be nice if they were consistent?”, you’re right…
You don’t need to do both of these, that would be redundant. The next exercise asks you to choose one.
To answer such a question we need to arrange the data in descending order. For example, if earlier we were interested in the major with the highest unemployment rate, we would use the following:
The desc
function specifies that we want unemployment_rate
in descending order.
college_recent_grads %>%
arrange(desc(unemployment_rate)) %>%
select(rank, major, unemployment_rate)
head(3)
at the end of the pipeline.A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. For example, the 20th percentile is the value below which 20% of the observations may be found. (Source: Wikipedia)
There are three types of incomes reported in this data frame: p25th
, median
, and p75th
. These correspond to the 25th, 50th, and 75th percentiles of the income distribution of sampled individuals for a given major.
The question we want to answer “How do the distributions of median income compare across major categories?”. We need to do a few things to answer this question: First, we need to group the data by major_category
. Then, we need a way to summarize the distributions of median income within these groups. This decision will depend on the shapes of these distributions. So first, we need to visualize the data.
We use the ggplot
function to do this. The first argument is the data frame, and the next argument gives the mapping of the variables of the data to the aes
thetic elements of the plot.
Let’s start simple and take a look at the distribution of all median incomes, without considering the major categories.
Along with the plot, we get a message:
`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
This is telling us that we might want to reconsider the binwidth we chose for our histogram – or more accurately, the binwidth we didn’t specify. It’s good practice to always thing in the context of the data and try out a few binwidths before settling on a binwidth. You might ask yourself: “What would be a meaningful difference in median incomes?” $1 is obviously too little, $10000 might be too high.
geom_histogram
function. So to specify a binwidth of $1000, you would use geom_histogram(binwidth = 1000)
.We can also calculate summary statistics for this distribution using the summarise
function:
college_recent_grads %>%
summarise(min = min(median), max = max(median),
mean = mean(median), med = median(median),
sd = sd(median),
q1 = quantile(median, probs = 0.25),
q3 = quantile(median, probs = 0.75))
Next, we facet the plot by major category.
ggplot(data = college_recent_grads, mapping = aes(x = median)) +
geom_histogram() +
facet_wrap( ~ major_category, ncol = 4)
median
income using a histogram, faceted by major_category
. Use the binwidth
you chose in the earlier exercise.Now that we’ve seen the shapes of the distributions of median incomes for each major category, we should have a better idea for which summary statistic to use to quantify the typical median income.
count
, which first groups the data and then counts the number of observations in each category (see below). Add to the pipeline appropriately to arrange the results so that the major with the lowest observations is on top.One of the sections of the FiveThirtyEight story is “All STEM fields aren’t the same”. Let’s see if this is true.
First, let’s create a new vector called stem_categories
that lists the major categories that are considered STEM fields.
stem_categories <- c("Biology & Life Science",
"Computers & Mathematics",
"Engineering",
"Physical Sciences")
Then, we can use this to create a new variable in our data frame indicating whether a major is STEM or not.
college_recent_grads <- college_recent_grads %>%
mutate(major_type = ifelse(major_category %in% stem_categories, "stem", "not stem"))
Let’s unpack this: with mutate
we create a new variable called major_type
, which is defined as "stem"
if the major_category
is in the vector called stem_categories
we created earlier, and as "not stem"
otherwise.
%in%
is a logical operator. Other logical operators that are commonly used are
Operator | Operation |
---|---|
x < y |
less than |
x > y |
greater than |
x <= y |
less than or equal to |
x >= y |
greater than or equal to |
x != y |
not equal to |
x == y |
equal to |
x %in% y |
contains |
x | y |
or |
x & y |
and |
!x |
not |
We can use the logical operators to also filter
our data for STEM majors whose median earnings is less than median for all majors’s median earnings, which we found to be $36,000 earlier.
Discover how to change the scale of the x-axis labels of the plot created in Exercise 10 to dollars. Re-plot with this new scale.
Total | 100 pts |
---|---|
Filename your-name-data-wrangle-visualize.Rmd | 5 pts |
Changed author in YAML |
5 pts |
All plots include title, labels | 10 pts |
Exercises 1-10 | 80 pts |
Lab adapted from datasciencebox.org by Dr. Lucy D’Agostino McGowan