age <- c(2, 3, 5)
cats coat weight likes_string
1 calico 2.1 1
2 black 5.0 0
3 tabby 3.2 1Extra tutorials 2: Data Types - subsetting cats and gapminder
Key insights and activities
At this point, you’ve seen it all: in the last lesson, we toured all the basic data types and data structures in R. Everything you do will be a manipulation of those tools. But most of the time, the star of the show is the data frame—the table that we created by loading information from a csv file. In this lesson, we’ll learn a few more things about working with data frames.
Adding columns and rows in data frames
We already learned that the columns of a data frame are vectors, so that our data are consistent in type throughout the columns. As such, if we want to add a new column, we can start by making a new vector:
We can then add this as a column via:
cbind(cats, age) coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5Note that if we tried to add a vector of ages with a different number of entries than the number of rows in the data frame, it would fail:
age <- c(2, 3, 5, 12)
cbind(cats, age)Error in data.frame(..., check.names = FALSE): arguments imply differing number of rows: 3, 4age <- c(2, 3)
cbind(cats, age)Error in data.frame(..., check.names = FALSE): arguments imply differing number of rows: 3, 2Why didn’t this work? Of course, R wants to see one element in our new column for every row in the table:
nrow(cats)[1] 3length(age)[1] 2So for it to work we need to have nrow(cats) = length(age). Let’s overwrite the content of cats with our new data frame.
age <- c(2, 3, 5)
cats <- cbind(cats, age)Now how about adding rows? We already know that the rows of a data frame are lists:
newRow <- list("tortoiseshell", 3.3, TRUE, 9)
cats <- rbind(cats, newRow)Warning in `[<-.factor`(`*tmp*`, ri, value = "tortoiseshell"): invalid factor
level, NA generatedLooks like our attempt to use the rbind() function returns a warning. Recall that, unlike errors, warnings do not necessarily stop a function from performing its intended action. You can confirm this by taking a look at the cats data frame.
cats coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
4 <NA> 3.3 1 9Notice that not only did we successfully add a new row, but there is NA in the column coats where we expected “tortoiseshell” to be. Why did this happen?
Factors
For an object containing the data type factor, each different value represents what is called a level. In our case, the factor “coat” has 3 levels: “black”, “calico”, and “tabby”. R will only accept values that match one of the levels. If you add a new value, it will become NA.
The warning is telling us that we unsuccessfully added “tortoiseshell” to our coat factor, but 3.3 (a numeric), TRUE (a logical), and 9 (a numeric) were successfully added to weight, likes_string, and age, respectively, since those variables are not factors. To successfully add a cat with a “tortoiseshell” coat, add “tortoiseshell” as a possible level of the factor:
levels(cats$coat)[1] "black" "calico" "tabby" levels(cats$coat) <- c(levels(cats$coat), "tortoiseshell")
cats <- rbind(cats, list("tortoiseshell", 3.3, TRUE, 9))Alternatively, we can change a factor into a character vector; we lose the handy categories of the factor, but we can subsequently add any word we want to the column without babysitting the factor levels:
str(cats)'data.frame': 5 obs. of 4 variables:
$ coat : Factor w/ 4 levels "black","calico",..: 2 1 3 NA 4
$ weight : num 2.1 5 3.2 3.3 3.3
$ likes_string: int 1 0 1 1 1
$ age : num 2 3 5 9 9cats$coat <- as.character(cats$coat)
str(cats)'data.frame': 5 obs. of 4 variables:
$ coat : chr "calico" "black" "tabby" NA ...
$ weight : num 2.1 5 3.2 3.3 3.3
$ likes_string: int 1 0 1 1 1
$ age : num 2 3 5 9 9Removing rows
We now know how to add rows and columns to our data frame in R—but in our first attempt to add a “tortoiseshell” cat to the data frame we have accidentally added a garbage row:
cats coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
4 <NA> 3.3 1 9
5 tortoiseshell 3.3 1 9We can ask for a data frame minus this offending row:
cats[-4, ] coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
5 tortoiseshell 3.3 1 9Notice the comma with nothing after it to indicate that we want to drop the entire fourth row.
Note: we could also remove both new rows at once by putting the row numbers inside of a vector: cats[c(-4,-5), ]
Alternatively, we can drop all rows with NA values:
na.omit(cats) coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
5 tortoiseshell 3.3 1 9Let’s reassign the output to cats, so that our changes will be permanent:
cats <- na.omit(cats)Removing columns
We can also remove columns in our data frame. What if we want to remove the column “age”. We can remove it in two ways, by variable number or by index.
cats[,-4] coat weight likes_string
1 calico 2.1 1
2 black 5.0 0
3 tabby 3.2 1
5 tortoiseshell 3.3 1Notice the comma with nothing before it, indicating we want to keep all of the rows.
Alternatively, we can drop the column by using the index name and the %in% operator. The %in% operator goes through each element of its left argument, in this case the names of cats, and asks, “Does this element occur in the second argument?”
drop <- names(cats) %in% c("age")
cats[,!drop] coat weight likes_string
1 calico 2.1 1
2 black 5.0 0
3 tabby 3.2 1
5 tortoiseshell 3.3 1We will cover subsetting with logical operators like %in% in more detail in the next episode. See the section Subsetting through other logical operations
Appending to a data frame
The key to remember when adding data to a data frame is that columns are vectors and rows are lists. We can also glue two data frames together with rbind:
cats <- rbind(cats, cats)
cats coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
5 tortoiseshell 3.3 1 9
11 calico 2.1 1 2
21 black 5.0 0 3
31 tabby 3.2 1 5
51 tortoiseshell 3.3 1 9But now the row names are unnecessarily complicated. We can remove the rownames, and R will automatically re-name them sequentially:
rownames(cats) <- NULL
cats coat weight likes_string age
1 calico 2.1 1 2
2 black 5.0 0 3
3 tabby 3.2 1 5
4 tortoiseshell 3.3 1 9
5 calico 2.1 1 2
6 black 5.0 0 3
7 tabby 3.2 1 5
8 tortoiseshell 3.3 1 9Challenge 2
You can create a new data frame right from within R with the following syntax:
df <- data.frame(id = c("a", "b", "c"), x = 1:3, y = c(TRUE, TRUE, FALSE) )Make a data frame that holds the following information for yourself:
- first name
- last name
- lucky number
Then use
rbindto add an entry for the people sitting beside you. Finally, usecbindto add a column with each person’s answer to the question, “Is it time for coffee break?”Solution to Challenge 2
df <- data.frame(first = c("Grace"), last = c("Hopper"), lucky_number = c(0)) df <- rbind(df, list("Marie", "Curie", 238) ) df <- cbind(df, coffeetime = c(TRUE,TRUE)){: .solution} {: .challenge}
Realistic example
So far, you have seen the basics of manipulating data frames with our cat data; now let’s use those skills to digest a more realistic dataset. Let’s read in the gapminder dataset
library(gapminder)
gapminder <- gapminderLet’s investigate gapminder a bit; the first thing we should always do is check out what the data looks like with
str:
str(gapminder)tibble [1,704 × 6] (S3: tbl_df/tbl/data.frame)
$ country : Factor w/ 142 levels "Afghanistan",..: 1 1 1 1 1 1 1 1 1 1 ...
$ continent: Factor w/ 5 levels "Africa","Americas",..: 3 3 3 3 3 3 3 3 3 3 ...
$ year : int [1:1704] 1952 1957 1962 1967 1972 1977 1982 1987 1992 1997 ...
$ lifeExp : num [1:1704] 28.8 30.3 32 34 36.1 ...
$ pop : int [1:1704] 8425333 9240934 10267083 11537966 13079460 14880372 12881816 13867957 16317921 22227415 ...
$ gdpPercap: num [1:1704] 779 821 853 836 740 ...An additional method for examining the structure of gapminder is to use the summary function. This function can be used on various objects in R. For data frames, summary yields a numeric, tabular, or descriptive summary of each column. Factor columns are summarized by the number of items in each level, numeric or integer columns by the descriptive statistics (quartiles and mean), and character columns by its length, class, and mode.
summary(gapminder$country) Afghanistan Albania Algeria
12 12 12
Angola Argentina Australia
12 12 12
Austria Bahrain Bangladesh
12 12 12
Belgium Benin Bolivia
12 12 12
Bosnia and Herzegovina Botswana Brazil
12 12 12
Bulgaria Burkina Faso Burundi
12 12 12
Cambodia Cameroon Canada
12 12 12
Central African Republic Chad Chile
12 12 12
China Colombia Comoros
12 12 12
Congo, Dem. Rep. Congo, Rep. Costa Rica
12 12 12
Cote d'Ivoire Croatia Cuba
12 12 12
Czech Republic Denmark Djibouti
12 12 12
Dominican Republic Ecuador Egypt
12 12 12
El Salvador Equatorial Guinea Eritrea
12 12 12
Ethiopia Finland France
12 12 12
Gabon Gambia Germany
12 12 12
Ghana Greece Guatemala
12 12 12
Guinea Guinea-Bissau Haiti
12 12 12
Honduras Hong Kong, China Hungary
12 12 12
Iceland India Indonesia
12 12 12
Iran Iraq Ireland
12 12 12
Israel Italy Jamaica
12 12 12
Japan Jordan Kenya
12 12 12
Korea, Dem. Rep. Korea, Rep. Kuwait
12 12 12
Lebanon Lesotho Liberia
12 12 12
Libya Madagascar Malawi
12 12 12
Malaysia Mali Mauritania
12 12 12
Mauritius Mexico Mongolia
12 12 12
Montenegro Morocco Mozambique
12 12 12
Myanmar Namibia Nepal
12 12 12
Netherlands New Zealand Nicaragua
12 12 12
Niger Nigeria Norway
12 12 12
Oman Pakistan Panama
12 12 12
(Other)
516 nrow(gapminder)[1] 1704ncol(gapminder)[1] 6Or, both at once:
dim(gapminder)[1] 1704 6We’ll also likely want to know what the titles of all the columns are, so we can ask for them later:
colnames(gapminder)[1] "country" "continent" "year" "lifeExp" "pop" "gdpPercap"At this stage, it’s important to ask ourselves if the structure R is reporting matches our intuition or expectations; do the basic data types reported for each column make sense? If not, we need to sort any problems out now before they turn into bad surprises down the road, using what we’ve learned about how R interprets data, and the importance of strict consistency in how we record our data.
Once we’re happy that the data types and structures seem reasonable, it’s time to start digging into our data proper. Check out the first few lines:
head(gapminder)# A tibble: 6 × 6
country continent year lifeExp pop gdpPercap
<fct> <fct> <int> <dbl> <int> <dbl>
1 Afghanistan Asia 1952 28.8 8425333 779.
2 Afghanistan Asia 1957 30.3 9240934 821.
3 Afghanistan Asia 1962 32.0 10267083 853.
4 Afghanistan Asia 1967 34.0 11537966 836.
5 Afghanistan Asia 1972 36.1 13079460 740.
6 Afghanistan Asia 1977 38.4 14880372 786.Challenge 3
It’s good practice to also check the last few lines of your data and some in the middle. How would you do this?
Searching for ones specifically in the middle isn’t too hard, but we could ask for a few lines at random. How would you code this?
Solution to Challenge 3
To check the last few lines it’s relatively simple as R already has a function for this:
tail(gapminder) tail(gapminder, n = 15)
Challenge 5
Read the output of
str(gapminder)again; this time, use what you’ve learned about factors, lists and vectors, as well as the output of functions likecolnamesanddimto explain what everything thatstrprints out for gapminder means. If there are any parts you can’t interpret, discuss with your neighbors!Solution to Challenge 5
The object
gapminderis a data frame with columns -countryandcontinentare factors. -yearis an integer vector. -pop,lifeExp, andgdpPercapare numeric vectors.{: .solution} {: .challenge}
Data frames
Remember the data frames are lists underneath the hood, so similar rules apply. However they are also two dimensional objects:
[ with one argument will act the same way as for lists, where each list element corresponds to a column. The resulting object will be a data frame:
head(gapminder[3])# A tibble: 6 × 1
year
<int>
1 1952
2 1957
3 1962
4 1967
5 1972
6 1977Similarly, [[ will act to extract a single column:
head(gapminder[["lifeExp"]])[1] 28.801 30.332 31.997 34.020 36.088 38.438And $ provides a convenient shorthand to extract columns by name:
head(gapminder$year)[1] 1952 1957 1962 1967 1972 1977With two arguments, [ behaves the same way as for matrices:
gapminder[1:3,]# A tibble: 3 × 6
country continent year lifeExp pop gdpPercap
<fct> <fct> <int> <dbl> <int> <dbl>
1 Afghanistan Asia 1952 28.8 8425333 779.
2 Afghanistan Asia 1957 30.3 9240934 821.
3 Afghanistan Asia 1962 32.0 10267083 853.If we subset a single row, the result will be a data frame (because the elements are mixed types):
gapminder[3,]# A tibble: 1 × 6
country continent year lifeExp pop gdpPercap
<fct> <fct> <int> <dbl> <int> <dbl>
1 Afghanistan Asia 1962 32.0 10267083 853.But for a single column the result will be a vector (this can be changed with the third argument, drop = FALSE).
Challenge 7
Fix each of the following common data frame subsetting errors:
Extract observations collected for the year 1957
gapminder[gapminder$year = 1957,]Extract all columns except 1 through to 4
gapminder[,-1:4]Extract the rows where the life expectancy is longer the 80 years
gapminder[gapminder$lifeExp > 80]Extract the first row, and the fourth and fifth columns (
continentandlifeExp).gapminder[1, 4, 5]Advanced: extract rows that contain information for the years 2002 and 2007
gapminder[gapminder$year == 2002 | 2007,]Solution to challenge 7
Fix each of the following common data frame subsetting errors:
Extract observations collected for the year 1957
# gapminder[gapminder$year = 1957,] gapminder[gapminder$year == 1957,]Extract all columns except 1 through to 4
# gapminder[,-1:4] gapminder[,-c(1:4)]Extract the rows where the life expectancy is longer than 80 years
# gapminder[gapminder$lifeExp > 80] gapminder[gapminder$lifeExp > 80,]Extract the first row, and the fourth and fifth columns (
continentandlifeExp).# gapminder[1, 4, 5] gapminder[1, c(4, 5)]Advanced: extract rows that contain information for the years 2002 and 2007
# gapminder[gapminder$year == 2002 | 2007,] gapminder[gapminder$year == 2002 | gapminder$year == 2007,] gapminder[gapminder$year %in% c(2002, 2007),]{: .solution} {: .challenge}
Challenge 8
Why does
gapminder[1:20]return an error? How does it differ fromgapminder[1:20, ]?Create a new
data.framecalledgapminder_smallthat only contains rows 1 through 9 and 19 through 23. You can do this in one or two steps.Solution to challenge 8
gapminderis a data.frame so needs to be subsetted on two dimensions.gapminder[1:20, ]subsets the data to give the first 20 rows and all columns.gapminder_small <- gapminder[c(1:9, 19:23),]{: .solution} {: .challenge}