A regular expression … is a sequence of characters that defines a search pattern. Usually such patterns are used by string searching algorithms for “find” or “find and replace” operations on strings, or for input validation. It is a technique developed in theoretical computer science and formal language theory.
Tools for characterizing a regular expression
Escape sequences
Just to scratch the surface, here are a few special characters that cannot be directly coded. Therefore, they are escaped with a backslash, \.
\': single quote.
\": double quote.
\n: new line.
\r: carriage return.
\t: tab character.
Quantifiers
Quantifiers specify how many repetitions of the pattern.
*: matches at least 0 times.
+: matches at least 1 times.
?: matches at most 1 times.
{n}: matches exactly n times.
{n,}: matches at least n times.
{n,m}: matches between n and m times.
str_*() review:
From the stringr cheat sheet: https://rstudio.github.io/cheatsheets/html/strings.html
[1] "a" "a" "a" "a" "e" "i" "a" "a" "o" "u" "o" "e" "a" "a" "e" "e" "i" "e" "o"
[20] "o" "a" "u" "u" "a" "a" "a" "u" "e" "e" "e" "i" "o" "o" "a" "a" "u" "o" "u"
[39] "a" "a" "u" "i" "u" "e" "i" "o" "e" "a" "a" "u" "e" "u" "o" "o" "a" "a" "a"
[58] "e" "e" "e" "a" "i" "u" "o" "o" "u" "u" "a" "a" "a" "e" "o" "a" "a" "a" "a"
[77] "a" "a" "u" "a"
strings <-c("apple", "applet", "pineapple", "apple pie", "I love apple pie")str_subset(strings, "\\bapple\\b")str_subset(strings, "^apple$")str_subset(strings, "\\bapple pie\\b")str_subset(strings, "^apple pie$")str_extract(strings, "\\bapple\\b")str_extract(strings, "^apple$")str_extract(strings, "\\bapple pie\\b")str_extract(strings, "^apple pie$")
Bounding words vs. phrases - solutions
strings <-c("apple", "applet", "pineapple", "apple pie", "I love apple pie")str_subset(strings, "\\bapple\\b")
[1] "apple" "apple pie" "I love apple pie"
str_subset(strings, "^apple$")
[1] "apple"
str_subset(strings, "\\bapple pie\\b")
[1] "apple pie" "I love apple pie"
str_subset(strings, "^apple pie$")
[1] "apple pie"
str_extract(strings, "\\bapple\\b")
[1] "apple" NA NA "apple" "apple"
str_extract(strings, "^apple$")
[1] "apple" NA NA NA NA
str_extract(strings, "\\bapple pie\\b")
[1] NA NA NA "apple pie" "apple pie"
str_extract(strings, "^apple pie$")
[1] NA NA NA "apple pie" NA
Operators
.: matches any single character, including letters, numbers, punctuation, and spaces (not new line)
[...]: a character list, matches any one of the characters inside the square brackets. A - inside the brackets specifies a range of characters.
Note: if the dash - is last in the character list, it will match to a dash: [A-Z-]
[^...]: an inverted character list, similar to [...], but matches any characters except those inside the square brackets.
\: suppress the special meaning of metacharacters in regular expression, i.e. $ * + . ? [ ] ^ { } | ( ) \. Since \ itself needs to be escaped in R, we need to escape metacharacters with double backslash like \\$.
|: an “or” operator, matches patterns on either side of the |.
(...): grouping in regular expressions. This allows you to retrieve the bits that matched various parts of your regular expression so you can alter them or use them for building up a new string.
note: both (ab|cde) or ab|cde match either the string ab or the string cde. However, ab | cde matches ab or cde (and does not match either of ab or cde) because the “or” is now whitespace on either side of |. That is, it matches ab and a space or space and cde.
Character classes allow specifying entire classes of characters, such as numbers, letters, etc. There are two flavors of character classes, one uses [: and :] around a predefined name inside square brackets and the other uses \ and a special character. They are sometimes interchangeable.
(?i) before the string indicates that the match should be case insensitive (will make the rest of the string that follows case insensitive).
[:digit:] or \d: digits, 0 1 2 3 4 5 6 7 8 9, equivalent to [0-9].
\D: non-digits, equivalent to [^0-9].
[:lower:]: lower-case letters, equivalent to [a-z].
[:upper:]: upper-case letters, equivalent to [A-Z].
[:alpha:]: alphabetic characters, equivalent to [[:lower:][:upper:]] or [A-z].
[:alnum:]: alphanumeric characters, equivalent to [[:alpha:][:digit:]] or [A-z0-9].
\w: word characters, equivalent to [[:alnum:]_] or [A-z0-9_] (letter, number, or underscore).
\W: not word, equivalent to [^A-z0-9_].
[:blank:]: blank characters, i.e. space and tab.
[:space:]: space characters: tab, new line, vertical tab, form feed, carriage return, space.
. matches any single character except a newline \n (including letters, numbers, punctuation, and spaces).
. does match whitespace (e.g., a space or tab)
\s matches any whitespace including: spaces, tabs, new lines, and carriage returns
[ \t] matches spaces and tabs only (not new lines or carriage returns)
[^\s] matches any character except whitespace (including spaces, tabs, and new lines)
[^\s] and [\S] are functionally equivalent
The pattern [\s\S] matches any character including newlines and tabs.
\w matches any single word character (including letters, digits, and the underscore character _)
\B: matches the empty string provided it is not at an edge of a word.
Some examples
More examples for practice!
Case insenstive
Match only the word meter in “The cemetery is 1 meter from the stop sign.”
Also match Meter in “The cemetery is 1 Meter from the stop sign.”
Case insenstive
Match only the word meter in “The cemetery is 1 meter from the stop sign.”
Also match Meter and meTer …
string <-c("The cemetery is 1 meter from the stop sign.","The cemetery is 1 Meter from the stop sign.","The cemetery is 1 meTer from the stop sign.")str_extract(string, "(?i)\\bmeter\\b")
[1] "meter" "Meter" "meTer"
Proper times and dates
Match dates like 01/15/24 and also like 01.15.24 and like 01-15-24.
Match a time of day such as “9:17 am” or “12:30 pm”. Require that the time be a valid time (not “99:99 pm”). Assume no leading zeros (i.e., “09:17 am”).
Match a time of day such as “9:17 am” or “12:30 pm”. Require that the time be a valid time (not “99:99 pm”). Assume no leading zeros (i.e., “09:17 am”).
# A tibble: 10 × 6
Task Description episode contestant score series
<chr> <chr> <chr> <chr> <chr> <dbl>
1 1 Prize: Best thing you can ca… Episod… Charlotte… 1 11
2 1 Prize: Best thing you can ca… Episod… Jamali Ma… 2 11
3 1 Prize: Best thing you can ca… Episod… Lee Mack 4 11
4 1 Prize: Best thing you can ca… Episod… Mike Wozn… 5 11
5 1 Prize: Best thing you can ca… Episod… Sarah Ken… 3 11
6 2 Do the most impressive thing… Episod… Charlotte… 2 11
7 2 Do the most impressive thing… Episod… Jamali Ma… 3[1] 11
8 2 Do the most impressive thing… Episod… Lee Mack 3 11
9 2 Do the most impressive thing… Episod… Mike Wozn… 5 11
10 2 Do the most impressive thing… Episod… Sarah Ken… 4 11
more succinct results
Task Description episode contestant score series1 Prize: Best thing… Episode 1… Charlotte… 1111 Prize: Best thing… Episode 1… Jamali Ma… 2111 Prize: Best thing… Episode 1… Lee Mack 4111 Prize: Best thing… Episode 1… Mike Wozn… 5111 Prize: Best thing… Episode 1… Sarah Ken… 3112 Do the most… Episode 1… Charlotte… 2112 Do the most… Episode 1… Jamali Ma… 3[1] 112 Do the most… Episode 1… Lee Mack 3112 Do the most… Episode 1… Mike Wozn… 5112 Do the most… Episode 1… Sarah Ken… 411
Currently, the episode column contains entries like
How should the scores be stored? What is the cleaning task?
Figure 3: Taskmaster Wiki https://taskmaster.fandom.com/wiki/Series_11
Extracting numeric information
Suppose we have the following string:
"3[1]"
And we want to extract just the number “3”:
str_extract("3[1]", "3")
[1] "3"
Extracting numeric information
What if we don’t know which number to extract?
str_extract("3[1]", "\\d")
[1] "3"
str_extract("4[1]", "\\d")
[1] "4"
str_extract("10[1]", "\\d")
[1] "1"
str_extract("10[1]", "\\d+")
[1] "10"
str_extract("DQ", "\\d")
[1] NA
str_extract()
str_extract() is an R function in the stringr package which can find regular expressions in strings of text.
str_extract("My cat is 3 years old", "cat")
[1] "cat"
str_extract("My cat is 3 years old", "3")
[1] "3"
Matching multiple options
str_extract() returns the first match; str_extract_all() allows more than one match.
str_extract("My cat is 3 years old", "cat|dog")
[1] "cat"
str_extract("My dog is 10 years old", "cat|dog")
[1] "dog"
str_extract("My dog is 10 years old, my cat is 3 years old", "cat|dog")
[1] "dog"
str_extract_all("My dog is 10 years old, my cat is 3 years old", "cat|dog")
[[1]]
[1] "dog" "cat"
Matching groups of characters
What if I want to extract a number?
str_extract("My cat is 3 years old", "\\d")
[1] "3"
What will the result be for the following code?
str_extract("My dog is 10 years old", "\\d")
Matching groups of characters
What if I want to extract a number?
str_extract("My cat is 3 years old", "\\d")
[1] "3"
What will the result be for the following code?
str_extract("My dog is 10 years old", "\\d")
[1] "1"
Matching groups of characters
What if I want to extract a number?
str_extract("My cat is 3 years old", "\\d")
[1] "3"
What will the result be for the following code?
str_extract("My dog is 10 years old", "\\d")
[1] "1"
The + symbol in a regular expression means “repeated one or more times”
str_extract("My dog is 10 years old", "\\d+")
[1] "10"
Extracting from multiple strings
strings <-c("My cat is 3 years old", "My dog is 10 years old")str_extract(strings, "\\d+")
[1] "3" "10"
Extracting from multiple strings
What if we have multiple instances across multiple strings? We need to be careful working with lists (instead of vectors).
strings <-c("My cat is 3 years old", "My dog is 10 years old")str_extract(strings, "\\w")str_extract_all(strings, "\\w")str_extract(strings, "\\w+")str_extract_all(strings, "\\w+")
Extracting from multiple strings
What if we have multiple instances across multiple strings? We need to be careful working with lists (instead of vectors).
strings <-c("My cat is 3 years old", "My dog is 10 years old")str_extract(strings, "\\w")
Currently, the episode column contains entries like:
"Episode 2: The pie whisperer. (4 August 2015)"
How would I extract just the episode number?
Extracting episode information
Currently, the episode column contains entries like:
"Episode 2: The pie whisperer. (4 August 2015)"
How would I extract just the episode number?
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "\\d+")
[1] "2"
Extracting episode information
Currently, the episode column contains entries like:
"Episode 2: The pie whisperer. (4 August 2015)"
How would I extract the episode name?
Goal: find a pattern to match: anything that starts with a :, ends with a .
Let’s break down that task into pieces.
Extracting episode information
How can we find the period at the end of the sentence? What does each of these lines of code return?
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".")
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".+")
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "\\.")
Extracting episode information - solution
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".")
[1] "E"
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".+")
[1] "Episode 2: The pie whisperer. (4 August 2015)"
We use an escape character when we actually want to choose a period:
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "\\.")
[1] "."
Extracting episode information
Goal: find a pattern to match: anything that starts with a :, ends with a .
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ":.+\\.")
[1] ": The pie whisperer."
Lookaround (again)
Figure 4: Image credit: Stefan Judis https://www.stefanjudis.com/blog/a-regular-expression-lookahead-lookbehind-cheat-sheet/
Lookbehinds
(?<=y)x – positive lookbehind (matches ‘x’ when it is preceded by ‘y’)
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "(?<=: ).+")
[1] "The pie whisperer. (4 August 2015)"
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "(?<=\\. ).+")
[1] "(4 August 2015)"
Lookaheads
x(?=y) – positive lookahead (matches ‘x’ when it is followed by ‘y’)
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".+(?=\\.)")
[1] "Episode 2: The pie whisperer"
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ".+(?=:)")
[1] "Episode 2"
Extracting episode information
Getting everything between the : and the .
str_extract("Episode 2: The pie whisperer. (4 August 2015)", "(?<=: ).+(?=\\.)")
[1] "The pie whisperer"
Extracting air date
I want to extract just the air date. What pattern do I want to match?
str_extract("Episode 2: The pie whisperer. (4 August 2015)", ...)
Extracting air date
str_extract("Episode 2: The pie whisperer. (4 August 2015)","(?<=\\().+(?=\\))")
[1] "4 August 2015"
Wrangling the episode info
Currently:
# A tibble: 270 × 1
episode
<chr>
1 Episode 1: It's not your fault. (18 March 2021)
2 Episode 1: It's not your fault. (18 March 2021)
3 Episode 1: It's not your fault. (18 March 2021)
4 Episode 1: It's not your fault. (18 March 2021)
5 Episode 1: It's not your fault. (18 March 2021)
6 Episode 1: It's not your fault. (18 March 2021)
7 Episode 1: It's not your fault. (18 March 2021)
8 Episode 1: It's not your fault. (18 March 2021)
9 Episode 1: It's not your fault. (18 March 2021)
10 Episode 1: It's not your fault. (18 March 2021)
# ℹ 260 more rows
# A tibble: 270 × 4
episode episode_name air_date air_date2
<chr> <chr> <chr> <date>
1 1 It's not your fault 18 March 2021 2021-03-18
2 1 It's not your fault 18 March 2021 2021-03-18
3 1 It's not your fault 18 March 2021 2021-03-18
4 1 It's not your fault 18 March 2021 2021-03-18
5 1 It's not your fault 18 March 2021 2021-03-18
6 1 It's not your fault 18 March 2021 2021-03-18
7 1 It's not your fault 18 March 2021 2021-03-18
8 1 It's not your fault 18 March 2021 2021-03-18
9 1 It's not your fault 18 March 2021 2021-03-18
10 1 It's not your fault 18 March 2021 2021-03-18
# ℹ 260 more rows
# A tibble: 270 × 4
episode episode_name air_date air_date2
<chr> <chr> <chr> <date>
1 1 It's not your fault 18 March 2021 2021-03-18
2 1 It's not your fault 18 March 2021 2021-03-18
3 1 It's not your fault 18 March 2021 2021-03-18
4 1 It's not your fault 18 March 2021 2021-03-18
5 1 It's not your fault 18 March 2021 2021-03-18
6 1 It's not your fault 18 March 2021 2021-03-18
7 1 It's not your fault 18 March 2021 2021-03-18
8 1 It's not your fault 18 March 2021 2021-03-18
9 1 It's not your fault 18 March 2021 2021-03-18
10 1 It's not your fault 18 March 2021 2021-03-18
# ℹ 260 more rows