Visualizing text data

36-315: Statistical Graphics and Visualization, Summer 2026

Text data example

The Office, Season 4 Episode 13: Dinner Party
Access via the schrute package

library(tidyverse)
theme_set(theme_light())

library(schrute)
transcripts <- theoffice |>
  filter(season == 4, episode == 13) |>
  select(index, character, text)
head(transcripts)

# A tibble: 6 × 3
  index character text                                                          
  <int> <chr>     <chr>                                                         
1 16791 Stanley   This is ridiculous.                                           
2 16792 Phyllis   Do you have any idea what time we'll get out of here?         
3 16793 Michael   Nobody likes to work late, least of all me. Do you have plans…
4 16794 Jim       Nope I don't, remember when you told us not to make plans 'ca…
5 16795 Michael   Yes I remember. Mmm, this is B.S. This is B.S. Why are we her…
6 16796 Dwight    Thank you Michael.

Bag of words representation of text

Most common way to store text data: document-term matrix (DTM)

	Word 1	Word 2	\(\dots\)	Word \(J\)
Document \(1\)	\(w_{11}\)	\(w_{12}\)	\(\dots\)	\(w_{1J}\)
Document \(2\)	\(w_{21}\)	\(w_{22}\)	\(\dots\)	\(w_{2J}\)
\(\dots\)	\(\dots\)	\(\dots\)	\(\dots\)	\(\dots\)
Document \(N\)	\(w_{N1}\)	\(w_{N2}\)	\(\dots\)	\(w_{NJ}\)

\(w_{ij}\): count of word \(j\) in document \(i\), or term frequency

To reduce number of columns:

Stop words: remove extremely common, functional words (e.g., the, and, is, in, of, etc.)
Stemming: reduce all words to their “stem”
- Example: compute, computer, computed, computing, computation \(\rightarrow\) comput

Tokenize text into long format

Convert raw text into long, tidy table with one-token-per-document-per-row
- A token: a unit of text (typically a word)
Easy to convert text into DTM format using tidytext package

library(tidytext)
transcripts_tokens <- transcripts |>
  unnest_tokens(word, text)
head(transcripts_tokens)

# A tibble: 6 × 3
  index character word      
  <int> <chr>     <chr>     
1 16791 Stanley   this      
2 16791 Stanley   is        
3 16791 Stanley   ridiculous
4 16792 Phyllis   do        
5 16792 Phyllis   you       
6 16792 Phyllis   have

Remove stop words

tidytext provides stop_words data

head(stop_words)

# A tibble: 6 × 2
  word      lexicon
  <chr>     <chr>  
1 a         SMART  
2 a's       SMART  
3 able      SMART  
4 about     SMART  
5 above     SMART  
6 according SMART

Remove stop words from all tokens

transcripts_tokens <- transcripts_tokens |>
  filter(!(word %in% stop_words$word))
head(transcripts_tokens)

# A tibble: 6 × 3
  index character word      
  <int> <chr>     <chr>     
1 16791 Stanley   ridiculous
2 16792 Phyllis   idea      
3 16792 Phyllis   time      
4 16793 Michael   likes     
5 16793 Michael   late      
6 16793 Michael   plans

Visualize distribution of most common words

Code

transcripts_tokens |> 
  count(word, sort = TRUE) |> 
  filter(n > 5) |> 
  mutate(word = fct_reorder(word, n)) |> 
  ggplot(aes(n, word)) +
  geom_col() +
  labs(y = NULL)

Stemming

Use SnowballC package and wordStem() function to perform stemming

library(SnowballC)
transcripts_tokens <- transcripts_tokens |>
  mutate(stem = wordStem(word))
head(transcripts_tokens)

# A tibble: 6 × 4
  index character word       stem   
  <int> <chr>     <chr>      <chr>  
1 16791 Stanley   ridiculous ridicul
2 16792 Phyllis   idea       idea   
3 16792 Phyllis   time       time   
4 16793 Michael   likes      like   
5 16793 Michael   late       late   
6 16793 Michael   plans      plan

Tokenize by n-gram

An n-gram is a sequence of n adjacent items (e.g., words, letters, symbols, etc.)

Consider pairs of two consecutive words (“bigrams”) instead of single words

transcripts_bigrams <- transcripts |>
  unnest_tokens(bigram,text, token = "ngrams", n = 2) |> 
  filter(!is.na(bigram))
head(transcripts_bigrams)

# A tibble: 6 × 3
  index character bigram       
  <int> <chr>     <chr>        
1 16791 Stanley   this is      
2 16791 Stanley   is ridiculous
3 16792 Phyllis   do you       
4 16792 Phyllis   you have     
5 16792 Phyllis   have any     
6 16792 Phyllis   any idea

Remove stop words: separate() into two words, then filter(), then unite()

transcripts_bigrams <- transcripts_bigrams |> 
  separate(bigram, c("word1", "word2"), sep = " ") |> 
  filter(!word1 %in% stop_words$word,
         !word2 %in% stop_words$word) |> 
  unite(bigram, word1, word2, sep = " ")
head(transcripts_bigrams)

# A tibble: 6 × 3
  index character bigram             
  <int> <chr>     <chr>              
1 16793 Michael   plans tonight      
2 16795 Michael   remember mmm       
3 16795 Michael   call corporate     
4 16795 Michael   michael scott      
5 16795 Michael   scott scranton     
6 16795 Michael   overtime assignment

Can perform the same frequency analysis as for single words

Word clouds

A word cloud displays all words mentioned across documents, where more common words are larger
- First, compute the word counts \[w_{\cdot 1} = \sum_{i=1}^N w_{i1} \hspace{0.1in} \dots \hspace{0.1in} w_{\cdot J} = \sum_{i=1}^N w_{iJ}\]
- Then, the size of word \(j\) is proportional to \(w_{\cdot j}\)

Create word clouds in R using wordcloud package
wordcloud() function takes in two main inputs: words(vector of unique words) and freq (vector of frequencies)

Word clouds

Code

token_summary <- transcripts_tokens |>
  count(stem)
library(wordcloud)
wordcloud(words = token_summary$stem, 
          freq = token_summary$n, 
          random.order = FALSE, # place biggest words in center
          max.words = 100, 
          colors = brewer.pal(8, "Dark2"))

Comparison clouds

Goal: create word clouds to compare two different collections of documents \(\mathcal{A}\) and \(\mathcal{B}\)

Construct vectors of total words for each collection
- \(\mathbf{w}^{\mathcal{A}} = (w_{\cdot 1}^{\mathcal{A}}, \dots, w_{\cdot J}^{\mathcal{A}})\)
- \(\mathbf{w}^{\mathcal{B}} = (w_{\cdot 1}^{\mathcal{B}}, \dots, w_{\cdot J}^{\mathcal{B}})\)

Consider the \(j\)th word (say “dinner”)
- If \(w_{\cdot j}^{\mathcal{A}}\) is large, then “dinner” is large in the word cloud for \(\mathcal{A}\)
- If \(w_{\cdot j}^{\mathcal{B}}\) is large, then “dinner” is large in the word cloud for \(\mathcal{B}\)
- But if both are large, this does not indicate whether \(w_{\cdot j}^{\mathcal{A}}\) or \(w_{\cdot j}^{\mathcal{B}}\) is bigger

Comparison clouds

For word \(j\), compute \(\bar{w}_{\cdot j} = \textsf{average}(w_{\cdot j}^{\mathcal{A}}, w_{\cdot j}^{\mathcal{B}})\)

Compute \(w_{\cdot j}^{\mathcal{A}} - \bar{w}_{\cdot j}\) and \(w_{\cdot j}^{\mathcal{B}} - \bar{w}_{\cdot j}\)

If \(w_{\cdot j}^{\mathcal{A}} - \bar{w}_{\cdot j}\) is very positive, make it large for the \(\mathcal{A}\) word cloud
If \(w_{\cdot j}^{\mathcal{B}} - \bar{w}_{\cdot j}\) is very positive, make it large for the \(\mathcal{B}\) word cloud

Comparison clouds

Code

michael_jan_summary <- transcripts_tokens |>
  count(character, stem) |> 
  filter(character %in% c("Michael", "Jan"))

library(reshape2)
michael_jan_summary |>
  acast(stem ~ character, value.var = "n", fill = 0) |>
  comparison.cloud(colors = c("red", "blue"), max.words = 100, scale = c(2, 0.1), title.size = 2)

Term frequency-inverse document frequency (tf-idf)

Goal: compare text across documents (e.g., The Office characters)
Determine the most unique words for each document using tf-idf
Approach: downweight words that occur frequently across all documents

Inverse document frequency (idf): For word \(j\), compute

\[\textsf{idf}_j = \log \frac{N}{N_j},\] where \(N\) is number of documents and \(N_j\) is number of documents containing word \(j\)

Term frequency-inverse document frequency (tf-idf)

\[\textsf{tf-idf}_j = \textsf{tf}_j \times \textsf{idf}_j = w_{ij} \times \textsf{idf}_j\]

The higher the tf-idf value (unitless), the more unique the word is in the document

tf-idf example

Use bind_tf_idf() from tidytext package

transcripts_tfidf <- transcripts_tokens |>
  count(character, stem) |> 
  bind_tf_idf(stem, character, n) 
head(transcripts_tfidf)

# A tibble: 6 × 6
  character stem       n     tf   idf tf_idf
  <chr>     <chr>  <int>  <dbl> <dbl>  <dbl>
1 All       cheer      1 1      2.77  2.77  
2 Andy      anim       1 0.0476 2.77  0.132 
3 Andy      bet        1 0.0476 2.08  0.0990
4 Andy      capit      1 0.0476 2.77  0.132 
5 Andy      dinner     1 0.0476 0.981 0.0467
6 Andy      flower     2 0.0952 2.77  0.264

Visualize most unique words according to tf-idf

Code

transcripts_tfidf |>
  filter(character %in% c("Michael", "Jan", "Jim", "Pam")) |>
  group_by(character) |>
  slice_max(tf_idf, n = 7, with_ties = FALSE) |>
  mutate(stem = reorder_within(stem, tf_idf, character)) |>
  ggplot(aes(x = tf_idf, y = stem)) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ character, ncol = 2, scales = "free") +
  labs(x = "tf-idf", y = NULL)

Sentiment analysis

Goal: understand the emotional intent or sentiment of words

Lexicon-based approach
- Find a sentiment lexicon/dictionary
- Compute summary (e.g., average sentiment)

Different types of sentiment lexicons
- Positive or negative
- Numerical score (e.g., from −5 to +5)
- Specific emotions (e.g., sadness, anger, joy, etc.)

Sentiment analysis example

Consider the Bing sentiment lexicon
(positive/negative)

bing_sentiment <- get_sentiments("bing")
head(bing_sentiment)

# A tibble: 6 × 2
  word       sentiment
  <chr>      <chr>    
1 2-faces    negative 
2 abnormal   negative 
3 abolish    negative 
4 abominable negative 
5 abominably negative 
6 abominate  negative

Join sentiment to token table

transcripts_sentiment <- transcripts_tokens |> 
  inner_join(bing_sentiment)
head(transcripts_sentiment)

# A tibble: 6 × 5
  index character word       stem    sentiment
  <int> <chr>     <chr>      <chr>   <chr>    
1 16791 Stanley   ridiculous ridicul negative 
2 16793 Michael   likes      like    positive 
3 16795 Michael   mad        mad     negative 
4 16795 Michael   fair       fair    positive 
5 16795 Michael   enjoy      enjoi   positive 
6 16797 Michael   happy      happi   positive

Make different graphics to visualize sentiment

Sentiment word clouds

Code

transcripts_sentiment |> 
  count(word, sentiment) |> 
  acast(word ~ sentiment, value.var = "n", fill = 0) |> 
  comparison.cloud(colors = c("midnightblue", "red"), max.words = 100, scale = c(2, 0.5), title.size = 1)

Graphics replication

See this post