Introduction

This tutorial shows how to convert PDFs to simple txt (editor) files. The R Notebook for this tutorial can be downloaded here.

Preparation and session set up

This tutorial is based on R. If you have not installed R or are new to it, you will find an introduction to and more information how to use R here. For this tutorials, we need to install certain packages from an R library so that the scripts shown below are executed without errors. Before turning to the code below, please install the packages by running the code below this paragraph. If you have already installed the packages mentioned below, then you can skip ahead ignore this section. To install the necessary packages, simply run the following code - it may take some time (between 1 and 5 minutes to install all of the libraries so you do not need to worry if it takes some time).

# clean current workspace
rm(list=ls(all=T))
# set options
options(stringsAsFactors = F)
# install libraries
install.packages(c("pdftools", "dplyr", "stringr", "httr", "jsonlite"))

Once you have installed R-Studio and have also initiated the session by executing the code shown above, you are good to go.

How to use the R Notebook for this tutorial

To follow this tutorial interactively (by using the R Notebook), follow the instructions listed below.

  1. Create a folder somewhere on your computer
  2. Download the R Notebook and save it in the folder you have just created
  3. Open R Studio
  4. Click on File in the upper left corner of the R Studio interface
  5. Click on New Project...
  6. Select Existing Directory
  7. Browse to the folder you have just created and click on Open
  8. Now click on Files above the lower right panel
  9. Click on the file convertpdf2txt.Rmd
  • The Markdown file of this tutorial should now be open in the upper left panel of R Studio. To execute the code which prepare the session, load the data, create the graphs, and perform the statistics, simply click on the green arrows in the top right corner of the code boxes.
  • To render a PDF of this tutorial, simply click on Knit above the upper left panel in R Studio.

Converting PDFs into txt files

Now, we load the packages and inspect the data.

# activate packages
library(pdftools)
library(dplyr)
library(stringr)

Next, we define a path and convert the pdf that is located the path into a txt.

# you can use an url or a path that leads to a pdf dcument
pdf_path <- "https://slcladal.github.io/data/PDFs/pdf0.pdf"
# extract text
txt_output <- pdftools::pdf_text(pdf_path) %>%
  paste(sep = " ") %>%
  stringr::str_replace_all(fixed("\n"), " ") %>%
  stringr::str_replace_all(fixed("\r"), " ") %>%
  stringr::str_replace_all(fixed("\t"), " ") %>%
  stringr::str_replace_all(fixed("\""), " ") %>%
  paste(sep = " ", collapse = " ") %>%
  stringr::str_squish() %>%
  stringr::str_replace_all("- ", "") 
# inspect
str(txt_output)
##  chr "Corpus linguistics Wikipedia https://en.wikipedia.org/wiki/Corpus_linguistics Corpus linguistics Corpus linguis"| __truncated__

To convert many pdf-files, we write a function that preforms the conversion for many documents.

convertpdf2txt <- function(dirpath){
  files <- list.files(dirpath, full.names = T)
  x <- sapply(files, function(x){
  x <- pdftools::pdf_text(x) %>%
  paste(sep = " ") %>%
  stringr::str_replace_all(fixed("\n"), " ") %>%
  stringr::str_replace_all(fixed("\r"), " ") %>%
  stringr::str_replace_all(fixed("\t"), " ") %>%
  stringr::str_replace_all(fixed("\""), " ") %>%
  paste(sep = " ", collapse = " ") %>%
  stringr::str_squish() %>%
  stringr::str_replace_all("- ", "") 
  return(x)
    })
}

We can now apply the function to the folder in which we have stored the PDFs. The output is a vector with the texts of the PDFs.

# apply function
txts <- convertpdf2txt("data/PDFs/")
# inspect the structure of the txts element
str(txts)
##  Named chr [1:4] "Corpus linguistics Wikipedia https://en.wikipedia.org/wiki/Corpus_linguistics Corpus linguistics Corpus linguis"| __truncated__ ...
##  - attr(*, "names")= chr [1:4] "data/PDFs/pdf0.pdf" "data/PDFs/pdf1.pdf" "data/PDFs/pdf2.pdf" "data/PDFs/pdf3.pdf"

The output of the str() function shows that we have converted 4 PDFs into txt-files.

Saving the txt-files on your computer

To save the txt-files on your disc, simply replace the predefined location (“D:\Uni\UQ\SLC\LADAL\SLCLADAL.github.io\data/”) with the folder where you want to store the txt files and then execute the code below. Also, we will name the txts-elements as text plus their number.

# add names to txt files
names(txts) <- paste("text", 1:length(txts), sep = "")
# save result to disc
lapply(seq_along(txts), function(i)writeLines(text = unlist(txts[i]),
    con = paste("D:\\Uni\\UQ\\SLC\\LADAL\\SLCLADAL.github.io\\data/", names(txts)[i],".txt", sep = "")))

Citation & Session Info

Schweinberger, Martin. 2020. Converting PDFs to txt files with R. Brisbane: The University of Queensland. url: https://slcladal.github.io/convertpdf2txt.html (Version 2020.09.29).

@manual{schweinberger2020conv,
  author = {Schweinberger, Martin},
  title = {Converting PDFs to txt files with R},
  note = {https://slcladal.github.io/convertpdf2txt.html},
  year = {2020},
  organization = "The University of Queensland, Australia. School of Languages and Cultures},
  address = {Brisbane},
  edition = {2020/09/29}
}
sessionInfo()
## R version 4.0.2 (2020-06-22)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 18362)
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252   
## [3] LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
## [5] LC_TIME=German_Germany.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] stringr_1.4.0  dplyr_1.0.2    pdftools_2.3.1
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.5       knitr_1.30       magrittr_1.5     tidyselect_1.1.0
##  [5] R6_2.4.1         rlang_0.4.7      tools_4.0.2      xfun_0.16       
##  [9] htmltools_0.5.0  askpass_1.1      ellipsis_0.3.1   yaml_2.2.1      
## [13] digest_0.6.25    tibble_3.0.3     lifecycle_0.2.0  qpdf_1.1        
## [17] crayon_1.3.4     purrr_0.3.4      vctrs_0.3.4      glue_1.4.2      
## [21] evaluate_0.14    rmarkdown_2.3    stringi_1.5.3    compiler_4.0.2  
## [25] pillar_1.4.6     generics_0.0.2   pkgconfig_2.0.3

Main page


References