Data Management
Martin Schweinberger
2022-10-10
Introduction
This tutorial introduces basic data management techniques, version control measures, and issues relating to reproducible research.
This tutorial showsbeginners and intermediate users of R how to improve the reproducibility and transparency of your analyses generally but also specifically when working in RStudio. The aim is not to provide a fully-fledged analysis but rather to show and exemplify selected useful methods associated with reproducibility.
The entire R Notebook for the tutorial can be downloaded here. If you want to render the R Notebook on your machine, i.e. knitting the document to html or a pdf, you need to make sure that you have R and RStudio installed and you also need to download the bibliography file and store it in the same folder where you store the Rmd file.
In data science or when working with data, issues relating to organizing files and folders, managing data and projects, avoiding data loss, and efficient work flows are essential. The idea behind this tutorial is to address this issue and provide advice on how to handle data and creating efficient workflows.
Some of the contents of this tutorial build on the Digital Essentials module that is offered by the UQ library, the Reproducible Research resources (created by Griffith University’s Library and eResearch Services), and on Amanda Miotto’s Reproducible Reseach Things (see also here for an alternative version and here for the GitHub repo). You can find additional information on all things relating to computers, the digital world, and computer safety in the Digital Essentials course that is part of UQ’s library resources.
1 Data Management
This section introduces some basic concepts and provides useful tips for managing your research data.
Folder structure
The UQ Library offers the Digital Essentials module Working with Files. This module contains information on storage options, naming conventions, back up options, metadata, and file formats. Some of these issues are dealt with below but the materials provided by the library offer a more extensive introduction into these topics.
All of the ways to organize your folders have different advantages and problems but they all have in common that they rely on a tree-structure - more general folders contain more specialized ones. For example, if you want to find any file with as few clicks as possible, an alphabetical folder structure would be a good solution. Organized in this way, everything that starts with a certain letter will be stored by its initial letter (e.g. everything starting with a t such as travel under T or everything related to your courses under C). However, organizing your data alphabetically is not intuitive and completely unrelated topics will be located in the same folder.
A more common and intuitive way to organize your data is to separate your data into meaningful aspects of your life such as Work (containing, e.g., teaching and research), Living (including rent, finances, and insurances), and Media (including Movies, Music, and Audiobooks).
Naming conventions
A File Naming Convention (FNC) is a framework or protocol if you like for naming your files in a way that describes what files contain and importantly, how they relate to other files. It is essential prior to collecting data to establish an agreed FNC.
Folders (as well as files) should be labeled in a meaningful way. This means that you avoid names like Stuff or Document for folders and doc2 or homework for files.
Naming files consistently, logically and in a predictable manner will prevent against unorganized files, misplaced or lost data. It could also prevent possible backlogs or project delays. A file naming convention will ensure files are:
Easier to process - All team members won’t have to over think the file naming process
Easier to facilitate access, retrieval and storage of files
Easier to browse through files saving time and effort
Harder to lose!
Check for obsolete or duplicate records
The University of Edinburgh has a comprehensive and easy to follow list (with examples and explanations) of 13 Rules for file naming conventions. You can also use the recommendations of the
Australian National Data Services (ANDS) guide of file wrangling. Some of the suggestions are summarized below.
Like the different conventions for folders, there are different conventions for files. As a general rule, any special character symbols, such as +, !, ", -, ., ö, ü, ä, %, &, (, ), [, ], &, $, =, ?, ’, #, or / but also including white spaces, should be avoided (while _ is also a special character, it is still common practice to include them in file names for readability). One reason for this is that you will may encounter problems when sharing files if you avoid white spaces and special characters. Also, some software applications automatically replace or collapse white spaces. A common practice to avoid this issue is to capitalize initial letters in file names which allows you avoid white spaces. An example would be TutorialIntroComputerSkills or Tutorial_IntroComputerSkills.
When you want to include time-stamps in file names, the best way to do this is to use the YYYYMMDD format (rather than DDMMYYYY or even D.M.YY). The reason is that if dates are added this way, the files can be easily sorted in ascending or descending order. To elaborate on the examples shown before, we may use TutorialIntroComputerSkills20200413 or Tutorial_IntroComputerSkills_20200413
GOING FURTHER
For Beginners
- Let’s look at some easy naming convention for your data files and documents. Any dates are best stored with YYYY-MM-DD. Try to avoid spaces in your file names
For Intermediates
- Make sure you follow 13 Rules for file naming conventions
For Advanced file namers
- Do you have a policy in your team/section/cluster around naming conventions? If not, this is a great way of getting everyone on the same page.
Documentation and the Bus Factor
Documentation is the idea of documenting your work so that outsiders (or yourself after a long time) can understand what you did and how you did it. As a general rule, you should document where to find what as if you informed someone else how to find stuff on your computer.
Documentation can include where your results and data are saved but it can also go far beyond this. Documentation does not have to be complex and it can come in different forms, depending on your needs and your work. Documenting what you do can also include photos, word documents with descriptions, or websites that detail how you work.
The idea behind documentation is to keep a log of the contents of folders so that you at a later point in time or someone else can continue your work in case you or someone else is run over by a bus (hence the Bus Factor).
In fact, documentation is all about changing the Bus Factor - how many people on a project would need to be hit by a bus to make a project fail. Many times, projects can have a bus factor of one. Adding documentation means when someone goes on leave, needs to take leave suddenly or finishes their study, their work is preserved for your project.
If you work in a collaborative project, it is especially useful to have a log of where one can find relevant information and who to ask for help with what. Ideally you want to document anything that someone coming on board would need to know. Thus, if you have not created a log for on-boarding, the perfect person to create a log would be the last person who joined the project. Although there is no fixed rule, it is recommendable to store the log either as a ReadMe document or in a ReadMe folder on the top level of the project.
GOING FURTHER
For Beginners
Read this first: How to start Documenting and more by CESSDA ERIC
Start with documenting in a text file or document- any start is a good start
Have this document automatically synced to the cloud with your data or keep this in a shared place such as Google docs, Microsoft teams, or Owncloud. If you collaborate on a project and use UQ’s RDM, you should store a copy of your documentation there.
For Intermediates
- Once you have the basics in place, go into detail on how your workflow goes from your raw data to the finished results. This can be anything from a detailed description of how you analyse your data, over R Notebooks, to downloaded function lists from Virtual Lab.
For Advanced documentarians
- Learn about Git Repositories and wikis.
Keep copies
Keeping a copy of all your data (working, raw and completed) in the cloud is incredibly important. This ensures that if you have a computer failure, accidentally delete your data or your data is corrupted, your research is recoverable and restorable.
The 3-2-1 backup rule
Try to have at least three copies of your project that are in different locations. The rule is: keep at least three (3) copies of your data, and store backup-copies on two (2) different storage media , with one (1) of them located offsite. Although this is just a personal preference, I always safe my projects
on my personal notebook
on at least one additional hard drive (that you keep in a secure location)
in an online repository (for example, UQ’s Research Data Management system (RDM) OneDrive, MyDrive, GitHub, or GitLab)
Using online repositories ensures that you do not lose any data in case your computer crashes (or in case you spill lemonade over it - don’t ask…) but it comes at the cost that your data can be more accessible to (criminal or other) third parties. Thus, if you are dealing with sensitive data, I suggest to store it on an additional external hard drive and do not keep cloud-based back-ups. If you trust tech companies with your data (or think that they are not interested in stealing your data), cloud-based solutions such as OneDrive, Google’s MyDrive, or Dropbox are ideal and easy to use options (however, UQ’s RDM is a safer option).
The UQ library also offers additional information on complying with ARC and NHMRC data management plan requirements and that UQ RDM meets these requirements for sensitive data (see here).
GOING FURTHER
For Beginners
- Get your data into UQ’s RDM or Cloud Storage - If you need help, talk to the library or your tech/eResearch/QCIF Support
For Advanced backupers
- Build a policy for your team or group on where things are stored. Make sure the location of your data is saved in your documentation
Avoid duplicates
Many of the files on our computers have several duplicates or copies on the same machine. Optimally, each file should only be stored once (on one machine). Thus, to minimize the superfluous files on your computer, we can delete any duplicates of files.
You can, of course, do that manually but a better way to do this is to use programs that detect files that are identical in name, file size, and date of creation. One of these programs is the Douplicate Cleaner. A tutorial on how to use it can be found here.
How to organize projects
This section focuses on how to organize your projects and how to use your computer in doing so.
Project folder design
Having a standard folder structure can keep your files neat and tidy and save you time looking for data. It can also help if you are sharing files with colleagues and having a standard place to put working data and documentation.
Store your projects in a separate folder. For instance, if you are creating a folder for a research project, create the project folder within a separate project folder that is within a research folder. If you are creating a folder for a course, create the course folder within a courses folder within a teaching folder, etc.
Whenever you create a folder for a new project, try to have a set of standard folders. For example, when I create research project folders, I always have folders called archive, data, docs, and images. When I create course folders, I always have folders called slides, assignments, exam, studentmaterials, and correspondence. However, you are, of course, free to modify and come up or create your own basic project design. Also, by prefixing the folder names with numbers, you can force your files to be ordered by the steps in your workflow.
- Having different sub folders allows you to avoid having too many files and many different file types in a single folder. Folders with many different files and file types tend to be chaotic and can be confusing. In addition, I have one ReadMe file on the highest level (which only contains folders except for this one single file) in which I describe very briefly what the folder is about and which folders contain which documents as well as some basic information about the folder (e.g. why and when I created it). This ReadMe file is intended both for me as a reminder what this folder is about and what it contains but also for others in case I hand the project over to someone else who continues the project or for someone who takes over my course and needs to use my materials.
If you work in a team or share files and folders regularly, it makes sense to develop a logical structure for your team, you need to consider the following points:
Check to make sure there are no pre-existing folder structure agreements
Name folders appropriately and in a meaningful manner. Don’t use staff names and consider using the type of work
Consistency - make sure you use the agreed structure/hierarchy
Structure folders hierarchically - start with a limited number of folders for the broader topics, and then create more specific folders within these
Separate ongoing and completed work - as you start to create lots of folders and files, it is a good idea to start thinking about separating your older documents from those you are currently working on
Backup – ensure folders and files are backed up and retrievable in the event of a disaster. UQ like most universities, have safe storage solutions.
Clean up folders and files post project.
GOING FURTHER
For Beginners
Pick some of your projects and illustrate how you currently organize your files. See if you can devise a better naming convention or note one or two improvements you could make to how you name your files.
There are some really good folder template shapes around. Here is one you can download.
For Advanced Folder designers
- Come up with a policy for your team for folder structures. You could create a template and put it in a downloadable location for them to get them started.
How to organize data
This section will elaborate on who to organize and handle (research) data and introduce some basic principles that may help you to keep your data tidy.
Never delete data!
- The first thing to keep in mind when dealing with data is to keep at least one copy of the original data. The original data should never be deleted but, rather, you should copy the data and delete only sections of the copy while retaining the original data intact.
Tips for sensitive data
- Sensitive data are data that can be used to identify an individual, species, object, or location that introduces a risk of discrimination, harm, or unwanted attention. Major, familiar categories of sensitive data are: personal data - health and medical data - ecological data that may place vulnerable species at risk.
Separating identifying variables from your data
Separating or deidentifying your data has the purpose to protect an individual’s privacy. According to the Australian Privacy Act 1988, “personal information is deidentified if the information is no longer about an identifiable individual or an individual who is reasonably identifiable”. Deidentified information is no longer considered personal information and can be shared. More information on the Commonwealth Privacy Act can be located here.
Deidentifying aims to allow data to be used by others for publishing, sharing and reuse without the possibility of individuals/location being re-identified. It may also be used to protect the location of archaeological findings, cultural data of location of endangered species.
Any identifiers (name, date of birth, address or geospatial locations etc) should be removed from main data set and replaced with a code/key. The code/key is then preferably encrypted and stored separately. By storing deidentified data in a secure solution, you are meeting safety, controlled, ethical, privacy and funding agency requirements.
Re-identifying an individual is possible by recombining the deidentifiable data set and the identifiers.
Tips for managing deidentification (ARDC)
Plan deidentification early in the research as part of your data management planning
Retain original unedited versions of data for use within the research team and for preservation
Create a deidentification log of all replacements, aggregations or removals made
Store the log separately from the deidentified data files
Identify replacements in text in a meaningful way, e.g. in transcribed interviews indicate replaced text with [brackets] or use XML mark-up tags.
Management of identifiable data (ARDC)
Data may often need to be identifiable (i.e. contains personal information) during the process of research, e.g. for analysis. If data is identifiable then ethical and privacy requirements can be met through access control and data security. This may take the form of:
Control of access through physical or digital means (e.g. passwords)
Encryption of data, particularly if it is being moved between locations
Ensuring data is not stored in an identifiable and unencrypted format when on easily lost items such as USB keys, laptops and external hard drives.
Taking reasonable actions to prevent the inadvertent disclosure, release or loss of sensitive personal information.
Safely sharing sensitive data guide (ARDC)
Australian practical guidance for Deidentification (ARDC)
- Australian Research Data Commons (ARDC) formerly known as Australian National Data Service (ANDS) released a fabulous guide on Deidentification. The Deidentification guide is intended for researchers who own a data set and wish to share safely with fellow researchers or for publishing of data. The guide can be located here.
Nationally available guidelines for handling sensitive data
The Australian Government’s Office of the Australian Information Commissioner (OAIC) and CSIRO Data61 have released a Deidentification Decision Making Framework, which is a “practical guide to deidentification, focusing on operational advice”. The guide will assist organisations that handle personal information to deidentify their data effectively.
The OAIC also provides high-level guidance on deidentification of data and information, outlining what deidentification is, and how it can be achieved.
The Australian Government’s guidelines for the disclosure of health information, includes techniques for making a data set non-identifiable and example case studies.
Australian Bureau of Statistics’ National Statistical Service Handbook. Chapter 11 contains a summary of methods to maintain privacy.
med.data.edu.au gives information about anonymisation
The Office of the Information Commissioner Queensland’s guidance on deidentification techniques can be found here
Data as publications
More recently, regarding data as a form of publications has gain a lot of traction. This has the advantage that it rewards researchers who put a lot of work into compiling data and it has created an incentive for making data available, e.g. for replication. The UQ RDM and UQ eSpace can help with the process of publishing a dataset.
There are many platforms where data can be published and made available in a sustainable manner. Below are listed just some options that are recommendable:
UQ Research Data Manager
The UQ Research Data Manager (RDM) system is a robust, world-leading system designed and developed here at UQ. The UQ RDM provides the UQ research community with a collaborative, safe and secure large-scale storage facility to practice good stewardship of research data. The European Commission report “Turning FAIR into Reality” cites UQ’s RDM as an exemplar of, and approach to, good research data management practice. The disadvantage of RDM is that it is not available to everybody but restricted to UQ staff, affiliates, and collaborators.
Open Science Foundation
The Open Science Foundation (OSF) is a free, global open platform to support your research and enable collaboration.
TROLLing
TROLLing | DataverseNO (The Tromsø Repository of Language and Linguistics) is a repository of data, code, and other related materials used in linguistic research. The repository is open access, which means that all information is available to everyone. All postings are accompanied by searchable metadata that identify the researchers, the languages and linguistic phenomena involved, the statistical methods applied, and scholarly publications based on the data (where relevant).
Git
GitHub offers the distributed version control using Git. While GitHub is not designed to host research data, it can be used to share share small collections of research data and make them available to the public. The size restrictions and the fact that GitHub is a commercial enterprise owned by Microsoft are disadvantages of this as well as alternative, but comparable platforms such as GitLab.
Digital Object Identifier (DOI) and Persistent identifier (PiD)
Once you’ve completed your project, help make your research data discoverable, accessible and possibly re-usable using a PiD such as a DOI! A Digital Object Identifier (DOI) is a unique alphanumeric string assigned by either a publisher, organisation or agency that identifies content and provides a PERSISTENT link to its location on the internet, whether the object is digital or physical. It might look something like this http://dx.doi.org/10.4225/01/4F8E15A1B4D89.
DOIs are also considered a type of persistent identifiers (PiDs). An identifier is any label used to name some thing uniquely (whether digital or physical). URLs are an example of an identifier. So are serial numbers, and personal names. A persistent identifier is guaranteed to be managed and kept up to date over a defined time period.
Journal publishers assign DOIs to electronic copies of individual articles. DOIs can also be assigned by an organisation, research institutes or agencies and are generally managed by the relevant organisation and relevant policies. DOIs not only uniquely identify research data collections, it also supports citation and citation metrics.
A DOI will also be given to any data set published in UQ eSpace, whether added manually or uploaded from UQ RDM. For information on how cite data, have a look here.
Key points
DOIs are a persistent identifier and as such carry expectations of curation, persistent access and rich metadata
DOIs can be created for DATA SETS and associated outputs (e.g. grey literature, workflows, algorithms, software etc) - DOIs for data are equivalent with DOIs for other scholarly publications
DOIs enable accurate data citation and bibliometrics (both metrics and altmetrics)
Resolvable DOIs provide easy online access to research data for discovery, attribution and reuse
GOING FURTHER
For Beginners
- Ensure data you associate with a publication has a DOI- your library is the best group to talk to for this.
For Intermediates
Learn more about how your DOI can potentially increase your citation rates by watching this 4m:51s video
Learn more about how your DOI can potentially increase your citation rate by reading the ANDS Data Citation Guide
For Advanced identifiers
Learn more about PiDs and DOIs
Contact the Library for advice on how to obtain a DOI upon project completion.
Have a look at ANDS/ARDC - Citation and Identifiers
Check out the DOI system for research data
2 Optimizing workflows in RStudio
Most researchers in the language science are familiar to R to some extend. However, while workshops on how to perform particular tasks in R, such as how to do statistical analyses in R, are becoming more widespread, there rarely exist any workshops on how to work in R.
By working in R, I am referring to issues relating to reproducible and effective work flows and session management but also to formatting conventions for R code. The following sections will provide some guidance on how to optimize your work flows and how to make your research practices more transparent, efficient, and reproducible.
Rproj
If you’re using RStudio, you have the option of creating a new R project. A project is simply a working directory designated with a .RProj file. When you open a project (using File/Open Project in RStudio or by double–clicking on the .RProj file outside of R), the working directory will automatically be set to the directory that the .RProj file is located in.
I recommend creating a new R Project whenever you are starting a new research project. Once you’ve created a new R project, you should immediately create folders in the directory which will contain your R code, data files, notes, and other material relevant to your project (you can do this outside of R on your computer, or in the Files window of RStudio). For example, you could create a folder called R that contains all of your R code, a folder called data that contains all your data (etc.).
Before working with Rprojects, I set my working directories with setwd() but this is not optimal because it takes an absolute file path as an input then sets it as the current working directory of the R process. However, this makes scripts break easily and it makes it more difficult to share my analyses and projects with others. Hence, the setwd() approach makes it very difficult to share work and make it transparent and reproducible.
Folder structure
Like with all (project) folders, it is recommendable to use a common folder structure as to make it easier to find information or particular files.
Solving dependency issues: renv
The renv package is a new way to make Rprojects independent and thus to remove outside dependencies from Rprojects. The renv package creates a new library within your project so that your Rproject is independent of your personal library and when you share your project, you automatically also share the packages that you have used. The idea behind renv is to be a robust, stable replacement for the packrat package which was rather unsatisfactory to work with (speaking from experience here).
I have used renv and, while it took some time to generate the local library on first use, it was very easy to use, did not cause any issues and is overall very recommendable to get rid of outside dependencies and thus make Rprojects easier to share for transparency and reproducibility.
Underlying the philosophy of renv is that all existing work flows should just work as they did before – renv helps to isolate your project’s R dependencies (like package versioning).
You can get more information about renv and how it works, as well as how you can use it here.
Version Control with Git
Getting started with Git
To connect your Rproject with GitHub, you need to have Git installed (if you have not yet downloaded and installed Git, simply search for download git in your favorite search engine and follow the instructions) and you need to have a GitHub account. If you do not have a GitHub account, here is a video tutorial showing how you can do this. If you have trouble with this, you can also check out Happy Git and GitHub for the useR at happygitwithr.com.
Just as a word of warning: when I set up my connection to Git and GitHUb things worked very differently, so things may be a bit different on your machine. In any case, I highly recommend this YouTube tutorial which shows how to connect to Git and GitHub using the usethis package or this, slightly older, YouTube tutorial on how to get going with Git and GitHub when working with RStudio.
Old school
While many people use the usethis package to connect RStudio to GitHub, I still use a somewhat old school way to connect my projects with GitHub. I have decided to show you how to connect RStudio to GitHub using this method, as I actually think it is easier once you have installed Git and created a gitHub account.
Before you can use Git with R, you need to tell RStudio that you want to use Git. To do this, go to Tools, then Global Options and then to Git/SVN and make sure that the box labeled Enable version control interface for RStudio projects. is checked. You need to then browse to the Git executable file (for Window’s users this is called the Git.exe file).
Now, we can connect our project to Git (not to GitHub yet). To do this, go to Tools, then to Project Options... and in the Git/SVN tab, change Version Control System to Git (from None). Once you have accepted these changes, a new tab called Git appears in the upper right pane of RStudio (you may need to / be asked to restart RStudio at this point). You can now commit files and changes in files to Git.
To commit files, go to the Git tab and check the boxes next to the files you would like to commit (this is called staging; meaning that these files are now ready to be committed). Then, click on Commit and enter a message in the pop-up window that appears. Finally, click on the commit button under the message window.
Connecting your Rproj with GitHub
To connect your Rproject and GitHub, we go to our GitHub page and create a new GitHub repository (repo) that we call test (or whatever you want to call your repository). To create a new repository on GitHub, simply click on New Repository after you have clicked on the New icon. I recommend that you check Add a Readme in which you can describe what the repo contains, but you do not have to do this.
Once you have created a new GitHub repo, we need to connect this repo to our computer. To do this, we need to clone the repo which we do by clicking on the green Code icon. When you click on the green Code icon, a dropdown menu appears and you copy the url in the section clone with HTTPS.
Then, we go to the Terminal (in-between Console and Jobs) and we include the path we got from the git repository after the command git remote add origin. We then use the command git branch -M main (to merge a master and a main branch) and then, finally, push our files into the remote GitHub repo by using the command git push -u origin main.
# initiate the upstream tracking of the project on the GitHub repo
git remote add origin https://github.com/YourGitHUbUserName/YouGitHubRepositoryName.git
# set main as main branch (rather than master)
git branch -M main
# push content to main
git push -u origin mainWe can then commit changes and push them to the remote GitHub repo.
You can then go to your GitHub repo and check if the documents that we pushed are now in the remote repo.
From now on, you can simply commit all changes that you make to the GitHub repo associated with that Rproject. Other projects can, of course, be connected and push to other GitHub repos.
Solving path issues: here
The goal of the here package is to enable easy file referencing in project-oriented workflows. In contrast to using setwd(), which is fragile and dependent on the way you organize your files, here uses the top-level directory of a project to easily build paths to files.
This makes your projects more robust as the paths will still work if you put your project into another location or on another computer. Also, moving between Mac and Windows (which would require different kind of path specifications) is no longer an issue with the here package.
# define path
example_path_full <- "D:\\Uni\\Konferenzen\\ISLE\\ISLE_2021\\isle6_reprows/repro.Rmd"
# show path
example_path_full## [1] "D:\\Uni\\Konferenzen\\ISLE\\ISLE_2021\\isle6_reprows/repro.Rmd"With the here package, the path starts in folder where the Rproj file is. As the Rmd file is in the same folder, we only need to specify the Rmd file and the here package will add the rest.
# load package
library(here)
# define path using here
example_path_here <- here::here("repro.Rmd")
#show path
example_path_here## [1] "/home/sam/programming/SLCLADAL.github.io/repro.Rmd"Reproducible randomness: set.seed
The set.seed function in R sets the seed of R‘s random number generator, which is useful for creating simulations or random objects that can be reproduced. This means that when you call a function that uses some form of randomness (e.g. when using random forests), using the set.seed function allows you to replicate results.
Below is an example of what I mean. First, we generate a random sample from a vector.
numbers <- 1:10
randomnumbers1 <- sample(numbers, 10)
randomnumbers1## [1] 7 6 5 10 3 9 8 2 1 4We now draw another random sample using the same sample call.
randomnumbers2 <- sample(numbers, 10)
randomnumbers2## [1] 10 6 2 3 5 8 1 4 9 7As you can see, we now have a different string of numbers although we used the same call. However, when we set the seed and then generate a string of numbers as show below, we create a reproducible random sample.
set.seed(123)
randomnumbers3 <- sample(numbers, 10)
randomnumbers3## [1] 3 10 2 8 6 9 1 7 5 4To show that we can reproduce this sample, we call the same seed and then generate another random sample which will be the same as the previous one because we have set the seed.
set.seed(123)
randomnumbers4 <- sample(numbers, 10)
randomnumbers4## [1] 3 10 2 8 6 9 1 7 5 4R Markdown
R Markdown provides an authoring framework for data science. You can use a single R Markdown file to both
save and execute code
generate high quality reports that can be shared with an audience
R Markdown documents are fully reproducible and support dozens of static and dynamic output formats. This 1-minute video provides a quick tour of what’s possible with R Markdown:
Check out this introduction to R Markdown by RStudio and have a look at this R Markdown cheat sheet.
Tidy data principles
The same (underlying) data can be represented in multiple ways. The following three tables are show the same data but in different ways.
| country | continent | 2002 | 2007 |
|---|---|---|---|
| Afghanistan | Asia | 42.129 | 43.828 |
| Australia | Oceania | 80.370 | 81.235 |
| China | Asia | 72.028 | 72.961 |
| Germany | Europe | 78.670 | 79.406 |
| Tanzania | Africa | 49.651 | 52.517 |
| year | Afghanistan (Asia) | Australia (Oceania) | China (Asia) | Germany (Europe) | Tanzania (Africa) |
|---|---|---|---|---|---|
| 2002 | 42.129 | 80.370 | 72.028 | 78.670 | 49.651 |
| 2007 | 43.828 | 81.235 | 72.961 | 79.406 | 52.517 |
| country | year | continent | lifeExp |
|---|---|---|---|
| Afghanistan | 2002 | Asia | 42.129 |
| Afghanistan | 2007 | Asia | 43.828 |
| Australia | 2002 | Oceania | 80.370 |
| Australia | 2007 | Oceania | 81.235 |
| China | 2002 | Asia | 72.028 |
| China | 2007 | Asia | 72.961 |
| Germany | 2002 | Europe | 78.670 |
| Germany | 2007 | Europe | 79.406 |
| Tanzania | 2002 | Africa | 49.651 |
| Tanzania | 2007 | Africa | 52.517 |
Table 3 should be the easiest to parse and understand. This is so because only Table 3 is tidy. Unfortunately, however, most data that you will encounter will be untidy. There are two main reasons:
Most people aren’t familiar with the principles of tidy data, and it’s hard to derive them yourself unless you spend a lot of time working with data.
Data is often organised to facilitate some use other than analysis. For example, data is often organised to make entry as easy as possible.
This means that for most real analyses, you’ll need to do some tidying. The first step is always to figure out what the variables and observations are. Sometimes this is easy; other times you’ll need to consult with the people who originally generated the data. The second step is to resolve one of two common problems:
One variable might be spread across multiple columns.
One observation might be scattered across multiple rows.
To avoid structuring in ways that make it harder to parse, there are three interrelated principles which make a data set tidy:
Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.
An additional advantage of tidy data is that is can be transformed more easily into any other format when needed.
How to minimize storage space
Most people use or rely on data that comes in spreadsheets and use software such as Microsoft Excel or OpenOffice Calc. However, spreadsheets produced by these software applications take up a lot of storage space.
One way to minimize the space, that your data takes up, is to copy the data and paste it into a simple editor or txt-file. The good thing about txt files is that they take up only very little space and they can be viewed easily so that you can open the file to see what the data looks like. You could then delete the spread sheet because you can copy and paste the content of the txt file right back into a spread sheet when you need it.
If you work with R, you may also consider to save your data as .rda files which is a very efficient way to save and story data in an R environment
Below is an example for how you can load, process, and save your data as .rda in RStudio.
# load data
lmm <- read.delim("https://slcladal.github.io/data/lmmdata.txt", header = TRUE)
# convert strings to factors
lmm <- lmm %>%
mutate(Genre = factor(Genre),
Text = factor(Text),
Region = factor(Region))
# save data
base::saveRDS(lmm, file = here::here("data", "lmm_out.rda"))
# remove lmm object
rm(lmm)
# load .rda data
lmm <- base::readRDS(file = here::here("data", "lmm_out.rda"))
# or from web
lmm <- base::readRDS(url("https://slcladal.github.io/data/lmm.rda", "rb"))
# inspect data
str(lmm)## 'data.frame': 537 obs. of 5 variables:
## $ Date : int 1736 1711 1808 1878 1743 1908 1906 1897 1785 1776 ...
## $ Genre : Factor w/ 16 levels "Bible","Biography",..: 13 4 10 4 4 4 3 9 9 3 ...
## $ Text : Factor w/ 271 levels "abott","albin",..: 2 6 12 16 17 20 20 24 26 27 ...
## $ Prepositions: num 166 140 131 151 146 ...
## $ Region : Factor w/ 2 levels "North","South": 1 1 1 1 1 1 1 1 1 1 ...To compare, the lmmdata.txt requires 19.2KB while the lmmdata.rda only requires 5.2KB (and only 4.1KB with xz compression). If stored as an Excel spreadsheet, the same file requires 28.6KB.
Citation & Session Info
Schweinberger, Martin. 2022. Data Management, Version Control, and Reproducibility. Brisbane: The University of Queensland. url: https://slcladal.github.io/repro.html (Version 2022.10.10).
@manual{schweinberger2022repro,
author = {Schweinberger, Martin},
title = {Data Management, Version Control, and Reproducibility},
note = {https://slcladal.github.io/repro.html},
year = {2022},
organization = {The University of Queensland, School of Languages and Cultures},
address = {Brisbane},
edition = {2022.10.10}
}sessionInfo()## R version 4.2.1 (2022-06-23)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.1 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
##
## locale:
## [1] LC_CTYPE=en_AU.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_AU.UTF-8 LC_COLLATE=en_AU.UTF-8
## [5] LC_MONETARY=en_AU.UTF-8 LC_MESSAGES=en_AU.UTF-8
## [7] LC_PAPER=en_AU.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] stats graphics grDevices datasets utils methods base
##
## other attached packages:
## [1] gapminder_0.3.0 forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9
## [5] purrr_0.3.4 readr_2.1.2 tibble_3.1.7 ggplot2_3.3.6
## [9] tidyverse_1.3.2 tidyr_1.2.0 here_1.0.1 DT_0.24
## [13] kableExtra_1.3.4 knitr_1.39
##
## loaded via a namespace (and not attached):
## [1] lubridate_1.8.0 svglite_2.1.0 assertthat_0.2.1
## [4] rprojroot_2.0.3 digest_0.6.29 utf8_1.2.2
## [7] R6_2.5.1 cellranger_1.1.0 backports_1.4.1
## [10] reprex_2.0.1 evaluate_0.15 httr_1.4.3
## [13] highr_0.9 pillar_1.7.0 rlang_1.0.4
## [16] readxl_1.4.0 googlesheets4_1.0.0 rstudioapi_0.13
## [19] jquerylib_0.1.4 rmarkdown_2.14 webshot_0.5.3
## [22] googledrive_2.0.0 htmlwidgets_1.5.4 munsell_0.5.0
## [25] broom_1.0.0 modelr_0.1.8 compiler_4.2.1
## [28] xfun_0.31 pkgconfig_2.0.3 systemfonts_1.0.4
## [31] htmltools_0.5.2 tidyselect_1.1.2 fansi_1.0.3
## [34] viridisLite_0.4.0 withr_2.5.0 tzdb_0.3.0
## [37] crayon_1.5.1 dbplyr_2.2.1 grid_4.2.1
## [40] jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.1
## [43] DBI_1.1.3 magrittr_2.0.3 scales_1.2.0
## [46] cli_3.3.0 stringi_1.7.8 renv_0.15.4
## [49] fs_1.5.2 xml2_1.3.3 bslib_0.3.1
## [52] ellipsis_0.3.2 generics_0.1.3 vctrs_0.4.1
## [55] tools_4.2.1 glue_1.6.2 hms_1.1.1
## [58] fastmap_1.1.0 yaml_2.3.5 colorspace_2.0-3
## [61] gargle_1.2.0 rvest_1.0.2 haven_2.5.0
## [64] sass_0.4.1