Features of Research Flow

Using this, any visitor to a Research Flow repo can:

• browse the most recently published results by visiting the master branch
• check-out a previous publication via its tag, eg. git checkout v1.1.0 (2019-10-15)
• view available publications, and see how they have evolved, by visiting the repo’s releases page
• peek at work-in-progress in the devel branch
• contribute to work-in-progress by creating a descriptively-named research branch, adding their updates, and submitting a pull request to merge into devel or another research branch

Releases on master are now publications

In release-based software development, software features are developed and incorporated into a particular release. In the research process, we publish what we believe to be the truth periodically via publications (think journal article, a group presentation, or even an email).

In Research Flow:

• master holds a record of tagged releases that represent publications
• cloning the repo or visiting its default main branch (master) allows one to browse the most recent publication
• visiting the repo’s releases page shows what publications exist
• comparing releases allows one to see see how publications have changed.

Semantic versioning applied to publication

In release-based software development, releases are commonly labeled with a semantic version that defines how the software functionality has changed from the previous release.

In our research-flow context, master holds a record of published works, semantically versioned to show how the published information has changed.

To help quickly understand how publications have changed, we recommend versioning publications with Major.Minor.Patch (eg. 1.0.0), followed by an optional publication date:

• Major: tracks changes to already published information including information removal
• Minor: tracks new information added that doesn’t conflict with already published information
• Patch: tracks adjustments to how information that has already been published is presented, no substantial changes to the information itself

An example of a sequence of publication release tags on master:

v1.0.0 (2019-10-01)  # first publication with result A (first publication, major)
v1.1.0 (2019-10-15)  # added code and new results B, C (minor)
v1.1.1 (2019-10-16)  # fixed typo, added graph showing C more clearly (patch)
v2.0.0 (2019-11-30)  # removed result A based on now invalidated data (major)
v3.0.0 (2019-12-05)  # fixed bug in calculation of B, meaning the B result has changed, reformatted graph (major overrides patch)

Research branches

Research branches are just normal branches that can be compared to other research branches or the development branch. They can be created by branching off of devel or another research branch.

Research branch examples

Isn’t this overkill? We usually only publish one thing and then make small adjustments before starting a new project.

Many research projects might never need to use more than a master and develop branch. They might only publish once and not need any semantic understanding of what was changed between publications. They might even just start out committing everything to master. That’s fine. The purpose of Research Flow is to make it easier to understand and manage different branches of research when things get more complex.

Perhaps the simplest answer:

If Research Flow isn’t solving a problem you feel like you have run into before, don’t use it

When should we create a research branch?

Possibly the most obvious reason to create a research branch:

you or a collaborator want to significantly alter an existing branch but also want to easily be able to “go back to how it was”

Why not just do everything on devel or create separate files or functions to explore different research ideas?

This does work in simple scenarios, however when projects get complicated and involve multi-file, multi-function, or multi-researcher processes, keeping track of dependencies becomes challenging. Undo’ing and redo’ing changes does too. Separating work into research branches isolates processes until the team decides what should be incorporated into published results.

Can I publish two articles from the same repo?

Remember, we are not just publishing an article, we’re publishing the entire devel branch, articles and code included. Therefore, the publication’s version needs to consider the state of the repo, not a particular article. If a second article is produced, it would resemble a new file in the repo. This represents new information to be shared, hence at least an increase of the minor version in a subsequent publication. The articles themselves may need their own version histories or supporting documentation via something like a CHANGES.md file.

What about persistent data across multiple branches?

When using multiple branches, special care must be taken to handle and separate branch-specific data, ie. the rsch/1 branch should not be overwriting data that the rsch/2 branch might access. This can be done fairly easily, but it is crucial that it is done.

The GitHub permalink

As progress is made in any branch, one can link to a specific line of code in an R script using GitHub permalinks.

Imagine you have published your results to master. You present your results and share a presentation that includes a link to the the R/results.R script file on your master branch so that a reader can reproduce the exact same graph. As long as you never change R/results.R, everything will work fine.

But now say you decide to reformat your graph extensively by changing the code in R/results.R. You do so and then publish again to master. The issue now is that the presentation you shared still references the same R/results.R file. When someone reading the presentation visits the link to R/results.R to reproduce your results, the graph be different. Fix this by sharing a permalink to the exact line of code that produced the graph- this will never change, no matter how many times you publish to master.

Until recently, we avoided using RMarkdown files because they were rendered by GitHub as markdown and hence did not have permlink features. We are glad to report that this is no longer the case.

Archiving versioned copies of presentations

When presenting results, archive a version of the presentation aligned to the release on master. In other words, “Save a copy” of your presentation documents and label them with the master version that they refer to. In a google-doc, for example, the original cloud document might continue to evolve, but you can view the copy of the document (or even use google docs’ built-in versionin’s “name this version” functionality) that has the same version as the earlier publication. This is often useful when having to consider “what were we thinking when we presented this?”.

Branch histories in repo

Including a ## Branch history section in README.md, or branch_history.md will help you understand what the branch was attempting to address and what the results were. For example, suppose your branching looks like this (o represents a commit, and \ represents a branching action):

master o-------------------------
devel   \--o--o--o---------------
rsch/1      \--o--o--o-----o-----
rsch/1b      \        \--o--o--o-
rsch/2        \--o--o----------o-

In other words, you were working on devel and then branched off rsch/1 and rsch/2 to work on two separate issues. You then added commits to devel, rsch/1, and rsch/2 before branching off from rsch/1 to rsch/1b.

Say you’re on a particular sub-branch, like rsch/1b: how do you know what issues you were attempting to address and what the results were?

A simple solution is to look at the commit history using git log, or even view the history in GitHub. However, when results are derived from the commits they may need to be summarized. Logging comments to a file allows you to remind your future self of what you were doing and what the results were.

In this case, on rsch/1b we might have something like:

## Branch history

Most recent parent branch first:

#rsch/1b
removed secondary dataset and verified we
produce the same results

# rsch/1
Branched from devel to modify all queries 
for data provider 1 instead of 0

Branching rsch/1b to verify elimination of
secondary data set produces same results

# devel
Developing initial model with data provider 0

Branching to rsch/1 and rsch/2 to test model
on data providers 1 and 2

# master
first commit

Whereas on devel, we might have something like:

## Branch history

Most recent parent branch first:

# devel
Developing initial model with data provider 0

Branching to rsch/1 and rsch/2 to test model
on data providers 1 and 2

Added output metrics for model

# master
first commit

Additionally, when deciding to abandon a research branch, it is useful to log why things didn’t work out. We have found that we don’t often come back to abandoned branches, but when we do, it can save days of work.

Comparing branches

GitHub allows you to compare changes between branches through its pull request feature. To do this, push a branch to the repo (or to a fork of the repo) and then go to the repo’s main page. Click on “New pull request” next to the branch button and then choose the base: and compare: branch options to choose the two branches you want to compare. You don’t have to create the pull request, you can just view the differences between files. This is often helpful in understanding what changed between a research branch and the devel branch prior to merging the research branch modifications into devel. This functionality is also available on the Git command-line-interface via git diff.