Writing papers with Gonito

If you write a paper, Gonito can make your life a little bit easier: you won't need to copy&paste evaluation results manually and all your results will referenced in a proper manner.

Just add the following piece of LaTeX code to your paper:

\usepackage{hyperref}
\usepackage{xstring}
% Format a reference to a Gonito submission
\newcommand{\gonitoref}[1]{\{\href{https://gonito.net/q/#1}{\StrMid{#1}{1}{6}}\}}
% A bare score from Gonito
\newcommand{\gonitobarescore}[1]{\minput{scores/#1.txt}}
% A score from Gonito along with a reference
\newcommand{\gonitoscore}[1]{\gonitobarescore{#1} \gonitoref{#1}}
% A reference and a score as two cells in a table
\newcommand{\gonitoentry}[1]{\gonitoref{#1} & \minput{scores/#1.txt}}

Now you will be able to reference your Gonito submissions using its git commit hash, e.g.: \gonitoref{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5}. The hash will be printed in a shorter form (just first 6 digits) and it will be clickable leading to the Gonito entry describing the submission (information how to get the data will be presented there).

You might explain the idea like this:

\gonitoscore{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5}%
\footnote{Reference codes to repositories stored at
Gonito.net~\cite{gonito2016} are given in curly brackets. Such a repository may be also accessed by going
to \url{http://gonito.net/q} and entering the code there.}

Here is the BiBTeX entry referenced in the above snippet:

@incollection { gonito2016,
  title = {Gonito.net -- Open Platform for Research Competition, Cooperation and Reproducibility},
  author = "Grali{\'n}ski, Filip and Jaworski, Rafa{\l} and Borchmann, {\L}ukasz and Wierzcho{\'n}, Piotr",
  editor = "Branco, António and Calzolari , Nicoletta and Choukri, Khalid",
  booktitle = {Proceedings of the 4REAL Workshop: Workshop on Research Results Reproducibility and Resources Citation in Science and Technology of Language},
  year = "2016",
  pages = "13-20"
}

Hashes

Two kinds of (SHA1) hashes can be used here:

submission hashes, i.e. the commit hash with the submission; the problem is that they can contain multiple output files (so-called variants), in such a case…
… output hashes can be used (i.e. the SHA1 digest of a specific output file)

In general, Gonito tries to guess the right metric in case of ambiguity. It is, however, a good idea to give the metric name along the reference codes, e.g.:

\gonitoscore{433e8cfdc4b5e20e276f4ddef5885c5ed5947ae5-Accuracy}

Actually getting the scores

But, wait, with Gonito you can give evaluation scores without manual copy&paste. The \gonitoscore command gives a score and a reference. The score is taken from a file scores/HASH.txt. You need to get it from Gonito, but it's not difficult to set it up in such a way that the scores could be downloaded automatically. For instance, if you use Makefile for building your papers, you could use the following snippet:

SCOREFILES=$(shell ./extract-score-files.pl main.tex)

paper.pdf: paper.tex main.tex $(SCOREFILES)
	... building instructions ...

scores/%.txt:
	mkdir -p scores
	wget "https://gonito.net/api/txt/score/"$* -O $@

The script extract-score-files.pl is like this:

#!/usr/bin/perl

use strict;

open(my $ih, '<', $ARGV[0]);
binmode($ih, ':utf8');

my %found = ();

while (my $line=<$ih>) {
    while ($line =~ m<\\gonito(?:barescore|score|entry)\{([^\}]+)\}>g) {
        $found{$1} = 1;
    }
}

print join(" ", map { "scores/${_}.txt" } sort keys %found);

The \gonitoentry command is very similar to \gonitoscore, it just formats the infromation as two cells in a table (i.e. \gonitoentry can be used only in tables).

Note that you can commit score files to your repository, so that everything is OK when your paper is edited at Overleaf or a similar service.