Challenge How To

Install GEval (recommended, but not obligatory)

In order to evaluate your work locally, you need the GEval tool. There are two ways to get it.

option A — just download the GEval fully static binary (simpler if you have a standard 64-bit Linux)

wget https://gonito.net/get/bin/geval
chmod u+x geval
./geval --help

option B — install GEval with Haskell Stack (needs some time and a couple of gigabytes of hard disk space)

First, install Haskell Stack:

curl -sSL https://get.haskellstack.org/ | sh

and then install GEval itself:

git clone git://gonito.net/geval
cd geval
stack setup
stack install
~/.local/bin/geval --help

For troubleshooting, see GEval readme.

(GEval will make your life easier, but is not obligatory, so if you have any trouble with installing/running GEval, you can skip this step, for the time being.)

Get your repo

You need to set up your ID and upload your SSH public key (see your account) if you want to have a repo hosted on Gonito.net, then:

git clone --single-branch ssh://gitolite@gonito.net/YOURID/retro-gap

(Warning about empty repository is expected, don't worry about it.)

cd retro-gap
git pull ssh://gitolite@gonito.net/retro-gap

Using an external repository

Alternatively, you can use any other Git repo, e.g. GitLab, GitHub or your own repo, make sure Gonito.net has access to your repo.

(And initiate it with ssh://gitolite@gonito.net/retro-gap.)

Work on your solution

You need to generate your solution for the test set as test-A/out.tsv. You may also generate the solution for the dev set (dev-0/out.tsv).

You can evaluate results for the dev set locally:

cd retro-gap
geval --test-name dev-0

Push your solution to the git repo

Commit and push out.tsv files to your repo. It is also recommended to push your source codes along with out.tsv files.

cd retro-gap
git add foo.py build.sh # add your source codes
git add dev-0/out.tsv test-A/out.tsv # add your output files
git commit -m 'my brilliant solution'
git push origin master

Repos hosted on Gonito.net

If you use a repo hosted here, a submission and evaluation is triggered automatically. You'll see the evaluation results in your console while pushing.

External repos

If you use an external repo (e.g. at your own of Gitolite or at GitLab/GitHub), you can configure a webhook.

Manual submission

In case other methods fail, you can submit your solution manually — go to the submit form.

Submission metadata

Gonito can take the metadata (description, tags, parameters) of a submission from a number of sources (in order of precedence):

  1. the YAML files specified in the param-files field of the gonito.yaml file
    • only applicable for parameters,
    • gonito.yaml will be always skipped even if matches a mask given in the param-files field,
    • parameters blacklisted in the unwanted-params field of the gonito.yaml file will be discarded;
  2. gonito.yaml file committed to the repository
    • description given in the description field,
    • tags given in tags field,
    • parameters given in params field,
  3. names of output files (only for parameters)
    • e.g. if the output file is out-epochs=10,learning-rate=0.01.tsv, then parameters epochs=10 and learning-rare=0.01 will be extracted;
  4. submission form (when submitting manually);
  5. git commit message
    • description taken from the first paragraph
    • tags taken from a line starting with tags:

Here is an example of gonito.yaml, in which all metadata could be given:

description: This my brilliant solution
tags:
  - neural-network
  - left-to-right
params:
  epochs: 10
  learning-rate: 0.01
unwanted-params:
  - model-file
  - vocab-file
param-files:
  - "*.yaml"
  - config/*.yaml

It might seem a little bit complicated, but you could simply use the method which is the most convenient for you.