Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser.

Gonito.net

open platform for research competition, cooperation and reproducibility

Filip Graliński, Rafał Jaworski,
Łukasz Borchmann, Piotr Wierzchoń

LREC 2016 / 4REAL Workshop

Motivation

Alternatives?

That's why we created

Gonito.net

open platform for machine learning competition

… and

GEval

companion Haskell library and stand-alone tool for machine learning evaluation

A challenge…

"He Said She Said" classification challenge

Guess whether a text in Polish was written by a man or woman.

This challenge is based on the "He Said She Said" corpus for Polish. The corpus was created by grepping gender-specific first person expressions (e.g. "zrobiłem/zrobiłam", "jestem zadowolony/zadowolona", "będę robił/robiła") in the Common Crawl corpus. Such expressions were normalised here into masculine forms.

Classes

  • F — text written by a woman
  • M — text written by a man

Directory structure

  • README.md — this file
  • config.txt — configuration file
  • train/ — directory with training data
  • train/train.tsv.gz — train set (gzipped), the class is given in the first column, a text fragment in the second one
  • train/meta.tsv.gz — metadata (do not use during training)
  • dev-0/ — directory with dev (test) data
  • dev-0/in.tsv — input data for the dev set (text fragments)
  • dev-0/expected.tsv — expected (reference) data for the dev set
  • dev-0/meta.tsv — metadata (not used during testing)
  • test-A — directory with test data
  • test-A/in.tsv — input data for the test set (text fragments)
  • test-A/expected.tsv — expected (reference) data for the test set (hidden)

Install GEval

git clone git://gonito.net/geval
cd geval
stack setup
stack install
# add ~/.local/bin to $PATH

Current status…

# submitter when description test-A/Accuracy ×
1 p/tlen 2016-05-15 18:31 VW tokens + 3-gram LM (+fix for the latest grep) 0.710777835465144 17
2 nozdi 2016-06-17 09:52 3 best voting 0.704700995893708 21
3 Marta 2016-05-23 04:15 nozdi Naive Bayes + Tfidf + swear words + emoticons 0.6583164204465 4
4 [anonymised] 2016-06-23 09:01 3gram model KenLM + stemming 0.656475669042337 4
5 Marek 2016-05-24 11:55 klon rozwiazania Mateusza + RandomForestClassifier 0.652923962807382 1
6 Przemysław Nowaczyk 2015-12-10 09:05 naive bayes by Przemysław Nowaczyk kod zrodlowy i zasoby 0.646864822768679 2
7 asdf 2016-05-24 17:13 lemma + nozdi naive bayes 0.639885307027894 1
8 Veal 2016-02-15 18:27 Fixed source code, added makefile. 0.634050361070468 5
9 [anonymised] 2015-12-17 07:34 pliki zrodlowe w odp. folderach 0.628840798602917 2
10 Jacek 2016-05-16 16:08 100k samle with RandomForestClassifier 0.619176853731061 7

How to replicate the best result?

git clone git://gonito.net/petite-difference-challenge -b submission-00115
cd petite-difference-challenge
make clean
make
geval --test-name dev-0

I wanna do something better

Sign up, add SSH key etc.

Clone the repo:

git clone ssh://gitolite@gonito.net/john/petite-difference-challenge
cd petite-difference-challenge
git pull ssh://gitolite@gonito.net/petite-difference-challenge
git push origin master
cd petite-difference-challenge

Work on your solution and check locally:

geval --test-name dev-0

Push your brilliant solution:

git add dev-0/out.tsv test-A/out.tsv # ...
git commit -m 'my brilliant solution'
git push origin master

… and let Gonito.net know about your submission (or configure a Git hook)

… and see your results

submitter when description dev-0/Accuracy test-A/Accuracy
p/tlen 2016-05-30 20:51 simple NN trained on all (3 passes) with logistic regression 0.65064504387916 0.645265964978525
p/tlen 2016-05-30 19:40 simple NN trained on all (3 passes) 0.647173804613041 0.64280572992873
p/tlen 2016-05-30 10:41 simple NN train on all 0.646250387565549 0.640817482418464
p/tlen 2016-05-30 07:48 simple NN trained on 1M utterances 0.641019937719901 0.634840940199179
p/tlen 2016-05-30 07:46 simple NN with 1M utterances 0.641019937719901 N/A
p/tlen 2016-05-29 20:09 skeleton for NN solutions 0.4042949003114 0.408269221692547
p/tlen 2016-05-15 18:31 VW tokens + 3-gram LM (+fix for the latest grep) 0.72310295089039 0.710777835465144
p/tlen 2016-03-24 22:01 Vowpal Wabbit -nn 6 on morphosyntactic tags 0.595024332376215 0.591742577995941
p/tlen 2016-03-24 21:40 6-gram LM on morphosyntactic tags 0.598711260295763 0.605825506206636
p/tlen 2016-03-24 21:21 6-gram LM on morphosyntactic tags 0.689570105552635 0.605825506206636

Be open!

You can open   your submission so that everybody could see it

… and you fork somebody else's submission   and build upon it

git clone ssh://gitolite@gonito.net/john/petite-difference-challenge
cd petite-difference-challenge
git fetch ssh://gitolite@gonito.net/petite-difference-challenge submission-00115
git reset --hard FETCH_HEAD

… or you can browse the code at http://gonito.net/gitlist/petite-difference-challenge.git/submission-00115

Who forked and submitted what and when?

Click here to see the graph

Yet another challenge

# submitter when description test-A/RMSE ×
1 p/tlen 2015-12-13 14:31 VW -nn 6 on up to 4-grams and [5-7] tokens vowpal-wabbit neural-network 24.8 18
2 R.J. 2016-05-29 14:08 VW -nn 6 on up to 4-grams and [5-7] tokens + wiki years 24.9 4
3 Marcin Junczys-Dowmunt 2015-12-12 22:02 The same as last, best epoch 24.9 11
4 [anonymised] 2016-05-13 13:21 Max entropy solution for 4 grams and 8 class 35.8 16
5 [anonymised] 2015-12-16 20:13 + script 50.3 3
6 [anonymised] 2015-12-16 23:38 first submission 75.0 1
7 antystenes 2017-05-29 15:18 Basic linear regression haskell linear-regression self-made 85.4 1
8 EmEm 2017-05-28 13:55 50 ep ,00001 lr, 1814-2014 linear-regression self-made 93.3 3

Future?

Thanks!

And check it out at gonito.net: