Results of the RUSSE: submissions on the testset
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
ABCDEFGHIJKLMNOPQRSTUVW
1
Model IDHJRT-AVEPAE-AVEPAE2-AVEPMethod Description
2
5-rt-30.76250.92280.88870.9749word2vec (skip-gram, window size 10, 300d vectors) on ruwac + lib.ru + ru-wiki, synonym database, prefix dictionary, orthographic similarity
3
5-rt-10.76110.91360.88470.9747word2vec (skip-gram, window size 10, 300d vectors) on ruwac + lib.ru + ru-wiki, synonym database, prefix dictionary, orthographic similarity
4
5-rt-20.76110.91360.88470.9747word2vec (skip-gram, window size 10, 300d vectors) on ruwac + lib.ru + ru-wiki, synonym database, prefix dictionary, orthographic similarity
5
9-ae-40.71870.87130.82950.9488CBOW model with context window size 5 trained on Russian National Corpus
6
9-ae-70.71870.88390.83290.9522
CBOW model with context window size 5 trained on Russian National Corpus, augmented with CBOW model with context window size 10 trained on web corpus
7
9-ae-90.71870.88390.83420.9517
CBOW model with context window size 5 trained on Russian National Corpus, augmented with Skip-gram model with context window size 20 trained on news corpus
8
9-rt-40.71870.87130.82950.9488CBOW model with context window size 5 trained on Russian National Corpus
9
9-rt-70.71870.87130.82950.9488CBOW model with context window size 5 trained on Russian National Corpus
10
9-rt-60.71320.87440.84480.9506CBOW model with context window size 2 trained on Russian National Corpus
11
9-rt-90.71320.87440.84480.9506CBOW model with context window size 2 trained on Russian National Corpus
12
9-ae-50.71140.85390.80860.9410CBOW model with context window size 10 trained on Russian National Corpus
13
9-ae-80.71140.85390.80860.9410CBOW model with context window size 10 trained on Russian National Corpus
14
5-ae-30.70710.91850.95500.9835
word2vec (skip-gram, window size 10, 300d vectors) on ruwac + lib.ru + ru-wiki, bigrams on the same corpus, synonym database, prefix dictionary, orthographic similarity
15
5-ae-10.70680.90970.95530.9835
word2vec (skip-gram, window size 10, 300d vectors) on ruwac + lib.ru + ru-wiki, bigrams on the same corpus, synonym database, prefix dictionary, orthographic similarity
16
9-ae-60.70440.86250.82680.9649CBOW model with context window size 10 trained on web corpus
17
9-rt-50.70440.86250.82680.9649CBOW model with context window size 10 trained on web corpus
18
9-rt-80.70440.86250.82680.9649CBOW model with context window size 10 trained on web corpus
19
17-rt-10.70290.81460.89450.9490Distributional vector-based model, window size 5, trained on RUWAC and NRC, plmi-weighting
20
9-ae-10.70100.86030.83260.9418CBOW model with context window size 5 trained on Russian National Corpus (without some tricks for MWEs)
21
9-rt-20.70100.86030.83260.9418CBOW model with context window size 5 trained on Russian National Corpus (without some tricks for MWEs)
22
9-ae-30.70100.86910.85400.9662CBOW model with context window size 5 trained on web corpus
23
9-rt-30.70100.86910.85400.9662CBOW model with context window size 5 trained on web corpus
24
15-rt-40.67730.90520.90720.9651
word2vec trained on 150G of texts from lib.rus.ec (skip-gramm, 500d vectors, window size 20, 1 iteration, min cnt 5) + some hacks for oov-words
25
14-ae-10.67300.69320.76240.7856
26
14-rt-10.67300.69320.76240.7856
27
9-ae-20.66500.84750.89950.9555Skip-gram model with context window size 10 trained on news corpus
28
9-rt-10.66500.84750.89950.9555Skip-gram model with context window size 10 trained on news corpus
29
1-rt-10.66410.91330.83920.9658Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features
30
15-rt-20.65370.90340.91230.9646word2vec trained on 150G of texts from lib.rus.ec (skip-gramm, 500d vectors, window size 5, 3 iteration, min cnt 5)
31
15-ae-10.65320.88990.90440.9627word2vec trained on 150G of texts from lib.rus.ec (skip-gramm, 500d vectors, window size 20, 1 iteration, min cnt 5)
32
15-rt-30.65120.85590.91670.9646word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 300d vectors, window size 20, 1 iteration, min cnt 10)
33
16-ae-10.63950.85360.94930.9565GloVe (100d vectors) on RuWac (lemmatized, normalized)
34
1-ae-10.63780.92010.92770.9849Desicion trees based on n-grams (Wikipedia titles and search queries), morphological features and Word2Vec
35
16-ae-20.62710.82060.69710.8766GloVe (300d vectors, 840B tokens) on english Common Crawl corpus with Ru->En translation for evaluation
36
15-rt-10.62130.84720.91200.9669word2vec trained on 150G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 100)
37
6-ae-10.61860.83040.85600.9078
38
6-rt-10.61860.83040.85600.9078
39
6-ae-20.61170.81810.77980.8644
40
6-rt-20.61170.81810.77980.8644
41
15-ae-20.58250.77550.72730.9315word2vec trained on 3G of texts from wikipedia, lemmatized (skip-gramm, 500d vectors, window size 20, 9 iterations, min cnt 5)
42
17-rt-20.56720.68940.81180.8619Distributional vector-based model, window size 5, trained on RUWAC and NRC, ppmi-weighting
43
8-rt-10.5495-0.99170.58610.7721This semantic similarity method is based on a large linguistic ontology.
44
1-ae-20.51250.90760.83560.9613Desicion trees based on n-grams (Wikipedia titles and search queries), morphological features and Word2Vec
45
1-rt-40.51250.90760.83560.9613Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features
46
1-rt-30.49390.92090.85000.9723Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features with AUC maximization
47
1-rt-60.49390.92090.85000.9723Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features with AUC maximization
48
12-rt-30.47100.95890.56510.7756Applying Knowledge Extracted from Wikipedia and Wiktionary for Computing Semantic Relatedness
49
12-rt-10.46380.95880.55750.7738Applying Knowledge Extracted from Wikipedia and Wiktionary for Computing Semantic Relatedness
50
12-rt-20.46380.95880.55750.7738Applying Knowledge Extracted from Wikipedia and Wiktionary for Computing Semantic Relatedness
51
18-ae-30.43360.76300.68420.9065
52
18-ae-60.43360.76300.68420.9065
53
18-rt-30.43360.76300.68420.9065
54
18-rt-60.43360.76300.68420.9065
55
18-rt-70.43360.76300.68420.9065
56
19-rt-10.43240.51140.50680.7058network-based semantic similarity measure. the network is derived from a simple distributional corpus-based measure
57
1-rt-20.42900.87950.82740.9395Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features
58
1-rt-50.42900.87950.82740.9395Logistic regression trained on synonyms, hyponyms and hypernyms on word2vec features
59
18-ae-20.42460.75770.65930.9018
60
18-ae-50.42460.75770.65930.9018
61
18-rt-20.42460.75770.65930.9018
62
18-rt-50.42460.75770.65930.9018
63
18-rt-90.42460.75770.65930.9018
64
18-ae-10.41430.75160.63390.8969
65
18-ae-40.41430.75160.63390.8969
66
18-rt-10.41430.75160.63390.8969
67
18-rt-40.41430.75160.63390.8969
68
18-rt-80.41430.75160.63390.8969
69
11-rt-40.41330.72790.81570.8384
Distributional VSM trained on a custom 110M corpus, window size 1 (+grammatical features), PMI, Pearson correlation; test set tweaked (lemmatization, splitting MWEs and compounds)
70
11-rt-30.41120.72840.81420.8382
Distributional VSM trained on a custom 110M corpus, window size 1, PMI, Pearson correlation; test set tweaked (lemmatization, splitting MWEs and compounds)
71
11-rt-20.39340.71410.64050.8080Distributional VSM trained on a custom 110M corpus, window size 1 (+grammatical features), PMI, Pearson correlation
72
11-rt-10.39140.71480.64030.8079Distributional VSM trained on a custom 110M corpus, window size 1, PMI, Pearson correlation
73
15-ae-40.38920.70730.75940.9077
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of ae2 training set
74
15-ae-50.38920.70730.75940.9077
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of ae2 training set
75
15-ae-70.38920.70730.75940.9077
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of ae2 training set
76
2-ae-40.37170.68150.51950.7282
folksonomy graph based model, trained on ae training data (the subset is larger than in case of the first and second submissions) and manually selected negative examples, using conditional-tree machine learning algorithm
77
2-rt-40.37170.68150.51950.7282
folksonomy graph based model, trained on ae training data (the subset is larger than in case of the first and second submissions) and manually selected negative examples, using conditional-tree machine learning algorithm
78
4-rt-10.32390.75300.69190.7836measure based on lexico-syntactic patterns, wikipedia + web corpora
79
15-ae-30.31880.68910.73520.9065
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on ae2 training set
80
15-rt-50.31880.68910.73520.9065
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on rt training set
81
15-rt-60.30780.83790.71890.7574
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of rt training set
82
15-rt-70.30780.83790.71890.7574
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of rt training set
83
15-rt-80.30780.83790.71890.7574
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of rt training set
84
15-rt-90.30780.83790.71890.7574
word2vec trained on 28G of texts from lib.rus.ec (skip-gramm, 100d vectors, window size 10, 1 iteration, min cnt 10) + FFNN trained on part of rt training set
85
10-rt-20.30310.61230.73370.6760Distributional vector-based model. Right context, window size 4, trained on Google N-grams
86
10-rt-30.30310.61230.73370.6760Distributional vector-based model. Right context, window size 4, trained on Google N-grams
87
4-rt-20.27620.74900.70570.7647measure based on lexico-syntactic patterns, wikipedia corpus
88
4-rt-30.26620.74450.67330.7629measure based on lexico-syntactic patterns, web corpus
89
7-rt-10.25920.73150.69540.7132
Word vectors computed by Recurrent neural network trained on 30M words of custom Russian text corpus combined with automatically translated English word vectors
90
3-ae-20.25090.66760.61100.7178Russian Twitter corpus, post-reply statistical model, probability of word2 to appear in reply to word1
91
2-ae-10.24900.72750.59850.7301folksonomy graph based model, trained on ae training data, using AdaBoost machine learning algorithm
92
2-ae-20.24900.72750.59850.7301folksonomy graph based model, trained on ae training data, using AdaBoost machine learning algorithm
93
2-rt-10.24900.72750.59850.7301folksonomy graph based model, trained on ae training data, using AdaBoost machine learning algorithm
94
2-rt-20.24900.72750.59850.7301folksonomy graph based model, trained on ae training data, using AdaBoost machine learning algorithm
95
2-ae-30.24360.71830.58020.6732
folksonomy graph based model, trained on ae training data (the subset is larger than in case of the first submission) and manually selected negative examples, using AdaBoost machine learning algorithm
96
2-rt-30.24360.71830.58020.6732
folksonomy graph based model, trained on ae training data (the subset is larger than in case of the first submission) and manually selected negative examples, using AdaBoost machine learning algorithm
97
7-rt-20.23530.70920.68640.6997
Word vectors computed by Recurrent neural network trained on 30M words of custom Russian text corpus combined with automatically translated English word vectors
98
10-rt-10.16260.58780.64210.6248Distributional vector-based model. Right context, window size 4, trained on Google N-grams
99
3-ae-10.14790.52370.46020.6036Russian Twitter corpus, coocurrence measure with log weighting, cosine similarity
100
3-rt-10.14790.52370.46020.6036Russian Twitter corpus, coocurrence measure with log weighting, cosine similarity
Loading...
 
 
 
Final results (January 31)
Final results with additional fields (January 31)
Preliminary results (January 26)