Seven recommendations for machine translation evaluation
Published:
Evaluating machine translation systems is not as obvious as it seems on first glance.
Posts by Tags
best practices
Using old training data or test sets
Published:
In MT research there is a long-standing tradition of using old data sets when newer versions are available for the same language pair and domain.
Statistical significance testing
Published:
In recent blog posts I have described many potential issues with MT evaluation. Surely statistical significance testing should help mitigate some of those problems? That may seem reasonable, but the truth is: it can be laughably easy to arrive at results that are statistically significant according to the most popular test in MT research, bootstrap resampling.
Single training runs and estimates of variance
Published:
Consider a very simple example of a table reporting BLEU scores:
Simulating low-resource experiments
Published:
Low-resource machine translation has been an active area of research for years. On a high level, what many papers on low-resource MT have in common is that they simulate low-resource scenarios.
Designing human evaluations and reporting the outcomes
Published:
Every once in a while, there is an MT paper claiming to have achieved human parity (e.g. Hassan et al., 2018, Popel et al., 2020). To be fair, a message like
Computing and reporting BLEU scores
Published:
BLEU scores are ubiquitous in MT research and they usually appear in tables that look like this:
Comparing to previous work
Published:
What is common between many of the questionable practices I blog about is that they are seemingly legitimized by saying
bleu
Using old training data or test sets
Published:
In MT research there is a long-standing tradition of using old data sets when newer versions are available for the same language pair and domain.
Computing and reporting BLEU scores
Published:
BLEU scores are ubiquitous in MT research and they usually appear in tables that look like this:
evaluation
Seven recommendations for machine translation evaluation
Published:
Evaluating machine translation systems is not as obvious as it seems on first glance.
Using old training data or test sets
Published:
In MT research there is a long-standing tradition of using old data sets when newer versions are available for the same language pair and domain.
Statistical significance testing
Published:
In recent blog posts I have described many potential issues with MT evaluation. Surely statistical significance testing should help mitigate some of those problems? That may seem reasonable, but the truth is: it can be laughably easy to arrive at results that are statistically significant according to the most popular test in MT research, bootstrap resampling.
Single training runs and estimates of variance
Published:
Consider a very simple example of a table reporting BLEU scores:
Simulating low-resource experiments
Published:
Low-resource machine translation has been an active area of research for years. On a high level, what many papers on low-resource MT have in common is that they simulate low-resource scenarios.
Designing human evaluations and reporting the outcomes
Published:
Every once in a while, there is an MT paper claiming to have achieved human parity (e.g. Hassan et al., 2018, Popel et al., 2020). To be fair, a message like
Computing and reporting BLEU scores
Published:
BLEU scores are ubiquitous in MT research and they usually appear in tables that look like this:
Comparing to previous work
Published:
What is common between many of the questionable practices I blog about is that they are seemingly legitimized by saying
human evaluation
Designing human evaluations and reporting the outcomes
Published:
Every once in a while, there is an MT paper claiming to have achieved human parity (e.g. Hassan et al., 2018, Popel et al., 2020). To be fair, a message like
low-resource
Simulating low-resource experiments
Published:
Low-resource machine translation has been an active area of research for years. On a high level, what many papers on low-resource MT have in common is that they simulate low-resource scenarios.
machine translation
Seven recommendations for machine translation evaluation
Published:
Evaluating machine translation systems is not as obvious as it seems on first glance.
Using old training data or test sets
Published:
In MT research there is a long-standing tradition of using old data sets when newer versions are available for the same language pair and domain.
Statistical significance testing
Published:
In recent blog posts I have described many potential issues with MT evaluation. Surely statistical significance testing should help mitigate some of those problems? That may seem reasonable, but the truth is: it can be laughably easy to arrive at results that are statistically significant according to the most popular test in MT research, bootstrap resampling.
Single training runs and estimates of variance
Published:
Consider a very simple example of a table reporting BLEU scores:
Simulating low-resource experiments
Published:
Low-resource machine translation has been an active area of research for years. On a high level, what many papers on low-resource MT have in common is that they simulate low-resource scenarios.
Designing human evaluations and reporting the outcomes
Published:
Every once in a while, there is an MT paper claiming to have achieved human parity (e.g. Hassan et al., 2018, Popel et al., 2020). To be fair, a message like
Computing and reporting BLEU scores
Published:
BLEU scores are ubiquitous in MT research and they usually appear in tables that look like this:
Comparing to previous work
Published:
What is common between many of the questionable practices I blog about is that they are seemingly legitimized by saying
parity
Designing human evaluations and reporting the outcomes
Published:
Every once in a while, there is an MT paper claiming to have achieved human parity (e.g. Hassan et al., 2018, Popel et al., 2020). To be fair, a message like
previous work
Comparing to previous work
Published:
What is common between many of the questionable practices I blog about is that they are seemingly legitimized by saying
randomness
Single training runs and estimates of variance
Published:
Consider a very simple example of a table reporting BLEU scores:
recommendations
Seven recommendations for machine translation evaluation
Published:
Evaluating machine translation systems is not as obvious as it seems on first glance.
reimplementation
Comparing to previous work
Published:
What is common between many of the questionable practices I blog about is that they are seemingly legitimized by saying
significance testing
Statistical significance testing
Published:
In recent blog posts I have described many potential issues with MT evaluation. Surely statistical significance testing should help mitigate some of those problems? That may seem reasonable, but the truth is: it can be laughably easy to arrive at results that are statistically significant according to the most popular test in MT research, bootstrap resampling.
variance
Single training runs and estimates of variance
Published:
Consider a very simple example of a table reporting BLEU scores: