Automatic Speech Recognition (ASR) Evaluation
If you're using any Speech-to-Text or Speech Recognition system to generate transcriptions from your audio/video content, then you can use this tool to compare how well it is doing against a human generated transcription. If you're not sure how to generate transcription, you can take a look here for list of tutorials to help you get started.
What can this utility do?
This is a simple utility to perform a quick evaluation on the results generated by any Speech to text (STT) or Automatic Speech Recognition (ASR) System.
This utility can calculate following metrics -
- Word Error Rate (WER), which is a most common metric of measuring the performance of a Speech Recognition or Machine translation system
- Word Information Loss (WIL), which is a simple approximation to the proportion of word information lost. Refer to this paper for more info.
- Levenshtein Distance calculated at word level.
- Number of Word level insertions, deletions and mismatches between the original file and the generated file.
- Number of Phrase level insertions, deletions and mismatches between the original file and the generated file.
- Color Highlighted text Comparison to visualize the differences.
- General Statistics about the original and generated files (bytes, characters, words, new lines etc.)
The utility also performs the pre-processing or normalization of the text in the provided files based on following operations -
- Remove Speaker Name: Remove the Speaker name at the beginning of the line.
- Remove Annotations: Remove any custom annotations added during transcriptions.
- Remove Whitespaces: Remove any extra white spaces.
- Remove Quotes: Remove any double quotes
- Remove Dashes: Remove any dashes
- Remove Punctuations: Remove any punctuations (.,?!)
- Convert contents to lower case
Pre-requisites
Make sure that you have NodeJS v8+ installed on your system.
Installation
npm install -g speech-recognition-evaluationVerify installation by simply running:
asr-evalUsage
Simplest way to run your first evaluation is by simply passing original and generated options to asr-eval command.
Where, original is a plain text file containing original transcript to be used as reference; usually this is generated by human beings.
And generated is a plain text file containing generated transcript by the STT/ASR system.
asr-eval --original ./original-file.txt --generated ./generated-file.txtThis would print simply the Word Error Rate (WER) between the provided files. This is how the output should look like:
Word Error Rate (WER): 13.61350109561817%
To find more information about all the available options:
asr-eval --helpAll the available usage options would be printed:
Synopsis
$ asr-eval --original file --generated file
$ asr-eval [options] --original file --generated file
$ asr-eval --help
Options
-o, --original file Original File to be used as reference. Usually, this should be the
transcribed file by a Human being.
-g, --generated file File with the output generated by Speech Recognition System.
-e, --wer [true|false] Default: true. Print Word Error Rate (WER).
-i, --wil [true|false] Default: true. Print Word Information Loss (WIL).
--distance [true|false] Default: false. Print total word distance after comparison.
--stats [true|false] Default: false. Print statistics about original and generate files, before
and after pre-processing. Also prints statistics about word level and phrase
level differences.
--pairs [true|false] Default: false. Print all the difference pairs with type of difference.
--textcomparison [true|false] Default: false. Print the text comparison between two files with
highlighting.
--removespeakers [true|false] Default: true. Remove the speaker at the start of each line in files before
calculations. The speaker should be separated by colon ":" i.e. speaker_name:
text For e.g. "John Doe: Hello, I am John." would get converted to simply
"Hello, I am John."
--removeannotations [true|false] Default: true. Remove any custom annotations in the transcript before
calculations. This is useful when removing custom annotations done by human
transcribers. Anything in square brackets [] are detected as annotations.
For e.g. "Hello, I am [inaudible 00:12] because of few reasons." would get
converted to "Hello, I am because of few reasons."
--removewhitespaces [true|false] Default: true. Remove any extra white spaces before calculations.
--removequotes [true|false] Default: true. Remove any double quotes '"' from the files before
calculations.
--removedashes [true|false] Default: true. Remove any dashes (hyphens) "-" from the files before
calculations.
--removepunctuations [true|false] Default: true. Remove any punctuations ".,?!" from the files before
calculations.
--lowercase [true|false] Default: true. Convert both files to lower case before calculations. This is
useful if evaluation needs to be done in case-insensitive way.
--help [true|false] Print this usage guide.
Getting help
If you need help installing or using the utility, please give a shout out in our slack channel
If you've instead found a bug or would like new features added, go ahead and open issues or pull requests against this repo!