Monday, September 17, 2012

Speech Recognition – Setting up sclite word alignment

Word alignment is used to measure the accuracy of a decoder.  Sphinx tutorial references sclite from National Institute of Standards and Technology (NIST). This time I’m going to share some notes on how to run and setup sclite.

Running sclite

Sclite is a tool for scoring and evaluating the output of speech recognition by comparing the hypothesized text (HYP) output by the speech recognizer to the correct, or reference (REF) text. After comparing REF to HYP, (a process called alignment), statistics are gathered during the scoring process and a variety of reports can be produced to summarize the performance of the recognition system.

This is an example output using the files located src\sclite\testdata\tests.hyp and sclite\testdata\tests.ref:

Parameters:
  • The '-h' option is a required argument which specifies the input hypothesis file.
  • The '-r' option, a required argument, specifies the input reference file which the hypothesis file(s) are compared to.

sclite -h C:\Project\SpeechRecognition\CMUSphinx\3rdPartyLibs\sctk-2.4.3\src\sclite\testdata\tests.hyp -r C:\Project\SpeechRecognition\CMUSphinx\3rdPartyLibs\sctk-2.4.3\src\sclite\testdata\tests.ref

New Picture (1)


Setup  sctk-2.4.0-20091110-0958.tar.bz2 on Windows 7

I downloaded Speech Recognition Scoring Toolkit (SCTK) which includes the SCLITE, ASCLITE, tranfilt, hubscr, SLATreport and utf_filt scoring tools.

I could compile it with gcc version 3.4.4, found in the following MinGW setup package  mingw-get-inst-20101030.exe.

It was also necessary to modify 'src/rfilter1/makefile.in' and change the value of OPTIONS to be blank  (as specified in the instructions)

The following compilation error is thrown when compiling using gcc version 4.6.2:

recording.h:122:29: error: 'Filter::Filter' cannot appear in a constant-expression
recording.h:122:36: error: template argument 2 is invalid
recording.h:122:36: error: template argument 4 is invalid
make[3]: *** [main.o] Error 1

Setup sctk-2.4.2-20120810-0938.tar.bz2 on Windows 7

Something similar happened with this version, I could compile it with gcc version 3.4.4.

The following compilation error is thrown when using gcc version 4.6.2:

In file included from asctools.h:23:0,
                 from asctools.cpp:22:
timeval.h:33:8: error: redefinition of 'struct timeval'
c:\mingw\bin\../lib/gcc/mingw32/4.6.2/../../../../include/winsock2.h:109:8: error: previous definition of 'struct timeval'
make[2]: *** [asctools.o] Error 1
make[2]: Leaving directory `/c/Project/SpeechRecognition/CMUSphinx/3rdPartyLibs/
sctk-2.4.2/src/asclite/test'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/c/Project/SpeechRecognition/CMUSphinx/3rdPartyLibs/
sctk-2.4.2/src/asclite'
make: *** [all] Error 2

This is thrown when compiling rfilter1

gcc  -o rfilter1 rfilter1.c
C:\Users\MANGEL~1\AppData\Local\Temp\ccuj1MU6.o:rfilter1.c:(.text+0x760): undefined reference to `strncmpi'
C:\Users\MANGEL~1\AppData\Local\Temp\ccuj1MU6.o:rfilter1.c:(.text+0x7c4): undefined reference to `strncmpi'
C:\Users\MANGEL~1\AppData\Local\Temp\ccuj1MU6.o:rfilter1.c:(.text+0x827): undefined reference to `strncmpi'
C:\Users\MANGEL~1\AppData\Local\Temp\ccuj1MU6.o:rfilter1.c:(.text+0x935): undefined reference to `strncmpi'
collect2: ld returned 1 exit status
make[1]: *** [rfilter1] Error 1
make[1]: Leaving directory `/c/Project/SpeechRecognition/CMUSphinx/3rdPartyLibs/sctk-2.4.2/src/rfilter1'
make: *** [all] Error 2

It is also possible to compile this with gcc version 4.6.2 after removing asclite tests and rfilter1 from make file.

After finishing the setup you are able to run sclite as described initially.

Resources:

6 comments:

  1. Could you please share the sources of sctk? NIST site link is not working. Thanks!

    ReplyDelete
  2. Just managed to compile sclite using standard MinGW installation and I think I solved some of the issues above. Posting it here in case someone finds it one day...

    1. Issue "Filter::Filter" is a bug on the programmers' behalf, I think. The Filter class int "src\asclite\core\filter.h" is not in any Filter namespace. The correct thing would be to change all occurances of "Filter::Filter" into "::Filter". That way everything works and compiles everywhere.

    2. The timeval thing is a strange hack. They recreated the timeval struct that already exists on windows platforms, probably for use in *nix. Really, they should have used something else, but the simple solution here is to remove their timeval include from the "src/asclite/test/asctools.h" and replace it with includes to and

    You can remove/rename the timeval.h from the same folder as well, just in case.

    3. The solution to the "strncmpi" problem is twofold. First, in my case the config didn't work well and you should make sure that the makefile oin the "src/rfilter1" folder doesn't contain "OPTIONS=-DNEED_STRCMP=1" is not there. If it's there, best to remove it altogether (setting it to 0 doesn't really work). This makes sure you don't get any redefinition errors for "strcmpi" and "strncmpi" functions.

    Second problem is the missing of the "strncmpi" definition in any library. It's strange, but MinGW has the strncmpi declarations, but no library that defines this method. Looking online, I found that some compilers (Borland in this case) simply map the "strncmpi" to "strnicmp". To do this I simply added "#define strncmpi strnicmp" in the end of "src/rfilter1/include/fcndcls.h"

    Reference: http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/devwin32/strncmpi_xml.html

    After these changes, runnink "make config", "make all" and "make install" in the msys shell made everything work fine.

    ReplyDelete
  3. I am struggling with Sclite supported formats. Does Sclite support utf-8 unicode?

    ReplyDelete
  4. @danijel Hi I also got that "Filter::Filter" bug when trying to install sctk on my Ubuntu 12.04. Can you please tell me in which file I should change "Filter::Filter" into "::Filter"? Thanks in advance!

    ReplyDelete
  5. Hi all,
    I have the sctk-2.4.0, also I compiled gcc version 3.4.4 (3 of them). tried to write the right path in the cmd, but I get the error " sclite is not recognized as a...."
    Please help me out to fix it.
    Thanks a lot!

    ReplyDelete
  6. This time I’m going to share some notes on how to run and setup sclite.speech recognition software

    ReplyDelete