JapaneseTokenizers Versions Save

For mecab wrapper class, you could call

• jumandic. It's alternative dictionary for mecab tokenizer. jumandic has rich morphological information.
• unidic. It's continuously maintained by NINJA. See more information here, JP only.

And, in this version, some arguments in mecab wrapper class is deleted because it's not consistent anymore.

• could not install some packages in setup.py because pip.main function is removed. Now setup.py calls subprocess.check_call function instead.
• pyknp package is updated and Jumanpp module is removed in the latest version. Now, JapaneseTokenizer package calls the latest pyknp

The following issue is fixed, which is specific to MacOS, I guess. https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/47

fixed issued in https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/44

Fixed this issue -> https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/39

fixed this issue https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/36

The bugs in the filtering process, they are fixed.

• issue33 https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/33
• issue31 https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/issues/31

There is a case which mecab + neologd returns a 10th-additional field. That causes value error inside a process of mecab-wrapper module. Detailed info. in https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers/pull/28

• Jumanpp interface works faster than the previous version.
  • This is only for UNIX distributions. Windows OS is an exception.