Vaporetto Versions Save

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

v0.6.3

1 year ago

Model Files

You can use the following assets: https://github.com/daac-tools/vaporetto/releases/tag/v0.5.0

v0.6.2

1 year ago

This release fixes the following bug:

  • #86 Cannot deserialize predictor (@rabbit19981023)

Model Files

You can use the following assets: https://github.com/daac-tools/vaporetto/releases/tag/v0.5.0

v0.6.1

1 year ago

Model Files

You can use the following assets: https://github.com/daac-tools/vaporetto/releases/tag/v0.5.0

v0.5.1

1 year ago

Model Files

You can use the following assets: https://github.com/daac-tools/vaporetto/releases/tag/v0.5.0

v0.5.0

1 year ago

Model Files

This software contains the results of joint research with the National Institute for Japanese Language and Linguistics (NINJAL).

We provide multiple model files for Vaporetto that you can download and use in your work. These models have been trained using BCCWJ and UniDic.

All of these models are trained with L1-regularization.

See below for license terms of each model.

(NOTE) Some of BCCWJ are not included in training data due to rights reasons.

Models With Dictionary

We provide models containing UniDic. These models have the highest accuracy in our distributions.

  • bccwj-suw+unidic+tag.model.zst: contains a tag prediction model. Tags are only trained using BCCWJ.
  • bccwj-suw+unidic+tag-huge.model.zst: contains a tag prediction model. Tags are trained using BCCWJ and UniDic.

Models Without Dictionary

We also provide models that do not contain UniDic. These models have been trained over three model sizes and two word units.

Short unit words (SUW) Long unit words (LUW)
Tiny (C=0.003) bccwj-suw-tiny.model.zst N/A
Small (C=0.1) bccwj-suw-small.model.zst bccwj-luw-small.model.zst
Middle (C=0.5) bccwj-suw-middle.model.zst bccwj-luw-middle.model.zst
Large (C=1.0) bccwj-suw-large.model.zst bccwj-luw-large.model.zst

License

The following models are licensed under 3-Clause BSD License.

  • bccwj-suw+unidic+tag.model.zst
  • bccwj-suw+unidic+tag-huge.model.zst

The following models are licensed under either of Apache License (Version 2.0) or MIT License at your option.

  • bccwj-suw-small.model.zst
  • bccwj-suw-middle.model.zst
  • bccwj-suw-large.model.zst
  • bccwj-luw-small.model.zst
  • bccwj-luw-middle.model.zst
  • bccwj-luw-large.model.zst

v0.4.0

2 years ago

Model Files

We provide multiple model files for Vaporetto that you can download and use in your work. These models have been trained using BCCWJ and UniDic.

All of these models are trained with L1-regularization.

See below for license terms of each model.

(NOTE) Some of BCCWJ are not included in training data due to rights reasons.

Models With Dictionary

We provide two models containing UniDic. These models have the highest accuracy in our distributions.

  • bccwj-suw+unidic+tag.model.zst: contains a tag prediction model
  • bccwj-suw+unidic.model.zst: does not contain a tag prediction model

Models Without Dictionary

We also provide models that do not contain UniDic. These models have been trained over three model sizes and two word units.

Short unit words (SUW) Long unit words (LUW)
Tiny (C=0.003) bccwj-suw-tiny.model.zst N/A
Small (C=0.1) bccwj-suw-small.model.zst bccwj-luw-small.model.zst
Middle (C=0.5) bccwj-suw-middle.model.zst bccwj-luw-middle.model.zst
Large (C=1.0) bccwj-suw-large.model.zst bccwj-luw-large.model.zst

License

The following models are licensed under 3-Clause BSD License.

  • bccwj-suw+unidic+tag.model.zst
  • bccwj-suw+unidic.model.zst

The following models are licensed under either of Apache License (Version 2.0) or MIT License at your option.

  • bccwj-suw-small.model.zst
  • bccwj-suw-middle.model.zst
  • bccwj-suw-large.model.zst
  • bccwj-luw-small.model.zst
  • bccwj-luw-middle.model.zst
  • bccwj-luw-large.model.zst

v0.3.0

2 years ago

Model Files

We provide multiple model files for Vaporetto that you can download and use in your work. These models have been trained using BCCWJ and UniDic.

All of these models are trained with L1-regularization.

See below for license terms of each model.

(NOTE) Some of BCCWJ are not included in training data due to rights reasons.

Models With Dictionary

We provide two models containing UniDic. These models have the highest accuracy in our distributions.

  • bccwj-suw+unidic+tag.model.zst: contains a tag prediction model
  • bccwj-suw+unidic.model.zst: does not contain a tag prediction model

Models Without Dictionary

We also provide models that do not contain UniDic. These models have been trained over three model sizes and two word units.

Short unit words (SUW) Long unit words (LUW)
Small (C=0.1) bccwj-suw-small.model.zst bccwj-luw-small.model.zst
Middle (C=0.5) bccwj-suw-middle.model.zst bccwj-luw-middle.model.zst
Large (C=1.0) bccwj-suw-large.model.zst bccwj-luw-large.model.zst

License

The following models are licensed under 3-Clause BSD License.

  • bccwj-suw+unidic+tag.model.zst
  • bccwj-suw+unidic.model.zst

The following models are licensed under either of Apache License (Version 2.0) or MIT License at your option.

  • bccwj-suw-small.model.zst
  • bccwj-suw-middle.model.zst
  • bccwj-suw-large.model.zst
  • bccwj-luw-small.model.zst
  • bccwj-luw-middle.model.zst
  • bccwj-luw-large.model.zst

0.2.0

2 years ago

0.1.6

2 years ago

0.1.5

2 years ago