NOTE: This repos is now obsolete. Interested programmers should consider to use the new repo vlp (github.com/phuonglh/vlp) We have preferred using Scala instead of Java since 2016.
This is the third release of a Vietnamese text processing toolkit, which is called "Vitk", developed by Phuong LE-HONG at College of Science, Vietnam National University in Hanoi.
There are some toolkits for Vietnamese text processing which are already published. However, most of them are not readily scalable for large data processing. This toolkit aims at the ability of processing big text data. For this reason, it uses Apache Spark as its core platform. Apache Spark is a fast and general engine for large scale data processing. Therefore, Vitk is a fast cluster computing toolkit.
If you don't want to use Apache Spark, you should download and use a standalone Vietnamese tokenizer or tagger from their website, only one JAR file is needed to run the program. vnTokenizer 5.1 and vnTagger
Despite of its name, this toolkit supports processing in various natural languages providing that suitable underlying models or linguistic resources are available for the different languages. The toolkit is packaged with models and resources for processing Vietnamese. The users can build models for other languages using the underlying tools.
Currently, Vitk consists of three fundamental tools for text processing:
The word segmentation tool is specific to the Vietnamese language. The other tools are general and can be trained to parse any language. We are working to develop and integrate more fundamental tools to Vitk such as named entity recognition, constituency parsing, opinion mining, etc.
Download a prebuilt version of Apache Spark.
Vitk uses Spark version 1.6.x. Unpack the compressed file to a directory,
~ is your home directory.
Download Vitk, either a binary archive or its source code. The repository URL of the project is Vitk. The source code version is preferable. It is easy to compile and package Vitk: go to the top-level directory of Vitk and invoke the following command at a shell window:
mvn compile package
Apache Maven will automatically resolve and download dependency
libraries required by Vitk. Once the process finishes, you should
have a binary jar file
vn.vitk-3.0.jar in the sub-directory
Data files used by Vitk are specified in sub-directories of the directory
corresponding to its integrated tools.
These folders can contain data specific to a natural language in
use. Each language is specified further by a sub-directory whose name
is an abbreviation of the language name, for example
en for English,
fr for French, etc.
Vitk can run as an application on a stand-alone cluster mode or on a real cluster. If it is run on a cluster, it is required that all machines in the cluster are able to access the same data files, which are normally located in a shared directory readable by all the machines.
If you use a Unix-like operating system (Unix/Linux/MacOS), it is easy to share or
"export" a directory via a network file system
searches for data files in the directory
you need to copy the sub-directories
dat/* into that directory, so
that you have some folders as follows:
If you run Vitk on a stand-alone cluster mode, it is sufficient to create the data folders specified above on your single machine. The NFS stuffs can be ignored.
Vitk is an Apache Spark application, you run it by submitting the
main JAR file
vn.vitk-3.0.jar to Apache Spark. The main class of the
vn.vitk.Vitk which selects the desired tool by following
arguments provided by the user.
The general arguments of Vitk are as follows:
-m <master-url>: the master URL for the cluster, for example
spark://192.168.1.1:7077. If you do not have a cluster, you can
ignore this parameter. In this case, Vitk uses the stand-alone
cluster mode, which is defined by
local[*], that is, it uses all
the CPU cores of your single machine for parallel processing.
-t <tool>: the tool to run, where
tool is an abbreviation
of the tool:
tok for word segmentation (or tokenization);
dep for dependency parsing. If this argument is not
specified, the default
tok tool is used.
-l <language>: the natural language to process, where
language is an abbreviation
of language name which is either
vi (Vietnamese) or
en (English). If this
argument is not specified, the default language is Vietnamese.
-v: this parameter does not require argument. If it is used, Vitk
runs in verbose mode, in which some intermediate information
will be printed out during the processing. This is useful for debugging.
Note that if you are processing very large texts, for a better performance,
you should consider to set appropriate options of the
command, in particular,
more on submitting Apache Spark
In addition to the general arguments above, a specific tool of Vitk requires its own arguments. The usage of each tool is described in their corresponding page as follows:
You can also import the source code of Vitk to your favorite IDE
(Eclipse, Netbeans, etc), compile and run from source, for example,
launch the class
vn.vitk.tok.Tokenizer for word segmentation,
providing appropriate arguments as described above.
The algorithms used by the tools of Vitk can be found in some related scientific publications. However, some of the main methods implemented in Vitk have been, and will be described in a more accessible way by blog posts. For example, the word segmentation method is described in:
Any bug reports, suggestions and collaborations are welcome. I am reachable at: