Open Korean Corpora Save

Open Korean Corpora: A Living Document for Korean NLP Dataset Curation

Overview

• Korean, a language with 80M users is often overlooked in NLP research
• The availability of public datasets and tasks has hindered investigation
• Even the publicly available datasets are not always accompanied by English documentation and have poor discoverability
• Our work attempts to tackle this problem by curating a living document of open resources for the Korean language

PACLIC 2023

We will participate in PACLIC 37 and will present our paper "Revisiting Korean Corpus Studies through Technological Advances". If you visit PolyU@HK 3 December, please drop by!

NLP-OSS @ EMNLP 2020

We will be in the live session and monitoring the slide chat during EMNLP 2020. If you have any questions or would simply want to drop by to say hello, please drop by!

Public Institutions

Multiple government-funded institutions create datasets for the Korean language

• National Institute of Korean Language (NIKL)
• Electronics and Telecommunications Research Institute (ETRI)
• NIA AI HUB

Generally, government funded datasets tend to be very restrictive at allowing access to non-Korean citizens

Thus, Open Corpora here denotes a freely accessible and downloadable (at least only with a simple sign-in) dataset

Open Dataset for Korean NLP

Our work focuses on curating open Korean corpora under the following criteria:

• Documentation status
• License for use and distribution

Documentation and License

For documentation status Docu. the following holds.

• docu - Does the corpus have any documentation on the usage?
• paper - Does the corpus have a related article?
• int'l - Does the corpus have a internationally available publication?

License

For License, we check the followings:

• Commercially available (all), academic use only (academic), unknown (unk)
• Redistribution is available with/without modification (rd/no for rd and binary for mod-x), unknown (unk)

Other Attributes

• In Typical Usage, we note the area of intended use of the dataset.
• In Providers, we note if the dataset is provided by universities or institutes (Academia), companies or the research group thereof (Industry), or something combined, as Competition purpose.
• In Volume, (w) denotes words, (s) denotes sentences, (p) denotes pairs (either document or sentence pairs), (d) denotes dialogues, (h) denotes hours, and (u) denotes speech utterances.
• In Goal, Eval is noted if the purpose is suggested as an evaluation.
• In Lang, we note all the languages that the dataset handles.

View at a Glance

The table below describes the open Korean corpora investigated so far. To be updated along with our survey or PR. You can visit Here for the Korean description, and more information regarding government-driven database. We removed the hyperlink for pages not working.

1. Benchmark studies

2. Parsing and tagging

3. Entailment, sentence similarity, and paraphrase

4. Intention understanding, sentiment analysis, and offensive language detection

5. QA and dialogue

6. Summarization, Translation, and Transliteration

7. Korean in multilingual corpora

8. Speech corpora

9. Other topics

Citation

To cite our work, please use the following: (Also available as cho-etal-2020-open in anthology.bib)

@inproceedings{cho-etal-2020-open,
    title = "Open {K}orean Corpora: A Practical Report",
    author = "Cho, Won Ik  and
      Moon, Sangwhan  and
      Song, Youngsook",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.12",
    pages = "85--93",
    abstract = "Korean is often referred to as a low-resource language in the research community. While this claim is partially true, it is also because the availability of resources is inadequately advertised and curated. This work curates and reviews a list of Korean corpora, first describing institution-level resource development, then further iterate through a list of current open datasets for different types of tasks. We then propose a direction on how open-source dataset construction and releases should be done for less-resourced languages to promote research.",
}

Contributing

Please read the contributor guidelines before sending a pull request.