Korean LegalQA using SentenceKoBART
Implementation of legal QA system based on SentenceKoBART
# install git lfs , https://github.com/git-lfs/git-lfs/wiki/Installation
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt install git-lfs
git clone https://github.com/haven-jeon/LegalQA.git
cd LegalQA
git lfs pull
# If the lfs quota is exceeded, please download it with the command below.
# https://drive.google.com/file/d/1DJFMknxT7OAAWYFV_WGW2UcCxmuf3cp_/view?usp=sharing
# mv SentenceKoBART.bin model/
# pip install --use-deprecated=legacy-resolver -r requirements.txt
pip install -r requirements.txt
python app.py -t index --flow flows/index_annlite.yml
GPU-based indexing available as an option
# test on bash
python app.py -t query --flow flows/query_annlite.yml
# test on REST API
python app.py -t query_restful --flow flows/query_annlite.yml
OPENAI_API_KEY=$OPENAI_KEY python app.py -t query --flow flows/query_annlite_openai.yml
OPENAI_API_KEY=$OPENAI_KEY python app.py -t query_restful --flow flows/query_annlite_openai.yml
streamlit run chat.py
https://user-images.githubusercontent.com/957840/227705344-27501a6f-1e0b-48c0-854d-62ebc8d3160d.mp4
Legal data is composed of technical terms, so it is difficult to search if you are not familiar with these terms. Because of these characteristics, I thought it was a good example to show the effectiveness of neural IR.
You can download SentenceKoBART.bin
from one of the two links below.
Model training, data crawling, and demo system were all supported by the AWS Hero program.
@misc{heewon2021,
author = {Heewon Jeon},
title = {LegalQA using SentenceKoBART},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/haven-jeon/LegalQA}}
data/legalqa.jsonlines
is crawled in www.freelawfirm.co.kr based on robots.txt
. Commercial use other than academic use is prohibited.