Intelligent Document Finder 2.0

A tool which can find your any document using semantic search.

This is an Improvised Version of Intelligent-Document-Finder
List of New Features--

1. Implemented Document Similarity Script, which allows you to see related or most similar documents.
2. Revamped website UI.
3. Reduces time complexities of search functions.

What is Intelligent Document Finder ?

How easy do you find it to remember the exact location of a document that you created last year? Not very easy, right? Big Organizations/people deal with hundreds of documents daily and forget about them, most of the time.
But what if we want that old documentation again for some work, but unfortunately you do not remember the name or the actual content of that document to retrieve it from the large storage of your computer.
In such cases, use of a Intelligent document finder can really make a huge difference. As, it can Search for the document(semantically) of your need based on a query input. This will not only help in faster access to the document, but will also help in grouping similar documents together and in analysing them.

Watch Project Demo:

Note

Currently this repositry is using predefined database of news articles gathered by web scraping. Due to the github restrictions on uploading the large files, we cannot upload it here.

Soon, we will add the support of the dynamic databases, so that you can use this tool for your own databases to build your own custom search engine.

Technologies Used

Python3.6 JavaScript jQuery HTML & CSS

Database Used:

SQlite

For implementing searching:

Various NLP(Natural Language Processing) techniques is used.

For website:

• Python-based Web framework : Flask
• JavaScript
• jQuery

Program Flow

Compatibility

• Backend (AI part) is compatible on any machine that has python and required dependencies installed.
• Recommended browsers: Mozilla Firefox and Google Chrome.

How to Install and Use ?

> mkdir IntelligentDocumentFinder

> cd IntelligentDocumentFinder

> git clone https://github.com/Sarthakjain1206/Intelligent_Document_Finder_2.0.git

Install Vitual Environment if not installed

• On Linux/MacOs > python3 -m pip install --user virtualenv
• On windows > py -m pip install --user virtualenv

Create Virtual Environment

• On macOS and Linux: > python3 -m venv env
• On Windows: > py -m venv env

Activate Environment:

• On macOS and Linux: > source env/bin/activate
• On Windows: > .\env\Scripts\activate

> pip install -r requirements.txt

Download Glove Word Embeddings from this link, decompress it and copy the glove.6B.100d file in DataBase folder

then, run initial_file.py through this command > python initial_file.py

Now you are good to go.. Just type this command everytime you want to access it, and open the website in chrome/firefox
> python src/app.py

Developers

You can get in touch with us on linkedln profiles

Sarthak Jain Machine Learning NLP Web Crawling

You can also follow me on Github to stay updated about my latest projects

Rishabh Mishra Full Stack Web Developer

You can also follow me on Github to stay updated about my latest projects

If you liked this repository, then do support it by giving it a star

Contributions

If you find any bug or have any suggestions to improve this project, then feel free to generate a pull request.

There are a lot of features that can be added to this tool.

1. Query Segmentation
2. Query Expansion (Mainly - Pseudo Relevance Feedback technique)
3. Improvising Spell Checker
4. Collocations For example- Currently this project consider "New York" as ["New","York"] i.e two different words but it should be consider as a single entity like ["New_York"], this can definitely make a big difference in search results.
5. Query Logs (Game changing technique for search engines)
6. Search result's segmentation [like- Luecene]

If you have any experience in implementing any of these features then, do contribue.

References

1. Awsome article of BM25 ranking algorithm on wikipedia - Okapi BM25

2. Read this article on Topic Modeling

3. Completely followed this beautiful article on SVOs tagging for generating tags for this project.

4. Used the BM25 ranking fuction implementation from this great repositry on github by dorianbrown.