HPT - Open Multimodal Large Language Models

Hyper-Pretrained Transformers (HPT) is a novel multimodal LLM framework from HyperGAI, and has been trained for vision-language models that are capable of understanding both textual and visual inputs. HPT has achieved highly competitive results with state-of-the-art models on a variety of multimodal LLM benchmarks. This repository contains the open-source implementation of inference code to reproduce the evaluation results of HPT Air on different benchmarks.

Release

• [5/03] :fire: Releasing HPT 1.5 Air, our best open-sourced 8B Multimodal LLM with Llama 3. Built with Meta Llama 3, Our hyper capable HPT 1.5 Air packs a punch on real world understanding and complex reasoning. HPT Air 1.5 achieves the best results among <10B models across a wide range of challenging benchmarks (MMMU, POPE, SEED-I, and more). HPT 1.5 Air is publicly available on [HuggingFace Repository]. Please read our our [technical blog post] for more details.
• [3/16] HPT 1.0 Air is out, our most efficient model as a cost-effective solution that is capable of solving a wide range of vision-and-language tasks. HPT 1.0 Air is publicly available and achieves state-of-the-art results among all the open-source multimodal LLM models of similar or smaller sizes on the challenging MMMU benchmark. Please read our [technical blog post] and [HuggingFace Repository] for more details.

Table of Contents

• Overview of Model Achitecture
• Quick Start
  • Installation
  • Prepare the Model
  • Demo
• Evaluations
• Benchmarks
• Pretrained Models Used
• Disclaimer and Responsible Use
• Contact Us
• License
• Acknowledgements

Overview of Model Achitecture

Quick Start

Installation

pip install -r requirements.txt
pip install -e .

Prepare the Model

You can download the model weights from HF into your [Local Path] and set the global_model_path as your [Local Path] in the model config file:

git lfs install
git clone https://huggingface.co/HyperGAI/HPT [Local Path]

or directly set global_model_path as the HF repo-id ('HyperGAI/HPT').

You can also set other strategies in the config file that are different from our default settings.

Demo

After setting up the config file, launch the model demo for a quick trial:

python demo/demo.py --image_path [Image]  --text [Text]  --model [Config]

Example:

python demo/demo.py --image_path demo/einstein.jpg  --text 'Question: What is unusual about this image?\nAnswer:'  --model hpt-air-demo

You can design different prompts here to boost the question.

Evaluations

Launch the model for evaluation:

torchrun --nproc-per-node=8 run.py --data [Dataset] --model [Config]

Example for HPT 1.5 Air:

torchrun --nproc-per-node=8 run.py --data MMMU_DEV_VAL --model hpt-air-1-5

Benchmarks

For HPT 1.5 Air

Pretrained Models Used

HPT 1.5 Air

• Pretrained LLM: Llama3-8B-Instruct

• Pretrained Visual Encoder: siglip-so400m-patch14-384

HPT 1.0 Air

• Pretrained LLM: Yi-6B-Chat

• Pretrained Visual Encoder: clip-vit-large-patch14-336

Disclaimer and Responsible Use

Note that the HPT Air is a quick open release of our models to facilitate the open, responsible AI research and community development. It does not have any moderation mechanism and provides no guarantees on their results. We hope to engage with the community to make the model finely respect guardrails to allow practical adoptions in real-world applications requiring moderated outputs.

Contact Us

• Contact: [email protected]
• Follow us on Twitter.
• Follow us on Linkedin.
• Visit our website to learn more about us.

License

This project is released under the Apache 2.0 license. Parts of this project contain code and models from other sources, which are subject to their respective licenses and you need to apply their respective license if you want to use for commercial purposes.

Acknowledgements

The evaluation code for running this demo was extended based on the VLMEvalKit project. We also thank OpenAI for open-sourcing their visual encoder models and 01.AI and Meta for open-sourcing their large language models.