Modelscope Swift Versions Save

ms-swift: Use PEFT or Full-parameter to finetune 200+ LLMs or 15+ MLLMs

v2.0.4

2 weeks ago

v2.0.3

3 weeks ago

v2.0.0

1 month ago

New Features

  1. Support for peft 0.10.x version, with the default value of the tuner_backend parameter changed to peft. The interface of peft has been dynamically patched to support parameters like lora_dtype.
  2. Support for vllm+lora inference.
  3. Refactored and updated the README file.
  4. Added English versions of the documentation. Currently, all documents have both English and Chinese versions.
  5. Support for training 70B models using FSDP+QLoRA on dual 24GB GPUs. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh
  6. Support for training agents and using the ModelScopeAgent framework. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/Agent%E5%BE%AE%E8%B0%83%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md
  7. Support for model evaluation and benchmark. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.md
  8. Support for multi-task experiment management. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A3.md
  9. Support for GaLore training.
  10. Support for training and inference of AQLM and AWQ quantized models.

New Models

New Datasets

What's Changed

New Contributors

Full Changelog: https://github.com/modelscope/swift/compare/v1.7.3...v2.0.0

v1.7.0

2 months ago

New Features:

  1. Added support for swift export, enabling awq-int4 quantization and gpt-int2,3,4,8 quantization. Models can be pushed to the Modelscope Hub. You can view the documentation here.
  2. Enabled fine-tuning of awq quantized models.
  3. Enabled fine-tuning of aqlm quantized models.
  4. Added support for deploying LLM with infer_backend='pt'.
  5. Added web-ui with task management and visualization of training loss, eval loss, etc. Inference is accelerated using VLLM.

New Tuners:

  1. Lora+.
  2. LlamaPro.

New Models:

  1. qwen1.5 awq series.
  2. gemma series.
  3. yi-9b.
  4. deepseek-math series.
  5. internlm2-1_8b series.
  6. openbuddy-mixtral-moe-7b-chat.
  7. llama2 aqlm series.

New Datasets:

  1. ms-bench-mini.
  2. hh-rlhf-cn series.
  3. disc-law-sft-zh, disc-med-sft-zh.
  4. pileval.

What's Changed

Full Changelog: https://github.com/modelscope/swift/compare/v1.6.0...v1.7.0

v1.6.1

2 months ago

New Models:

  1. deepseek-math series

New Datasets:

  1. sharegpt-gpt4-mini
  2. disc-law-sft-zh
  3. disc-med-sft-zh

Bug Fix

  1. Fix vllm==0.3 & swift deploy bug.
  2. Fix zero3 & swift lora bug.

Full Changelog: https://github.com/modelscope/swift/compare/v1.6.0...v1.6.1

v1.6.0

3 months ago

New Features:

  1. Agent Training
  2. AIGC support: controlnet, controlnet_sdxl, dreambooth, text_to_image, text_to_image_sdxl
  3. Compatibility with vllm==0.3.*

New Models:

  1. qwen1.5 series
  2. openbmb series

What's Changed

Full Changelog: https://github.com/modelscope/swift/compare/v1.5.4...v1.6.0

v1.5.4

3 months ago

New Features:

  1. Default zero3.json file
  2. Enhanced support for multi-modal models

New Models:

  1. Orion series
  2. Codefuse series
  3. Internlm2-math series
  4. Internlm2 series
  5. Qwen2 series
  6. Yi-vl series
  7. Internlm-xcomposer2

What's Changed

New Contributors

Full Changelog: https://github.com/modelscope/swift/compare/v1.5.3...v1.5.4

v1.5.2

4 months ago

English Version

  1. Support show log in text box of web-ui
  2. Support share=True in web-ui, only need to set WEBUI_SHARE=1 in environment variable
  3. Support deactivate all adapters
  4. Support more SFT arguments
  5. Add longlora/qalora script
  6. Support custom models in web-ui
  7. ModelScope SWIFT studio released: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
  8. Fix some bugs

中文版本

  1. 支持在web-ui中直接显示日志
  2. 支持share=True 仅需要在环境变量中设置WEBUI_SHARE=1
  3. 支持失活所有adapters
  4. 添加了更多SFT参数
  5. 添加了longlora/qalora的训练脚本
  6. web-ui支持了自己注册的自定义模型
  7. SWFT魔搭创空间上线了: https://www.modelscope.cn/studios/damo/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary
  8. 修复了一些bug

What's Changed

Full Changelog: https://github.com/modelscope/swift/compare/v1.5.1...v1.5.2

v1.5.1

4 months ago

English version

New Features

  1. Support dtype settings in LoRA
  2. Support deactivated tuners offloading
  3. Support deployment with OpenAI format restful API
  4. Make LongLoRA supports the latest llama2 code

新feature

  1. 支持LoRA设置dtype类型
  2. 支持将不使用的tuners offloading到cpu和meta设备上
  3. 支持OpenAI restful API方式的部署
  4. LongLoRA支持最新的llama2代码

What's Changed

Full Changelog: https://github.com/modelscope/swift/compare/v1.5.0...v1.5.1

v1.5.0

4 months ago

English Version

New features:

  1. Support multi-line inference
  2. Support multi node training
  3. Add benchmarks
  4. Support UI training, start by swift web-ui
  5. Support VLLM inference
  6. Support RLHF(DPO) training

New tuners:

  1. SCEdit, adopted by TongYi Lab, uses lesser memory but produces better performance than LoRA, and can be used to replace ControlNet in a series of scenarios like Pose control/In-Painting/Out-Paining/Label-removing, etc.

New models:

  1. SUS series models
  2. Mixtral-MoE series models
  3. deepseek series models
  4. phi2-3b
  5. cogagent-chat/cogagent-vqa
  6. codegeex2-6b

New datasets:

Datasets used in RLHF:

  1. hh-rlhf
  2. stack-exchange-paired

中文版

SWIFT本月新版本已经发布！

新特性:

  1. 支持多行输入推理
  2. 支持多卡训练
  3. 添加了模型训练的benchmarks
  4. 支持界面训练和推理，通过swift web-ui开启
  5. 支持VLLM推理
  6. 支持RLHF(DPO)训练

新tuners:

SCEdit: 通义实验室自研的优秀U-Net微调框架，显存占用远小于LoRA，效果较LoRA更好，且可以替代实现ControlNet的效果，实现In-Painting/Out-Paining/标签去除/Pose控制等能力

新模型：

SUS系列模型 Mixtral-MoE系列模型 deepseek系列模型 phi2-3b cogagent-chat/cogagent-vqa codegeex2-6b

新数据集:

用于RLHF的数据集： hh-rlhf stack-exchange-paired

What's Changed

Full Changelog: https://github.com/modelscope/swift/compare/v1.4.0...v1.5.0