New Models:

• Vision transformer backbone
  • MPViT
• Text-image retrieval model
  • LightningDOT
• Video retrieval models
  • BridgeFormer
  • collaborative-experts

Attention Visualization:

• Attention heatmap visualization for Vit, MViT models
• Cross-modality visualization for CLIP model
• Tools for embedding visualization

Faiss supports:

• Add new mixins to_faiss, faiss_search #1447

Bug Fix:

• Fix towhee.api.as_function() object has no attribute 'name' #1489
• Fix IndexError with towhee.glob #1474

Highlights

• Towhee supports video, audio and multimodal processing!
• Towhee supports text-video retrieval!
• Towhee supports video understanding/classification!
• Towhee supports multimodal!
• Towhee supports ensemble!
• Towhee supports stream processing for audio data!

7 SOTA video understanding/classification models

ACAR-Net, omnivore, timesformer, Movinet, TSM, Uniformer, Video Swin Transformer

4 SOTA video retrieval models

clip4clip, DRL, Frozen in time, mdmmt

Operators

• Operators suppport 7 SOTA video understanding/classification models: timesformer, ActionCLIP, omnivore, Movinet, TSM, Uniformer, Video Swin Transformer，intergration META's pytorchvideo in Action Classification
• Operators support 3 SOTA video retrieval models: clip4clip, mdmmt, frozen_in_time in Video/Text Embedding
• Operators support 2 SOTA ensemble algorithms: Greedy ensemble, Weighted Average Ensemble in Ensemble
• Operators support 6 SOTA multimodal models: CLIP, data2text, data2audio, data2vision, LightningDOT, UNITER in Multimodal
• Add 1 image embedding operator with DOLG in Image embedding （https://towhee.io/image-embedding/dolg）
• Add stream processing operators for audio and video with ffmpeg in Image decode and Audio decode
• Add audio embedding operator with VGGish and stream processing support! In Audio embedding

Bootcamp

• action classification
• reverse video search
• text video retrieval
• text-image-search

Showcase

• Build a Milvus Powered Image Search Engine in Minutes
• Deep Dive into Real-World Image Search Engine with Towhee

ChangeLog

• Towhee has a new repo for code examples: https://github.com/towhee-io/examples;
• Add support for Milvus data source;

Highlights

• A new programming API, DataCollection, is released in this version. The users can build their own unstructured data processing pipeline and application with a pythonic, method-chaining style DSL, based on DataCollection. Please check details in our API document;
• Towhee now provides a decorator towhee.register to help the users register their own code as an operator, and cooperate their code with the operators from towhee hub. The decorator also simplifies operator development. Please check our examples in the API document: towhee ‒ Towhee v0.6.0 documentation;
• towhee.train now integrate captum to explain model;

towhee

• The DataCollection API:
  • Core API #659: #727
  • Parallel execution: #676 #705 #837
  • DAG execution: #1024
  • Experimental ray backend: #933 #947
  • Mixins: #750 #834 #929 #961 #970 #1026
• The decorator for register operator: #897 #920
• Support for entity class: #799 #868 #998 #1035

towhee.models

towhee.models now is a separate python package , which makes it much easier to be upgraded independently.

• plot utils during training: #841 #842
• integrate captum to explain model: #854
• fix some training bugs: #780 #940

Highlights

• training/fine-tuning is supported! For the cases when an operator is a wrapper of neural network model, users can train or fine-tune the model via train() method of the operator.
• Towhee framework now supports video pipelines!
• New task types, word embedding, object detection, face detection, face feature encoding, are included in this version.
• 400+ new pipelines are available at towhee hub.

Thanks

Many thanks to all those who contributed to this release! @annisun2020 @binbinlv @Chiiizzzy @filip-halt @fzliu @GuoRentong @jaelgu @jeffoverflow @jingkl @jennyli-z @junjiejiangjjj @krishnakatyal @LoveEachDay @oneseer @reiase @shanghaikid @shiyu22 @Tumao727 @wxywb @yanliang567 @zc277584121

Highlights

• The Towhee website has a new look and feel! The new Towhee website includes several important docs, including in-depth Towhee tutorials, pipeline and operator summaries, development and contributing guides, and more. See https://docs.towhee.io. If you have any feedback for the website design or encounter any bugs, please open an issue through Github.
• Towhee now offers pre-built embedding pipelines that use transformer-based models: SwinTransformer and ViT.
• Towhee now offers pipelines with ensemble embedding models.
• Official support for several different operating systems (Ubuntu 18.04, Ubuntu 20.04, macOS 10, macOS 11 Intel CPU, and Windows 10) is here!
• A major refactor was completed - this refactor uses a new task scheduling framework and allows Operators to be based on generators, added several new iterators over input DataFrames (flatmap, concat, etc), added runners for operators, and more. See below for a full set of details.
• Towhee now supports concat operators in pipeline.
• Towhee now supports local caching - previously downloaded pipelines and operators are now cached within the ~/.towhee directory for future use.

Pipelines

• Image embedding pipeline(s) based on Resnet models (paper):
  • image-embedding-resnet50
  • image-embedding-resnet101
• Image embedding pipeline(s) based on EfficientNet models (paper):
  • image-embedding-efficientnetb5
  • image-embedding-efficientnetb7
• Image embedding pipeline(s) based on ViT (paper):
  • image-embedding-vitlarge
• Image embedding pipeline(s) based on Swin Transformers (paper):
  • image-embedding-swinbase
  • image-embedding-swinlarge
• Image embedding pipeline(s) based on multiple ensemble models:
  • image-embedding-efficientnetb7-swinlarge-concat
  • image-embedding-3ways-ensemble-large-v1
• Music embedding pipeline(s) based on VGGish (paper):
  • audio-embedding-vggish
• Music embedding pipeline(s) based on CLMR (paper):
  • audio-embedding-clmr

Operators

• Operator(s) based on Resnet models (paper):
  • resnet-image-embedding
• Operators(s) based on EfficientNet models (paper):
  • efficientnet-image-embedding
• Operators(s) based on ViT (paper):
  • vit-image-embedding
• Operators(s) based on Swin Transformers (paper):
  • swintransformer-image-embedding
• Operators(s) based on VGGish (paper):
  • tf-vggish-audioset
• Operators(s) based on CLMR (paper):
  • clmr-magnatagatune

Towhee Framework

• A major refactor was completed - this refactor uses a new task scheduling framework, it also allows Operators to be based on generators, added several new iterators over input DataFrames (flatmap, concat, etc), added runners for operators, and more. See below for a full set of details.
• Engine refactor: #272 #303 #324
• Support concat #296 #339 #359
• Hub pipeline/operator repo download by branch #286
• Local cache for pipeline/operator repo #297
• Adjust pipeline/operator repo file structure, add hub tools for pipeline/operator project init #315 #323
• Base classes for operators: NNOperator, PyOperator #333
• Clean up towhee third party dependency. Migrate requirements from towhee to operators. #308
• Allow instantiate pipelines by yaml description files. #320

Documentations

New pages

• "Overview"
• "Getting started / Quick start"
• "Getting started / Installation"
• "Tutorials / Reverse image search"
• "Tutorials / Image deduplication"
• "Tutorials / Music recognition"
• "Supported pipelines / Image embedding pipelines"
• "Supported pipelines / Music embedding pipelines"
• "Developer guides / Contributing / Contributing guide"

Thanks

Many thanks to all those who contributed to this release! @binbinlv @Chiiizzzy @derekdqc @filip-halt @fzliu @GuoRentong @guoxiangzhou @jaelgu @jeffoverflow @jennyli-z @junjiejiangjjj @LoveEachDay @NbnbZero @oneseer @shanghaikid @shiyu22 @Tumao727 @wxywb @yanliang567 @zc277584121