Your PyTorch AI Factory - Flash enables you to easily configure and run complex AI recipes for over 15 tasks across 7 data domains
jsonargparse avoiding reliance on non-public internal logic (#1620)pytorch-tabular>=1.0 (#1545)numpy (#1595)Full Changelog: https://github.com/Lightning-Universe/lightning-flash/compare/0.8.1.post0...0.8.2
torchmetrics version for compatibility by @Borda in https://github.com/Lightning-AI/lightning-flash/pull/1495
sahi to fix object detection when installing in a fresh environment by @ethanwharris in https://github.com/Lightning-AI/lightning-flash/pull/1496
numpy for type compatibility by @Borda in https://github.com/Lightning-AI/lightning-flash/pull/1504
Full Changelog: https://github.com/Lightning-AI/lightning-flash/compare/0.8.1...0.8.1.post0
lightning_utilties function into flash by @uakarsh in https://github.com/Lightning-AI/lightning-flash/pull/1457
image_classifier_head to classifier_head by @Abelarm in https://github.com/Lightning-AI/lightning-flash/pull/1464
icevision not installed if module isn't found (loading data) by @krshrimali in https://github.com/Lightning-AI/lightning-flash/pull/1474
Full Changelog: https://github.com/Lightning-AI/lightning-flash/compare/0.8.0...0.8.1
We are elated to announce the release of Lightning Flash v0.8, a feature-rich release with improved testing to ensure better user experience for all our lovely users! The team at Lightning AI and our community contributors have been working hard for this release, and nothing makes us happier to share all their lovely contributions with you.
We discuss major features and changes below. For a curated list, scroll to the bottom to see all the pull requests included for this release.
Before this release, Lightning Flash worked well on a single-core TPU (training, validation, and prediction), but failed comprehensively on multiple cores. This release has enabled training and validation support for multi-core TPUs, allowing users to try out their models on TPUs using Lightning Flash. Prediction of multi-core TPUs is an ongoing effort, and we hope to bring it to you in the near future.
| Before v0.8 | After v0.8 | |
|---|---|---|
| Single core | Training, Validation, Prediction | Training, Validation, Prediction |
| Multiple cores | Not supported | Training, Validation |
As we move ahead, and we see more users trying the TPUs with Lightning Flash, we expect that there might be unseen errors or issues, and we will be looking forward to addressing them as we get a chance. So please don't hesitate to let us know your experience!
fsspec arrives into Lightning Flash ☁️Before this release, users had to download a dataset or a file from the URL and pass it to our data loader classes. This was a pain point that we are happy to let go of in this release. Starting v0.8, you'll not have to download any of those files locally, and you can just pass the file URL - and expect it to work!
| Before v0.8 | After v0.8 | |
|---|---|---|
| Example |
Download titanic.csv from the URL and pass the path to the train_file argument:
|
Just pass the URL to train_file argument: |
For more details, feel free to check out the documentation here.
At times, it's required to load raw data, or pre-process videos before progressing to loading data and training the model. These raw data for Video Classification, are mostly available as tensors, and before this release - one had to save them again in video files, and pass the paths to the data loading classes in Flash. Starting this release, we now support loading data from tensors for Video Classification.
import torch
from flash.video import VideoClassifier, VideoClassificationData
import flash
# 5 number of frames, 3 channels, height = 10 and width = 10
mock_tensors = torch.randint(size=(3, 5, 10, 10), low=0, high=255)
datamodule = VideoClassificationData.from_tensors(
train_data=[mock_tensors, mock_tensors], # can also stack: torch.stack((mock_tensors, mock_tensors))
train_targets=["patient", "doctor"],
predict_data=[mock_tensors],
batch_size=1,
)
model = VideoClassifier(num_classes=datamodule.num_classes, pretrained=False, backbone="slow_r50", labels=datamodule.labels)
trainer = flash.Trainer(max_epochs=1)
trainer.finetune(model, datamodule=datamodule)
This will also come in handy for those having multi-modal pipelines who don't want to save the output of a model to files and instead pass the raw data to the next model, saving you quite a lot of time wasted in the conversion process.
One of the community-driven contributions that we are proud to share. Before this release, a user had to pass an input transform class for each stage, which was cumbersome. With this release, you can just pass transform=<YourTransformClass> to the required method. This is a breaking change, and if you are not sure how to resolve this, please create an issue and we'll be happy to help!
| Before v0.8 | After v0.8 | |
|---|---|---|
| Example |
|
|
Note that, within your InputTransform class, you can have <stage>_per_batch_transform_on_device methods to support various stages.
class SampleInputTransform(InputTransform):
def per_sample_transform(self):
def fn(x):
return x
return fn
def train_per_batch_transform_on_device(self) -> Callable:
return ...
def val_per_batch_transform_on_device(self) -> Callable:
return ...
def test_per_batch_transform_on_device(self) -> Callable:
return ...
def predict_per_batch_transform_on_device(self) -> Callable:
return ...
If you aren't aware yet, Lightning Flash supports serving models. Starting this release, Object Detection is added to the beautiful category of tasks that can be served using Lightning Flash. Below is an example of how the inference server code for object detection will look like:
# Inference Server
from flash.image import ObjectDetector
model = ObjectDetector.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/0.8.0/object_detection_model.pt")
model.serve()
For more details, check out the documentation here.
from_tensors for VideoClassification (#1389)torchvision as a requirement to datatype_audio.txt as it's used for Audio Classification (#1425)figsize and limit_nb_samples for showing batch images (#1381)from_lists for Tabular Classification and Regression (#1337)from_dicts for Tabular Classification and Regression (#1331)ImageEmbedder SSL training for all image classifier backbones (#1264)AudioClassificationData (#1085)ObjectDetector (#1370)ImageClassificationData from PIL images with from_images (#1372)ObjectDetectionData with from_numpy, from_images, and from_tensors (#1372)from_csv methods (#1387)as_embedder (#1396)SemanticSegmentationData (#1399)ImageEmbedder dependency on VISSL to optional (#1276)SemanticSegmentationData to use albumentations instead of Kornia (#1313)sd2 extension, because SoundFile (for sd2 extension) doesn't accept fsspec objects. (#1409)lightning-flash with learn2learn (#1376)Trainer class (#1410)on_<stage>_dataloader hooks (removed in PL 1.7.0) (#1410)probabilities list is empty (#1410)DataLoader is initialized (in PL 1.7.0 changing attributes after initialization doesn't do anything) (#1410)show_train_batch for subplots with rows > 1. (#1339)baal. (#1315)TabularClassifier or TabularRegressor checkpoint could not be served (#1324)freeze_unfreeze and unfreeze_milestones finetuning strategies could not be used in tandem with a onecyclelr LR scheduler (#1329)freeze_unfreeze or unfreeze_milestones strategies (#1329)processor_backbone argument to SpeechRecognition was not used for decoding outputs (#1362).npy files could not be used with SemanticSegmentationData (#1369)@akihironitta @aniketmaurya @Borda @carmocca @ethanwharris @JustinGoheen @krshrimali @ligaz @Nico995 @uakarsh
If we forgot someone let us know :smiley:
@Borda @ethanwharris @kaushikb11 @krshrimali
If we forgot someone let us know :smiley:
ObjectDetector, InstanceSegmentation, and KeypointDetector would error with the latest version of some requirements (#1306)LightningModule support for Flash data modules. (#1281)@Borda @ethanwharris @krshrimali @rohitgr7
If we forgot someone let us know :smiley:
ObjectDetector, InstanceSegmentation, and KeypointDetector (#1267)SpeechRecognition task (#1275)QuestionAnswering task with certain model types (#1275)@ethanwharris @krshrimali
If we forgot someone let us know :smiley:
punkt module needs to be downloaded first. (#1215)pretraining_transforms in the ImageEmbedder was never called. (1196)BASE_MODEL_NAME was not in the dict for dino and moco strategies. (1196)torch==1.11.0 (#1234)ObjectDetector, InstanceSegmentation, and KeypointDetector (#1222)InstanceSegmentation would fail if samples had an inconsistent number of bboxes, labels, and masks (these will now be treated as negative samples) (#1222)ImageEmbedder class. (#1217)ObjectDetector, InstanceSegmentation, and KeypointDetector would log train and validation metrics with the same name (#1252)ReduceLROnPlateau would raise an error (#1251)ImageEmbedder (#1256)@aisensiy @andife @aniketmaurya @Borda @dudeperf3ct @ethanwharris @krshrimali
If we forgot someone let us know :smiley:
torchvision.transforms.Normalize as transform_kwargs in the ImageClassificationInputTransform (#1178)available_outputs method to the Task (#1206)VideoClassifier (#1189)Task (#1203)from_* methods / the DataModule (#1204)@aisensiy @AndresAlgaba @Borda @ethanwharris
If we forgot someone due to not matching commit email with GitHub account, let us know :]
KeypointDetectionData.from_coco (#1102)from_csv and from_data_frame to VideoClassificationData (#1117)SemanticSegmentationData.from_folders where mask files have different extensions to the image files (#1130)FlashRegistry of Available Heads for flash.image.ImageClassifier (#1152)ObjectDetectionData.from_files (#1154)Output object (or a string e.g. "labels") to the flash.Trainer.predict method (#1157)TargetFormatter object to from_* methods for classification to override target handling (#1171)Wav2Vec2Processor to AutoProcessor and seperate it from backbone [optional] (#1075)ClassificationInput to ClassificationInputMixin (#1116)learning_rate for all tasks to be None, corresponding to the default for your chosen optimizer (#1172)embedding_sizes to the TabularClassifier and TabularRegressor tasks (#1067)val_split to the DataModule would not have the desired effect (#1079)predict_data_frame to ImageClassificationData.from_data_frame raised an error (#1088)AudioClassificationData.from_numpy (#1096)SpeechRecognitionData.from_files for training / validating / testing (#1097)SpeechRecognitionData.from_csv or from_json when predicting without targets (#1097)SpeechRecognitionData.from_datasets did not work as expected (#1097)SemanticSegmentationData.from_folders raised an error (#1101)predict_folder argument to from_coco / from_voc / from_via in IceVision tasks (#1102)ObjectDetectionData.from_voc and ObjectDetectionData.from_via (#1102)InstanceSegmentationData.from_coco would raise an error if not using file-based masks (#1102)InstanceSegmentationData.from_voc (#1102)from_fiftyone classmethods did not work correctly with a predict_dataset (#1136)labels property would return None when using ObjectDetectionData.from_fiftyone (#1136)TabularData would not work correctly with no categorical variables (#1144)TabularForecastingData for prediction would only yield a single sample per series (#1149)ObjectDetector, KeypointDetector, and InstanceSegmentation tasks were not always frozen correctly when finetuning (#1163)DataModule.multi_label would sometimes be None when it had been inferred to be False (#1165)Seq2SeqData base class (use TranslationData or SummarizationData directly) (#1128)Output object directly to the model (#1157)@Actis92 @AjinkyaIndulkar @bartonp2 @Borda @daMichaelB @ethanwharris @flozi00 @karthikrangasai @MikeTrizna
If we forgot someone due to not matching commit email with GitHub account, let us know :]