Spark Nlp Versions Save

State of the Art Natural Language Processing

4.4.0

1 year ago

:loudspeaker: Overview

We are thrilled to announce the release of Spark NLP πŸš€ 4.4.0! This release includes new features such as a New BART for NLG, translation, and comprehension; a new ConvNeXT Transformer for Image Classification, a new Zero-Shot Text Classification by BERT, 4000+ new state-of-the-art models, and more enhancements and bug fixes.

We want to thank our community for their valuable feedback, feature requests, and contributions. Our Models Hub now contains over 17,000+ free and truly open-source models & pipelines. πŸŽ‰

Spark NLP has a new home! https://sparknlp.org is where you can find all the documentation, models, and demos for Spark NLP. It aims to provide valuable resources to anyone interested in 100% open-source NLP solutions by using Spark NLP πŸš€.

:fire: New Features

ConvNeXT Image Classification (By Facebook)

NEW: Introducing ConvNextForImageClassification annotator in Spark NLP πŸš€. ConvNextForImageClassification can load ConvNeXT models that compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

This annotator is compatible with all the models trained/fine-tuned by using ConvNextForImageClassification for PyTorch or TFConvNextForImageClassification for TensorFlow models in HuggingFace πŸ€—

A ConvNet: ImageNet-1K classification results for β€’ ConvNets and β—¦ vision Transformers. Each bubble’s area is proportional to FLOPs of a variant in a model family. by Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie.

BART for NLG, Translation, and Comprehension (By Facebook)

NEW: Introducing BartTransformer annotator in Spark NLP πŸš€. BartTransformer can load BART models fine-tuned for tasks like summarizations.

This annotator is compatible with all the models trained/fine-tuned by using BartForConditionalGeneration for PyTorch or TFBartForConditionalGeneration for TensorFlow models in HuggingFace πŸ€—

The abstract explains that Bart uses a standard seq2seq/machine translation architecture, similar to BERT's bidirectional encoder and GPT's left-to-right decoder. The pretraining task involves randomly shuffling the original sentences and replacing text spans with a single mask token. BART is effective for text generation and comprehension tasks, matching RoBERTa's performance with similar training resources on GLUE and SQuAD. It also achieves new state-of-the-art results on various summarization, dialogue, and question-answering tasks with gains of up to 6 ROUGE.

The Bart model was proposed in BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer

Zero-Shot for Text Classification by BERT

NEW: Introducing BertForZeroShotClassification annotator for Zero-Shot Text Classification in Spark NLP πŸš€. You can use the BertForZeroShotClassification annotator for text classification with your labels! :100:

Zero-Shot Learning (ZSL): Traditionally, ZSL most often referred to a fairly specific type of task: learning a classifier on one set of labels and then evaluating on a different set of labels that the classifier has never seen before. Recently, especially in NLP, it's been used much more broadly to get a model to do something it wasn't explicitly trained to do. A well-known example of this is in the GPT-2 paper where the authors evaluate a language model on downstream tasks like machine translation without fine-tuning on these tasks directly.

Let's see how easy it is to just use any set of labels our trained model has never seen via the setCandidateLabels() param:

zero_shot_classifier = BertForZeroShotClassification \
    .pretrained() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("class") \
    .setCandidateLabels(["urgent", "mobile", "travel", "movie", "music", "sport", "weather", "technology"])

For Zero-Short Multi-class Text Classification:

+----------------------------------------------------------------------------------------------------------------+--------+
|result                                                                                                          |result  |
+----------------------------------------------------------------------------------------------------------------+--------+
|[I have a problem with my iPhone that needs to be resolved asap!!]                                              |[mobile]|
|[Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.]|[mobile]|
|[I have a phone and I love it!]                                                                                 |[mobile]|
|[I want to visit Germany and I am planning to go there next year.]                                              |[travel]|
|[Let's watch some movies tonight! I am in the mood for a horror movie.]                                         |[movie] |
|[Have you watched the match yesterday? It was a great game!]                                                    |[sport] |
|[We need to hurry up and get to the airport. We are going to miss our flight!]                                  |[urgent]|
+----------------------------------------------------------------------------------------------------------------+--------+

For Zero-Short Multi-class Text Classification:

+----------------------------------------------------------------------------------------------------------------+-----------------------------------+
|result                                                                                                          |result                             |
+----------------------------------------------------------------------------------------------------------------+-----------------------------------+
|[I have a problem with my iPhone that needs to be resolved asap!!]                                              |[urgent, mobile, movie, technology]|
|[Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.]|[urgent, technology]               |
|[I have a phone and I love it!]                                                                                 |[mobile]                           |
|[I want to visit Germany and I am planning to go there next year.]                                              |[travel]                           |
|[Let's watch some movies tonight! I am in the mood for a horror movie.]                                         |[movie]                            |
|[Have you watched the match yesterday? It was a great game!]                                                    |[sport]                            |
|[We need to hurry up and get to the airport. We are going to miss our flight!]                                  |[urgent, travel]                   |
+----------------------------------------------------------------------------------------------------------------+-----------------------------------+

:star::bug: Improvements & Bug Fixes

  • Add a new nerHasNoSchema param for NerConverter when labels coming from NerDLMOdel and NerCrfModel don't have any schema
  • Set custom entity name in Data2Chunk via setEntityName param
  • Fix loading WordEmbeddingsModel bug when loading a model from S3 via the cache_folder config
  • Fix the WordEmbeddingsModel bug failing when it's used with setEnableInMemoryStorage set to True and LightPipeline
  • Remove deprecated parameter enablePatternRegex from EntityRulerApproach & EntityRulerModel
  • Welcoming 3 new Databricks runtimes to our Spark NLP family:
    • Databricks 12.2 LTS
    • Databricks 12.2 LTS ML
    • Databricks 12.2 LTS ML GPU
  • Deprecate Python 3.6 in Spark NLP 4.4.0

:floppy_disk: Models

Spark NLP 4.4.0 comes with more than 4300+ new state-of-the-art pre-trained transformer models in multi-languages.

Model Name Lang
BertForZeroShotClassification bert_base_cased_zero_shot_classifier_xnli en
ConvNextForImageClassification image_classifier_convnext_tiny_224_local en
BartTransformer distilbart_xsum_12_6 en
BartTransformer bart_large_cnn en
BertForQuestionAnswering bert_qa_case_base en
HubertForCTC asr_swin_exp_w2v2t_nl_hubert_s319 nl
BertForTokenClassification bert_token_classifier_base_chinese_ner zh

The complete list of all 17000+ models & pipelines in 230+ languages is available on Models Hub

:notebook: New Notebooks

Notebooks Colab
Zero-Shot Text Classification Open In Colab
Document Summarization with BART Open In Colab

:book: Documentation

Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
  • Medium Spark NLP articles
  • YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==4.4.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.4.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.4.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.4.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.4.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.4.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.4.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.4.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>4.4.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.4.0</version>
</dependency>

FAT JARs

What's Changed

Contributors

@Cabir40 @bunyamin-polat @danilojsl @dcecchini @Meryem1425 @C-K-Loan @agsfer @maziyarpanahi @jfernandrezj @jsl-builder @DevinTDHa @josejuanmartinez @aymanechilah

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.3.2...4.4.0

4.3.2

1 year ago

:loudspeaker: Overview

Spark NLP 4.3.2 πŸš€ comes with a new support for S3 in training classes to read and load CoNLL and CoNLL-U formats, support for NER tags without any schema in NerConverter, improving dedicated and self-hosted examples with more guides, and other enhancements and bug fixes!

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: New Features & Enhancements

πŸ› Bug Fixes

:book: Documentation

Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
  • Medium Spark NLP articles
  • YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==4.3.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.3.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.3.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>4.3.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.3.2</version>
</dependency>

FAT JARs

What's Changed

New Contributors

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.3.1...4.3.2

4.3.1

1 year ago

:loudspeaker: Overview

Spark NLP 4.3.1 πŸš€ comes with a new SpacyToAnnotation feature to import documents, sentences, and tokens from spaCy and similar libraries into Spark NLP pipelines. We have also made other improvements in this patch release.

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: New Features & Enhancements

  • Easily use external Sentences and Tokens from external libraries such as spaCy in Spark NLP pipeline
# this is how your file from spaCy would look like
! cat ./multi_doc_tokens.json

[
  {
    "tokens": ["John", "went", "to", "the", "store", "last", "night", ".", "He", "bought", "some", "bread", "."],
    "token_spaces": [true, true, true, true, true, true, false, true, true, true, true, false, false],
    "sentence_ends": [7, 12]
  },
  {
    "tokens": ["Hello", "world", "!", "How", "are", "you", "today", "?", "I", "'m", "fine", "thanks", "."],
    "token_spaces": [true, false, true, true, true, true, false, true, false, true, true, false, false],
    "sentence_ends": [2, 7, 12]
  }
]

# we are now going to prepare these documents, sentence, and tokens for Spark NLP
from sparknlp.training import SpacyToAnnotation

nlp_reader = SpacyToAnnotation()
result = nlp_reader.readJsonFile(spark, "./multi_doc_tokens.json")

result.printSchema()
# now you have all the annotations for documents, sentences, and tokens needed in Spark NLP
root
 |-- document: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- annotatorType: string (nullable = true)
 |    |    |-- begin: integer (nullable = false)
 |    |    |-- end: integer (nullable = false)
 |    |    |-- result: string (nullable = true)
 |    |    |-- metadata: map (nullable = true)
 |    |    |    |-- key: string
 |    |    |    |-- value: string (valueContainsNull = true)
 |    |    |-- embeddings: array (nullable = true)
 |    |    |    |-- element: float (containsNull = false)
 |-- sentence: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- annotatorType: string (nullable = true)
 |    |    |-- begin: integer (nullable = false)
 |    |    |-- end: integer (nullable = false)
 |    |    |-- result: string (nullable = true)
 |    |    |-- metadata: map (nullable = true)
 |    |    |    |-- key: string
 |    |    |    |-- value: string (valueContainsNull = true)
 |    |    |-- embeddings: array (nullable = true)
 |    |    |    |-- element: float (containsNull = false)
 |-- token: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- annotatorType: string (nullable = true)
 |    |    |-- begin: integer (nullable = false)
 |    |    |-- end: integer (nullable = false)
 |    |    |-- result: string (nullable = true)
 |    |    |-- metadata: map (nullable = true)
 |    |    |    |-- key: string
 |    |    |    |-- value: string (valueContainsNull = true)
 |    |    |-- embeddings: array (nullable = true)
 |    |    |    |-- element: float (containsNull = false)
  • Implement params parameter which can supply custom configurations to the SparkSession in Scala (to be sync with Python)
val hadoopAwsVersion: String = "3.3.1"
val awsJavaSdkVersion: String = "1.11.901"

val extraParams: Map[String, String] = Map(
  "spark.jars.packages" -> ("org.apache.hadoop:hadoop-aws:" + hadoopAwsVersion + ",com.amazonaws:aws-java-sdk:" + awsJavaSdkVersion),
  "spark.hadoop.fs.s3a.path.style.access" -> "true")

val spark = SparkNLP.start(params = extraParams)
  • Add entity field to the metadata in Date2Chunk
  • Fix ViT models & pipelines examples in Models Hub

:notebook: New Notebooks

Spark NLP
Import Tokens from spaCy or a JSON file

:book: Documentation

Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
  • Medium Spark NLP articles
  • YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==4.3.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.1

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.1

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.3.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.3.1</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>4.3.1</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.3.1</version>
</dependency>

FAT JARs

What's Changed

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.3.0...4.3.1

4.3.0

1 year ago

:loudspeaker: Overview

We are very excited to release Spark NLP πŸš€ 4.3.0! This has been one of the biggest releases we have ever done and we are so proud to share this with our community! πŸŽ‰

This release extends support for another Image Classification by introducing Swin Transformer, also extending support for speech recognition by introducing HuBERT annotator, a brand new modern extractive transformer-based Question answering (QA) annotator for tasks like SQuAD based on CamemBERT architecture, new Databricks & EMR with support for Spark 3.3, 1000+ state-of-the-art models, and many more enhancements and bug fixes!

We are also celebrating crossing 12600+ free and open-source models & pipelines in our Models Hub. πŸŽ‰ As always, we would like to thank our community for their feedback, questions, and feature requests.

:fire: New Features

HuBERT

NEW: Introducing HubertForCTC annotator in Spark NLP πŸš€. HubertForCTC can load HuBERT models that match or surpasses the SOTA approaches for speech representation learning for speech recognition, generation, and compression. The Hidden-Unit BERT (HuBERT) approach was proposed for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. This annotator is compatible with all the models trained/fine-tuned by using HubertForCTC for PyTorch or TFHubertForCTC for TensorFlow models in HuggingFace πŸ€—

image

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.

Swin Transformer

NEW: Introducing SwinForImageClassification annotator in Spark NLP πŸš€. SwinForImageClassification can load transformer-based deep learning models with state-of-the-art performance in vision tasks. Swin Transformer precedes Vision Transformer (ViT) (Dosovitskiy et al., 2020) with great accuracy and efficiency. This annotator is compatible with all the models trained/fine-tuned by using SwinForImageClassification for PyTorch or TFSwinForImageClassification for TensorFlow models in HuggingFace πŸ€—

image

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows by Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.

Zero-Shot for Named Entity Recognition

Zero-Shot Learning refers to the process by which a model learns how to recognize objects (image, text, any features) without any labeled training data to help in the classification.

NEW: Introducing ZeroShotNerModel annotator in Spark NLP πŸš€. You can use the ZeroShotNerModel annotator to construct simple questions/answers mapped to NER labels like PERSON, NORP and etc. We use RoBERTa for Question Answering architecture behind the hood and this allows you to use any of the 460 models available on Models Hub to build your Zero-shot Entity Recognition with zero training dataset!

zero_shot_ner = ZeroShotNerModel.pretrained("roberta_base_qa_squad2", "en") \
    .setEntityDefinitions(
        {
            "NAME": ["What is his name?", "What is my name?", "What is her name?"],
            "CITY": ["Which city?", "Which is the city?"]
        }) \
    .setInputCols(["sentence", "token"]) \
    .setOutputCol("zero_shot_ner")

This powerful annotator with such simple rules can detect those entities from the following input: "My name is Clara, I live in New York and Hellen lives in Paris."

+-----------------------------------------------------------------+------+------+----------+------------------+
    |result                                                           |result|word  |confidence|question          |
    +-----------------------------------------------------------------+------+------+----------+------------------+
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-CITY|Paris |0.5328949 |Which is the city?|
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-NAME|Clara |0.9360068 |What is my name?  |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-CITY|New   |0.83294415|Which city?       |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|I-CITY|York  |0.83294415|Which city?       |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-NAME|Hellen|0.45366877|What is her name? |
    +-----------------------------------------------------------------+------+------+----------+------------------+

CamemBERT for Question Answering

NEW: Introducing CamemBertForQuestionAnswering annotator in Spark NLP πŸš€. CamemBertForQuestionAnswering can load CamemBERT Models with a span classification head on top for extractive question-answering tasks like SQuAD (linear layers on top of the hidden-states output to compute span start logits and span end logits). This annotator is compatible with all the models trained/fine-tuned by using CamembertForQuestionAnswering for PyTorch or TFCamembertForQuestionAnswering for TensorFlow in HuggingFace πŸ€—

Models Hub

Introduces a new filter by annotator which should help to navigate and find models easier:

image

:star::bug: Improvements & Bug Fixes

  • New Date2Chunk annotator to convert DATE outputs coming from DateMatcher and MultiDateMatcher annotators to CHUNK that is acceptable by a wider range of annotators
  • Spark NLP 4.3.0 supports Apple Silicon M1 and M2 (still under experimental status until GitHub supports Apple Silicon officially). We have refactored the name m1 to silicon and apple_silicon in our code for better clarity
  • Add new templates for issues, docs, and feature requests on GitHub
  • Add a new log4j2 properties for Spark 3.3.x coming with Log4j 2.x to control the logs on Apache Spark
  • Cross compatibility for all saved pipelines for all major releases of Apache Spark and PySpark
  • Relocating Spark NLP examples to the examples directory in our main repository. We will update them on each release, will keep a history of the changes for each version, adding more languages, especially more use cases with Java and Scala
  • Add PyDoc documentation for ResourceDownloader in Python (clearCache(), showPublicModels(), showPublicPipelines(), and showAvailableAnnotators() )
  • Fix calculating delimiter id in CamemBERT annotators. The delimiter id is actually correct and doesn't need any offset
  • Fix AnalysisException exception that requires a different caught message for Spark 3.3
  • Fix copying existing models & pipelines on S3 before unzipping when cache_pretrained is defined as S3 bucket
  • Fix copying existing models & pipelines on GCP before unzipping when cache_pretrained is defined as GCP bucket
  • Fix loadSavedModel() trying to load external models for private buckets on S3 with better error handling and warnings
  • Enable the params argument in the Spark NLP start function. You can create a params = {} with all Spark NLP and Apache Spark configs and pass it when starting the Spark NLP session
  • Add support for doc id in CoNLL() class when trying to read CoNLL files with id inside each document's header
  • Welcoming 6 new Databricks runtimes to our Spark NLP family:
    • Databricks 12.0
    • Databricks 12.0 ML
    • Databricks 12.0 ML GPU
    • Databricks 12.1
    • Databricks 12.1 ML
    • Databricks 12.1 ML GPU
  • Welcoming 2 new EMR 6.x series to our Spark NLP family:
    • EMR 6.8.0 (Apache Spark 3.3.0 / Hadoop 3.2.1)
    • EMR 6.9.0 (Apache Spark 3.3.0 / Hadoop 3.3.3)
  • New article for semantic similarity with Spark NLP on Play/API/Swagger/ https://medium.com/spark-nlp/semantic-similarity-with-sparknlp-da148fafa3d8

Dependencies & Code Changes

  • Update Apache Spark 3.3.1 (not shipped with Spark NLP
  • Update GCP to 2.16.0
  • Update Scala test to 3.2.14
  • Start publishing spark-nlp-m1 Maven package as spark-nlp-silicon
  • Rename all read model traits to a generic name. A new ai module paving a path to another DL engine
  • Rename TF backends to more generic DL names
  • Refactor more duplicate codes in transformer embeddings

:floppy_disk: Models

Spark NLP 4.3.0 comes with 1000+ state-of-the-art pre-trained transformer models in many languages.

Model Name Lang
DistilBertForQuestionAnswering distilbert_qa_en_de_vi_zh_es_model xx
DistilBertForQuestionAnswering distilbert_qa_extractive en
DistilBertForQuestionAnswering distilbert_qa_base_cased_squadv2 xx
RoBertaForQuestionAnswering roberta_qa_roberta en
RoBertaForQuestionAnswering roberta_qa_ca_v2_squac_ca_catalan ca
T5Transformer t5_flan_small xx
T5Transformer t5_t2t_adex_prompt en
T5Transformer t5_punctuation fr
T5Transformer t5_jainu ja
T5Transformer t5_diversiformer de
T5Transformer t5_small_summarization ro
T5Transformer t5_uk_summarizer uk
T5Transformer t5_vi_small vi
T5Transformer t5_mini_nl8 fi
HubertForCTC asr_hubert_large_ls960 en
SwinForImageClassification image_classifier_swin_tiny_patch4_window7_224 en
CamemBertForQuestionAnswering camembert_base_qa_fquad fr

Spark NLP covers the following languages:

English ,Multilingual ,Afrikaans ,Afro-Asiatic languages ,Albanian ,Altaic languages ,American Sign Language ,Amharic ,Arabic ,Argentine Sign Language ,Armenian ,Artificial languages ,Atlantic-Congo languages ,Austro-Asiatic languages ,Austronesian languages ,Azerbaijani ,Baltic languages ,Bantu languages ,Basque ,Basque (family) ,Belarusian ,Bemba (Zambia) ,Bengali, Bangla ,Berber languages ,Bihari ,Bislama ,Bosnian ,Brazilian Sign Language ,Breton ,Bulgarian ,Catalan ,Caucasian languages ,Cebuano ,Celtic languages ,Central Bikol ,Chichewa, Chewa, Nyanja ,Chilean Sign Language ,Chinese ,Chuukese ,Colombian Sign Language ,Congo Swahili ,Croatian ,Cushitic languages ,Czech ,Danish ,Dholuo, Luo (Kenya and Tanzania) ,Dravidian languages ,Dutch ,East Slavic languages ,Eastern Malayo-Polynesian languages ,Efik ,Esperanto ,Estonian ,Ewe ,Fijian ,Finnish ,Finnish Sign Language ,Finno-Ugrian languages ,French ,French-based creoles and pidgins ,Ga ,Galician ,Ganda ,Georgian ,German ,Germanic languages ,Gilbertese ,Greek (modern) ,Greek languages ,Gujarati ,Gun ,Haitian, Haitian Creole ,Hausa ,Hebrew (modern) ,Hiligaynon ,Hindi ,Hiri Motu ,Hungarian ,Icelandic ,Igbo ,Iloko ,Indic languages ,Indo-European languages ,Indo-Iranian languages ,Indonesian ,Irish ,Isoko ,Isthmus Zapotec ,Italian ,Italic languages ,Japanese ,Japanese ,Kabyle ,Kalaallisut, Greenlandic ,Kannada ,Kaonde ,Kinyarwanda ,Kirundi ,Kongo ,Korean ,Kwangali ,Kwanyama, Kuanyama ,Latin ,Latvian ,Lingala ,Lithuanian ,Louisiana Creole ,Lozi ,Luba-Katanga ,Luba-Lulua ,Lunda ,Lushai ,Luvale ,Macedonian ,Malagasy ,Malay ,Malayalam ,Malayo-Polynesian languages ,Maltese ,Manx ,Marathi (MarāṭhΔ«) ,Marshallese ,Mexican Sign Language ,Mon-Khmer languages ,Morisyen ,Mossi ,Multiple languages ,Ndonga ,Nepali ,Niger-Kordofanian languages ,Nigerian Pidgin ,Niuean ,North Germanic languages ,Northern Sotho, Pedi, Sepedi ,Norwegian ,Norwegian BokmΓ₯l ,Norwegian Nynorsk ,Nyaneka ,Oromo ,Pangasinan ,Papiamento ,Persian (Farsi) ,Peruvian Sign Language ,Philippine languages ,Pijin ,Pohnpeian ,Polish ,Portuguese ,Portuguese-based creoles and pidgins ,Punjabi (Eastern) ,Romance languages ,Romanian ,Rundi ,Russian ,Ruund ,Salishan languages ,Samoan ,San Salvador Kongo ,Sango ,Semitic languages ,Serbo-Croatian ,Seselwa Creole French ,Shona ,Sindhi ,Sino-Tibetan languages ,Slavic languages ,Slovak ,Slovene ,Somali ,South Caucasian languages ,South Slavic languages ,Southern Sotho ,Spanish ,Spanish Sign Language ,Sranan Tongo ,Swahili ,Swati ,Swedish ,Tagalog ,Tahitian ,Tai ,Tamil ,Telugu ,Tetela ,Tetun Dili ,Thai ,Tigrinya ,Tiv ,Tok Pisin ,Tonga (Tonga Islands) ,Tonga (Zambia) ,Tsonga ,Tswana ,Tumbuka ,Turkic languages ,Turkish ,Tuvalu ,Tzotzil ,Ukrainian ,Umbundu ,Uralic languages ,Urdu ,Venda ,Venezuelan Sign Language ,Vietnamese ,Wallisian ,Walloon ,Waray (Philippines) ,Welsh ,West Germanic languages ,West Slavic languages ,Western Malayo-Polynesian languages ,Wolaitta, Wolaytta ,Wolof ,Xhosa ,Yapese ,Yiddish ,Yoruba ,Yucatec Maya, Yucateco ,Zande (individual language) ,Zulu

The complete list of all 12600+ models & pipelines in 230+ languages is available on Models Hub

:notebook: New Notebooks

Notebooks
New params{} in Spark NLP start()
CamemBertForQuestionAnswering
Zero-shot NER

:book: Documentation

Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!
  • Medium Spark NLP articles
  • YouTube Spark NLP video tutorials

Installation

Python

#PyPI

pip install spark-nlp==4.3.0

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.3.0

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.3.0

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:4.3.0

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.0

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.3.0

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.3.0</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.3.0</version>
</dependency>

spark-nlp-silicon:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-silicon_2.12</artifactId>
    <version>4.3.0</version>
</dependency>

spark-nlp-aarch64:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.3.0</version>
</dependency>

FAT JARs

What's Changed

Contributors

@Cabir40 @bunyamin-polat @danilojsl @dcecchini @Meryem1425 @C-K-Loan @agsfer @maziyarpanahi @jfernandrezj @jsl-builder @DevinTDHa @josejuanmartinez @aymanechilah

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.8...4.3.0

4.2.8

1 year ago

:loudspeaker: Overview

Spark NLP 4.2.8 πŸš€ comes with some important bug fixes and improvements. As a result, we highly recommend to update to this latest version if you are using Spark NLP 4.2.x.

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: :bug: Bug Fixes & Improvements

  • Fix the issue with optional keys (labels) in metadata when using XXXForSequenceClassitication annotators. This fixes Some(neg) -> 0.13602075 as neg -> 0.13602075 to be in harmony with all the other classifiers. https://github.com/JohnSnowLabs/spark-nlp/pull/13396

before 4.2.8:

+-----------------------------------------------------------------------------------------------+
|label                                                                                          |
+-----------------------------------------------------------------------------------------------+
|[{category, 0, 87, pos, {sentence -> 0, Some(neg) -> 0.13602075, Some(pos) -> 0.8639792}, []}] |
|[{category, 0, 47, neg, {sentence -> 0, Some(neg) -> 0.7505674, Some(pos) -> 0.24943262}, []}] |
|[{category, 0, 17, pos, {sentence -> 0, Some(neg) -> 0.31065974, Some(pos) -> 0.6893403}, []}] |
|[{category, 0, 71, neg, {sentence -> 0, Some(neg) -> 0.5079189, Some(pos) -> 0.4920811}, []}]  |
+-----------------------------------------------------------------------------------------------+

after 4.2.8:

+-----------------------------------------------------------------------------------+
|label                                                                              |
+-----------------------------------------------------------------------------------+
|[{category, 0, 87, pos, {sentence -> 0, neg -> 0.13602075, pos -> 0.8639792}, []}] |
|[{category, 0, 47, neg, {sentence -> 0, neg -> 0.7505674, pos -> 0.24943262}, []}] |
|[{category, 0, 17, pos, {sentence -> 0, neg -> 0.31065974, pos -> 0.6893403}, []}] |
|[{category, 0, 71, neg, {sentence -> 0, neg -> 0.5079189, pos -> 0.4920811}, []}]  |
+-----------------------------------------------------------------------------------+
  • Introducing a config to skip LightPipeline validation for inputCols on the Python side for projects depending on Spark NLP. This toggle should only be used for specific annotators that do not follow the convention of predefined inputAnnotatorTypes and outputAnnotatorType https://github.com/JohnSnowLabs/spark-nlp/pull/13402

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.8

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

FAT JARs

What's Changed

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.7...4.2.8

4.2.7

1 year ago

:loudspeaker: Overview

Spark NLP 4.2.7 πŸš€ comes with some important bug fixes and improvements. As a result, we highly recommend to update to this latest version if you are using Spark NLP 4.2.x.

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:bug: :star: Bug Fixes & Enhancements

  • Fix outputAnnotatorType issue in pipelines with Finisher annotator. This change adds outputAnnotatorType to AnnotatorTransformer to avoid loading outputAnnotatorType attribute when a stage in pipeline does not use it.
  • Fix the wrong sentence index calculation in metadata by annotators in the pipeline when setExplodeSentences param was set to true in SentenceDetector annotator
  • Fix the issue in Tokenizer when a custom pattern is used with lookahead/-behinds and it has 0 width matches. This led to indexes not being calculated correctly
  • Fix missing to output embeddings in .fullAnnotate() method when parseEmbeddings param was set to True/true
  • Fix broken links to the Python API pages, as the generation of the PyDocs was slightly changed in a previous release. This makes the Python APIs accessible from the Annotators and Transformers pages like before
  • Change default values of explodeEntities and mergeEntities parameters to true in GraphExctraction annotator
  • Better error handling when there are empty paths/relations in GraphExctractionannotator. New message will better guide the user on how to configure GraphExtraction to output meaningful relationships
  • Removed the duplicated definition of method setWeightedDistPath from ContextSpellCheckerApproach

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.7

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

FAT JARs

What's Changed

@dcecchini @Cabir40 @agsfer @gadde5300 @bunyamin-polat @rpranab @jdobes-cz @josejuanmartinez @diatrambitas @maziyarpanahi

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.6...4.2.7

4.2.6

1 year ago

:star: Improvements

  • Updating Spark & PySpark dependencies from 3.2.1 to 3.2.3 in provided scripts and in all the documentation

:bug: Bug Fixes

  • Fix the broken TypedDependencyParserApproach and TypedDependencyParserModel annotators used in Python (this bug was introduced in 4.2.5 release)
  • Fix the broken Python API documentation

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.6

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.6

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.6

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.6

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.6

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

FAT JARs

What's Changed

Contributors

'@gadde5300 @diatrambitas @Cabir40 @josejuanmartinez @danilojsl @jsl-builder @DevinTDHa @maziyarpanahi @dcecchini @agsfer '

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.5...4.2.6

4.2.5

1 year ago

:loudspeaker: Overview

Spark NLP 4.2.5 πŸš€ comes with a new CamemBERT for sequence classification annotator (multi-class & multi-label), new pipeline validation for LightPipeline in Python, 26 updated noteooks to use the latest TensorFlow and Transformers libraries, support for new Databricks 11.3 runtime, support for new EMR versions of 6.8 and 6.9 (only EMR versions with Spark 3.3), over 400+ state-of-the-art multi-lingual pretrained models, and bug fixes.

Do not forget to visit Models Hub with over 11700+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: New Features & improvements

  • NEW: Introducing CamemBertForSequenceClassification annotator in Spark NLP πŸš€. CamemBertForSequenceClassification can load CamemBERT Models with sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for multi-class document classification tasks. This annotator is compatible with all the models trained/fine-tuned by using CamembertForSequenceClassification for PyTorch or TFCamembertForSequenceClassification for TensorFlow in HuggingFace πŸ€—
  • NEW: Add AnnotatorType validation in Spark NLP LightPipeline. Currently, a misconfiguration of inputCols in an annotator in a pipeline raises an exception when using transform method, but in LightPipeline it only outputs empty values. This behavior can confuse users, this change introduces a validation that will raise an exception now in LightPipeline too.
    • Add outputAnnotatorType for all annotators in Python
    • Add inputAnnotatorTypes and outputAnnotatorType requirement validation for all subclasses derived from AnnotatorApproach and AnnotatorModel
    • Adding AnnotatorType validation in LightPipeline
  • NEW: Migrate 26 notenooks to import external Transformer models into Spark NLP. These notebooks now come with latest TensorFlow 2.11.0 and HuggingFace 4.25.1 releases. The notebooks also have TF signatures with data input types explicitly set to guarantee model sanity once imported into Spark NLP
  • Add validation for the number and type of columns set in TFNerDLGraphBuilder annotator. In efforts to avoid wrong definition of columns when using Spark NLP annotators in Python
  • Add more details to Alphabet error message in EntityRuler annotator to better guide users
  • Add instructions on how to resolve RocksDB incompatibilities when using Spark NLP with an M1 machine
  • Welcoming new Databricks runtimes support
    • 11.3
    • 11.3 ML
    • 11.3 GPU
  • Welcoming new EMR versions support
    • 6.8.0
    • 6.9.0
  • Refactor and implement a better error handling in ResourceDownloader. This change removes getObjectFromS3 allowing AWS SDK to rise the correspondent error. In addition, this change also refactors ResourceDownloader to reflect the intention of each credential type on the downloader
  • Implement full build and test of all unit tests base on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x major releases
  • UpdateUpgrade sbt-assembly to 1.2.0 that comes with lots of performance improvements. This benefits those who are trying to package Spark NLP as a Fat JAR
  • Update sbt to 1.8.0 with improvements and bug fixes, but mostly for CVEs fixes:
  • Use the new withIncludeScala in assemblyOption instead of value

:bug: Bug Fixes

  • Fix an issue with the BigTextMatcher Annotator, where it would not match entities with overlapping definitions. For Example, if both lung and lung cancer are defined, lung would not be matched in a given text. This was due to an abstraction error of one of the subclasses of the BigTextMatcher during construction of the underlying data structure
  • Fix indexing issue for RegexTokenizer annotator. If the document was split into sentences, the index of the sentence inside the document was not taken into consideration for the indexes of the tokens. This would lead to further issues down the pipeline, where tokens would be filtered while unpacking them for other Annotators
  • Refactor the Resolvers object in Spark NLP's dependency to avoid the conflict with the Resolvers inside the new sbt

πŸ›‘ Known Issues

  • TypedDependencyParserModel annotator fails in Python in this release (will be fixed in 4.2.6 release next week)

Models

Spark NLP 4.2.5 comes with 400+ state-of-the-art pre-trained transformer models in many languages.

Model Name Lang
RoBertaForSequenceClassification roberta_classifier_autotrain_neurips_chanllenge_1287149282 en
RoBertaForSequenceClassification roberta_classifier_autonlp_imdb_rating_625417974 en
RoBertaForSequenceClassification RoBertaForSequenceClassification bn
RoBertaForSequenceClassification roberta_classifier_autotrain_citizen_nlu_hindi_1370952776 hi
RoBertaForSequenceClassification roberta_classifier_detect_acoso_twitter es
RoBertaForQuestionAnswering roberta_qa_deepset_base_squad2 en
RoBertaForQuestionAnswering roberta_qa_icebert is
RoBertaForQuestionAnswering roberta_qa_mrm8488_base_bne_finetuned_s_c es
RoBertaForQuestionAnswering roberta_qa_base_bne_squad2 es
BertEmbeddings bert_embeddings_rbt3 zh
BertEmbeddings bert_embeddings_base_it_cased it
BertEmbeddings bert_embeddings_base_indonesian_522m id
BertEmbeddings bert_embeddings_base_german_uncased `de
BertEmbeddings bert_embeddings_base_japanese_char ja
BertEmbeddings bert_embeddings_bangla_base bn
BertEmbeddings bert_embeddings_base_arabertv01 ar

Spark NLP covers the following languages:

English ,Multilingual ,Afrikaans ,Afro-Asiatic languages ,Albanian ,Altaic languages ,American Sign Language ,Amharic ,Arabic ,Argentine Sign Language ,Armenian ,Artificial languages ,Atlantic-Congo languages ,Austro-Asiatic languages ,Austronesian languages ,Azerbaijani ,Baltic languages ,Bantu languages ,Basque ,Basque (family) ,Belarusian ,Bemba (Zambia) ,Bengali, Bangla ,Berber languages ,Bihari ,Bislama ,Bosnian ,Brazilian Sign Language ,Breton ,Bulgarian ,Catalan ,Caucasian languages ,Cebuano ,Celtic languages ,Central Bikol ,Chichewa, Chewa, Nyanja ,Chilean Sign Language ,Chinese ,Chuukese ,Colombian Sign Language ,Congo Swahili ,Croatian ,Cushitic languages ,Czech ,Danish ,Dholuo, Luo (Kenya and Tanzania) ,Dravidian languages ,Dutch ,East Slavic languages ,Eastern Malayo-Polynesian languages ,Efik ,Esperanto ,Estonian ,Ewe ,Fijian ,Finnish ,Finnish Sign Language ,Finno-Ugrian languages ,French ,French-based creoles and pidgins ,Ga ,Galician ,Ganda ,Georgian ,German ,Germanic languages ,Gilbertese ,Greek (modern) ,Greek languages ,Gujarati ,Gun ,Haitian, Haitian Creole ,Hausa ,Hebrew (modern) ,Hiligaynon ,Hindi ,Hiri Motu ,Hungarian ,Icelandic ,Igbo ,Iloko ,Indic languages ,Indo-European languages ,Indo-Iranian languages ,Indonesian ,Irish ,Isoko ,Isthmus Zapotec ,Italian ,Italic languages ,Japanese ,Japanese ,Kabyle ,Kalaallisut, Greenlandic ,Kannada ,Kaonde ,Kinyarwanda ,Kirundi ,Kongo ,Korean ,Kwangali ,Kwanyama, Kuanyama ,Latin ,Latvian ,Lingala ,Lithuanian ,Louisiana Creole ,Lozi ,Luba-Katanga ,Luba-Lulua ,Lunda ,Lushai ,Luvale ,Macedonian ,Malagasy ,Malay ,Malayalam ,Malayo-Polynesian languages ,Maltese ,Manx ,Marathi (MarāṭhΔ«) ,Marshallese ,Mexican Sign Language ,Mon-Khmer languages ,Morisyen ,Mossi ,Multiple languages ,Ndonga ,Nepali ,Niger-Kordofanian languages ,Nigerian Pidgin ,Niuean ,North Germanic languages ,Northern Sotho, Pedi, Sepedi ,Norwegian ,Norwegian BokmΓ₯l ,Norwegian Nynorsk ,Nyaneka ,Oromo ,Pangasinan ,Papiamento ,Persian (Farsi) ,Peruvian Sign Language ,Philippine languages ,Pijin ,Pohnpeian ,Polish ,Portuguese ,Portuguese-based creoles and pidgins ,Punjabi (Eastern) ,Romance languages ,Romanian ,Rundi ,Russian ,Ruund ,Salishan languages ,Samoan ,San Salvador Kongo ,Sango ,Semitic languages ,Serbo-Croatian ,Seselwa Creole French ,Shona ,Sindhi ,Sino-Tibetan languages ,Slavic languages ,Slovak ,Slovene ,Somali ,South Caucasian languages ,South Slavic languages ,Southern Sotho ,Spanish ,Spanish Sign Language ,Sranan Tongo ,Swahili ,Swati ,Swedish ,Tagalog ,Tahitian ,Tai ,Tamil ,Telugu ,Tetela ,Tetun Dili ,Thai ,Tigrinya ,Tiv ,Tok Pisin ,Tonga (Tonga Islands) ,Tonga (Zambia) ,Tsonga ,Tswana ,Tumbuka ,Turkic languages ,Turkish ,Tuvalu ,Tzotzil ,Ukrainian ,Umbundu ,Uralic languages ,Urdu ,Venda ,Venezuelan Sign Language ,Vietnamese ,Wallisian ,Walloon ,Waray (Philippines) ,Welsh ,West Germanic languages ,West Slavic languages ,Western Malayo-Polynesian languages ,Wolaitta, Wolaytta ,Wolof ,Xhosa ,Yapese ,Yiddish ,Yoruba ,Yucatec Maya, Yucateco ,Zande (individual language) ,Zulu

The complete list of all 11700+ models & pipelines in 230+ languages is available on Models Hub

:notebook: New Notebooks

Spark NLP Notebooks Colab
CamemBertForTokenClassification HuggingFace in Spark NLP - CamemBertForSequenceClassification Open In Colab

:notebook: Updated Notebooks

The following notebooks have been updated to use the last release of TensorFLow 2.11 and Hugging Face 4.25 libraries

Spark NLP Notebooks Colab
BertEmbeddings HuggingFace in Spark NLP - BERT Open In Colab
BertSentenceEmbeddings HuggingFace in Spark NLP - BERT Sentence Open In Colab
DistilBertEmbeddings HuggingFace in Spark NLP - DistilBERT Open In Colab
CamemBertEmbeddings HuggingFace in Spark NLP - CamemBERT Open In Colab
RoBertaEmbeddings HuggingFace in Spark NLP - RoBERTa Open In Colab
DeBertaEmbeddings HuggingFace in Spark NLP - DeBERTa Open In Colab
XlmRoBertaEmbeddings HuggingFace in Spark NLP - XLM-RoBERTa Open In Colab
AlbertEmbeddings HuggingFace in Spark NLP - ALBERT Open In Colab
BertForTokenClassification HuggingFace in Spark NLP - BertForTokenClassification Open In Colab
DistilBertForTokenClassification HuggingFace in Spark NLP - DistilBertForTokenClassification Open In Colab
AlbertForTokenClassification HuggingFace in Spark NLP - AlbertForTokenClassification Open In Colab
RoBertaForTokenClassification HuggingFace in Spark NLP - RoBertaForTokenClassification Open In Colab
XlmRoBertaForTokenClassification HuggingFace in Spark NLP - XlmRoBertaForTokenClassification Open In Colab
CamemBertForTokenClassification HuggingFace in Spark NLP - CamemBertForTokenClassification Open In Colab
CamemBertForTokenClassification HuggingFace in Spark NLP - CamemBertForSequenceClassification Open In Colab
BertForSequenceClassification HuggingFace in Spark NLP - BertForSequenceClassification Open In Colab
DistilBertForSequenceClassification HuggingFace in Spark NLP - DistilBertForSequenceClassification Open In Colab
AlbertForSequenceClassification HuggingFace in Spark NLP - AlbertForSequenceClassification Open In Colab
RoBertaForSequenceClassification HuggingFace in Spark NLP - RoBertaForSequenceClassification Open In Colab
XlmRoBertaForSequenceClassification HuggingFace in Spark NLP - XlmRoBertaForSequenceClassification Open In Colab
AlbertForQuestionAnswering HuggingFace in Spark NLP - AlbertForQuestionAnswering Open In Colab
BertForQuestionAnswering HuggingFace in Spark NLP - BertForQuestionAnswering Open In Colab
DeBertaForQuestionAnswering HuggingFace in Spark NLP - DeBertaForQuestionAnswering Open In Colab
DistilBertForQuestionAnswering HuggingFace in Spark NLP - DistilBertForQuestionAnswering Open In Colab
RoBertaForQuestionAnswering HuggingFace in Spark NLP - RoBertaForQuestionAnswering Open In Colab
XlmRobertaForQuestionAnswering HuggingFace in Spark NLP - XlmRobertaForQuestionAnswering Open In Colab

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.5

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.5

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.5

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.5

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.5

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.5

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.5</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.5</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.5</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.5</version>
</dependency>

FAT JARs

What's Changed

Contributors

@Damla-Gurbaz @Cabir40 @josejuanmartinez @danilojsl @mhnavid @DevinTDHa @jsl-builder @KshitizGIT @suvrat-joshi @maziyarpanahi @agsfer

New Contributors

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.4...4.2.5

4.2.4

1 year ago

:loudspeaker: Overview

Spark NLP 4.2.4 πŸš€ comes with new support for GCP storage to automatically download and load models & pipelines via setting the cache_pretrained path, update to TensorFlow 2.7.4 with security patch fixes, lots of improvements in our documentation, improvements, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: New Features & improvements

  • Introducing support for GCP storage to automatically download and load pre-trained models/pipelines from cache_pretrained directory
  • Update to TensorFlow 2.7.4 with bug and CVEs fixes. Details about bugs and CVEs fixes: https://github.com/JohnSnowLabs/spark-nlp/commit/417e2a1ff2b0bca2d2046c4d4740f52ce770689f
  • Improve error handling while importing external TensorFlow models into Spark NLP
  • Improve error messages when importing external models from remote storages like DBFS, S3, and HDFS
  • Update documentation on how to use testDataset param in NerDLApproach, ClassifierDLApproach, MultiClassifierDLApproach, and SentimentDLApproach
  • Update installation instructions for the Apple M1 chip
  • Add support for future decoder-encoder models with 2 separated models

πŸ› Bug Fixes

  • Add missing setPreservePosition in NerConverter
  • Add missing inputAnnotatorTypes to BigTextMatcher, ViveknSentimentModel, and NerConverter annotators
  • Fix all wrong example codes provided for LemmatizerModel in Models Hub
  • Fix the t5_grammar_error_corrector model to be compatible with Spark NLP 4.0+
  • Fix provided notebook to import Longformer models from Hugging Face into Spark NLP

:notebook: New Notebooks

Spark NLP Notebooks Colab
Spark NLP Conf Dowbload and Load Model from GCP Storage Open In Colab
LongformerEmbeddings HuggingFace in Spark NLP - Longformer Open In Colab

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.4

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

FAT JARs

What's Changed

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.3...4.2.4

4.2.1

1 year ago

:loudspeaker: Overview

Spark NLP 4.2.1 πŸš€ comes with a new multi-lingual support for Word Segmentation mostly used for (but not limited to) Chinese, Japanese, Korean, and so on, adding Automatic Speech Recognition (ASR) pipelines to LightPipeline arsenal for faster computation of smaller datasets without Apache Spark (e.g. RESTful API use case), adding support for processed audio files in type of Double in addition to Float for Wav2Vec2, over 230+ state-of-the-art Transformer Vision (ViT) pretrained pipelines for 1-line Image Classification, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

:star: New Features & improvements

Bug Fixes

:notebook: New Notebooks

Spark NLP Notebooks Colab
SpanBertCorefModel Coreference Resolution with SpanBertCorefModel Open In Colab
WordSegmenter Train and inference multi-lingual Word Segmenter Open In Colab

Models

Spark NLP 4.2.1 comes with 230+ state-of-the-art pre-trained Transformer Vision (ViT) pipeline:

Pipeline Name Lang
PretrainedPipeline pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat en
PretrainedPipeline pipeline_image_classifier_vit_base_beans_demo_v5 en
PretrainedPipeline pipeline_image_classifier_vit_animal_classifier_huggingface en
PretrainedPipeline pipeline_image_classifier_vit_Infrastructures en
PretrainedPipeline pipeline_image_classifier_vit_blocks en
PretrainedPipeline pipeline_image_classifier_vit_beer_whisky_wine_detection en
PretrainedPipeline pipeline_image_classifier_vit_base_xray_pneumonia en
PretrainedPipeline pipeline_image_classifier_vit_baseball_stadium_foods en
PretrainedPipeline pipeline_image_classifier_vit_dog_vs_chicken en

Check 460+ Transformer Vision (ViT) models & pipelines for Models Hub - Image Classification

Spark NLP covers the following languages:

English ,Multilingual ,Afrikaans ,Afro-Asiatic languages ,Albanian ,Altaic languages ,American Sign Language ,Amharic ,Arabic ,Argentine Sign Language ,Armenian ,Artificial languages ,Atlantic-Congo languages ,Austro-Asiatic languages ,Austronesian languages ,Azerbaijani ,Baltic languages ,Bantu languages ,Basque ,Basque (family) ,Belarusian ,Bemba (Zambia) ,Bengali, Bangla ,Berber languages ,Bihari ,Bislama ,Bosnian ,Brazilian Sign Language ,Breton ,Bulgarian ,Catalan ,Caucasian languages ,Cebuano ,Celtic languages ,Central Bikol ,Chichewa, Chewa, Nyanja ,Chilean Sign Language ,Chinese ,Chuukese ,Colombian Sign Language ,Congo Swahili ,Croatian ,Cushitic languages ,Czech ,Danish ,Dholuo, Luo (Kenya and Tanzania) ,Dravidian languages ,Dutch ,East Slavic languages ,Eastern Malayo-Polynesian languages ,Efik ,Esperanto ,Estonian ,Ewe ,Fijian ,Finnish ,Finnish Sign Language ,Finno-Ugrian languages ,French ,French-based creoles and pidgins ,Ga ,Galician ,Ganda ,Georgian ,German ,Germanic languages ,Gilbertese ,Greek (modern) ,Greek languages ,Gujarati ,Gun ,Haitian, Haitian Creole ,Hausa ,Hebrew (modern) ,Hiligaynon ,Hindi ,Hiri Motu ,Hungarian ,Icelandic ,Igbo ,Iloko ,Indic languages ,Indo-European languages ,Indo-Iranian languages ,Indonesian ,Irish ,Isoko ,Isthmus Zapotec ,Italian ,Italic languages ,Japanese ,Japanese ,Kabyle ,Kalaallisut, Greenlandic ,Kannada ,Kaonde ,Kinyarwanda ,Kirundi ,Kongo ,Korean ,Kwangali ,Kwanyama, Kuanyama ,Latin ,Latvian ,Lingala ,Lithuanian ,Louisiana Creole ,Lozi ,Luba-Katanga ,Luba-Lulua ,Lunda ,Lushai ,Luvale ,Macedonian ,Malagasy ,Malay ,Malayalam ,Malayo-Polynesian languages ,Maltese ,Manx ,Marathi (MarāṭhΔ«) ,Marshallese ,Mexican Sign Language ,Mon-Khmer languages ,Morisyen ,Mossi ,Multiple languages ,Ndonga ,Nepali ,Niger-Kordofanian languages ,Nigerian Pidgin ,Niuean ,North Germanic languages ,Northern Sotho, Pedi, Sepedi ,Norwegian ,Norwegian BokmΓ₯l ,Norwegian Nynorsk ,Nyaneka ,Oromo ,Pangasinan ,Papiamento ,Persian (Farsi) ,Peruvian Sign Language ,Philippine languages ,Pijin ,Pohnpeian ,Polish ,Portuguese ,Portuguese-based creoles and pidgins ,Punjabi (Eastern) ,Romance languages ,Romanian ,Rundi ,Russian ,Ruund ,Salishan languages ,Samoan ,San Salvador Kongo ,Sango ,Semitic languages ,Serbo-Croatian ,Seselwa Creole French ,Shona ,Sindhi ,Sino-Tibetan languages ,Slavic languages ,Slovak ,Slovene ,Somali ,South Caucasian languages ,South Slavic languages ,Southern Sotho ,Spanish ,Spanish Sign Language ,Sranan Tongo ,Swahili ,Swati ,Swedish ,Tagalog ,Tahitian ,Tai ,Tamil ,Telugu ,Tetela ,Tetun Dili ,Thai ,Tigrinya ,Tiv ,Tok Pisin ,Tonga (Tonga Islands) ,Tonga (Zambia) ,Tsonga ,Tswana ,Tumbuka ,Turkic languages ,Turkish ,Tuvalu ,Tzotzil ,Ukrainian ,Umbundu ,Uralic languages ,Urdu ,Venda ,Venezuelan Sign Language ,Vietnamese ,Wallisian ,Walloon ,Waray (Philippines) ,Welsh ,West Germanic languages ,West Slavic languages ,Western Malayo-Polynesian languages ,Wolaitta, Wolaytta ,Wolof ,Xhosa ,Yapese ,Yiddish ,Yoruba ,Yucatec Maya, Yucateco ,Zande (individual language) ,Zulu

The complete list of all 11000+ models & pipelines in 230+ languages is available on Models Hub

:book: Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.1

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.1

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.1

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.1

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.1

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.1</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.1</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.1</version>
</dependency>

FAT JARs

What's Changed

Contributors

@Meryem1425 @muhammetsnts @jsl-models @josejuanmartinez @DevinTDHa @ArshaanNazir @C-K-Loan @KshitizGIT @agsfer @diatrambitas @danilojsl @Damla-Gurbaz @maziyarpanahi @jsl-builder

New Contributors

Full Changelog: https://github.com/JohnSnowLabs/spark-nlp/compare/4.2.0...4.2.1