Rubrix Versions Save

Argilla is a collaboration platform for AI engineers and domain experts that require high-quality outputs, full data ownership, and overall efficiency.

v1.21.0

5 months ago

🔆 Release highlights

Draft queue

We’ve added a new queue in the Feedback Task UI so that you can save your drafts and have them all together in a separate view. This allows you to save your responses and come back to them before submission.

Note that responses won’t be autosaved now and to save your changes you will need to click on “Save as draft” or use the shortcut command ⌘ + S (macOS), Ctrl + S (other).

Improved shortcuts

We’ve been working to improve the keyboard shortcuts within the Feedback Task UI to make them more productive and user-friendly.

You can now select labels in Label and Multi-label questions using the numerical keys in your keyboard. To know which number corresponds with each label you can simply show or hide helpers by pressing command ⌘ (MacOS) or Ctrl (other) for 2 seconds. You will then see the numbers next to the corresponding labels.

We’ve also simplified shortcuts for navigation and actions, so that they use as few keys as possible.

Check all available shortcuts here.

New `metrics` module

We've added a new module to analyze the annotations, both in terms of agreement between the annotators and in terms of data and model drift monitoring.

Agreement metrics

Easily measure the inter-annotator agreement to explore the quality of the annotation guidelines and consistency between annotators:

import argilla as rg
from argilla.client.feedback.metrics import AgreementMetric
feedback_dataset = rg.FeedbackDataset.from_argilla("...", workspace="...")
metric = AgreementMetric(dataset=feedback_dataset, question_name="question_name")
agreement_metrics = metric.compute("alpha")
#>>> agreement_metrics
#[AgreementMetricResult(metric_name='alpha', count=1000, result=0.467889)]

Model metrics

You can use ModelMetric to model monitor performance for data and model drift:

import argilla as rg
from argilla.client.feedback.metrics import ModelMetric
feedback_dataset = rg.FeedbackDataset.from_argilla("...", workspace="...")
metric = ModelMetric(dataset=feedback_dataset, question_name="question_name")
annotator_metrics = metric.compute("accuracy")
#>>> annotator_metrics
#{'00000000-0000-0000-0000-000000000001': [ModelMetricResult(metric_name='accuracy', count=3, result=0.5)], '00000000-0000-0000-0000-000000000002': [ModelMetricResult(metric_name='accuracy', count=3, result=0.25)], '00000000-0000-0000-0000-000000000003': [ModelMetricResult(metric_name='accuracy', count=3, result=0.5)]}

List aggregation support for `TermsMetadataProperty`

You can now pass a list of terms within a record’s metadata that will be aggregated and filterable as part of a TermsMetadataProperty.

Here is an example:

import argilla as rg

dataset = rg.FeedbackDataset(
    fields = ...,
    questions = ...,
    metadata_properties = [rg.TermsMetadataProperty(name="annotators")]
)

record = rg.FeedbackRecord(
    fields = ...,
    metadata = {"annotators": ["user_1", "user_2"]}
)

Reindex from CLI

Reindex all entities in your Argilla instance (datasets, records, responses, etc.) with a simple CLI command.

argilla server reindex

This is useful when you are working with an existing feedback datasets and you want to update the search engine info.

Changelog 1.21.0

Added

Added new draft queue for annotation view (#4334)
Added annotation metrics module for the FeedbackDataset (argilla.client.feedback.metrics). (#4175).
Added strategy to handle and translate errors from the server for 401 HTTP status code` (#4362)
Added integration for textdescriptives using TextDescriptivesExtractor to configure metadata_properties in FeedbackDataset and FeedbackRecord. (#4400). Contributed by @m-newhauser
Added POST /api/v1/me/responses/bulk endpoint to create responses in bulk for current user. (#4380)
Added list support for term metadata properties. (Closes #4359)
Added new CLI task to reindex datasets and records into the search engine. (#4404)
Added httpx_extra_kwargs argument to rg.init and Argilla to allow passing extra arguments to httpx.Client used by Argilla. (#4440)

Changed

More productive and simpler shortcuts system (#4215)
Move ArgillaSingleton, init and active_client to a new module singleton. (#4347)
Updated argilla.load functions to also work with FeedbackDatasets. (#4347)
[breaking] Updated argilla.delete functions to also work with FeedbackDatasets. It now raises an error if the dataset does not exist. (#4347)
Updated argilla.list_datasets functions to also work with FeedbackDatasets. (#4347)

Fixed

Fixed error in TextClassificationSettings.from_dict method in which the label_schema created was a list of dict instead of a list of str. (#4347)
Fixed total records on pagination component (#4424)

Removed

Removed draft auto save for annotation view (#4334)

v1.20.0

6 months ago

🔆 Release highlights

Responses and suggestions filters

We’ve added new filters in the Argilla UI to filter records within Feedback datasets based on response values and suggestions information. It is also possible to sort records based on suggestion scores. This is available for questions of the type: LabelQuestion, MultiLabelQuestion and RatingQuestion.

Screenshot of the response and suggestions filters

Screenshot of the suggestion score sort

Utils module

Assign records

We added several methods to assign records to annotators via controlled overlap assign_records and assign_workspaces.

from argilla.client.feedback.utils import assign_records

assignments = assign_records(
    users=users,
    records=records,
    overlap=1,
    shuffle=True
)

from argilla.client.feedback.utils import assign_workspaces

assignments = assign_workspaces(
    assignments=assignments,
    workspace_type="individual"
)

for username, records in assignments.items():
    dataset = rg.FeedbackDataset(
        fields=fields, questions=questions, metadata=metadata,
        vector_settings=vector_settings, guidelines=guidelines
    )
    dataset.add_records(records)
    remote_dataset = dataset.push_to_argilla(name="my_dataset", workspace=username)

Argilla supports basic handling of video, audio, and images within markdown fields, provided they are formatted in HTML. To facilitate this, we offer three functions: video_to_html, audio_to_html, and image_to_html. Note that performance differs per browser and database configuration.

from argilla.client.feedback.utils import audio_to_html, image_to_html, video_to_html

# Configure the FeedbackDataset
ds_multi_modal = rg.FeedbackDataset(
    fields=[rg.TextField(name="content", use_markdown=True, required=True)],
    questions=[rg.TextQuestion(name="description", title="Describe the content of the media:", use_markdown=True, required=True)],
)
# Add the records
records = [
    rg.FeedbackRecord(fields={"content": video_to_html("/content/snapshot.mp4")}),
    rg.FeedbackRecord(fields={"content": audio_to_html("/content/sea.wav")}),
    rg.FeedbackRecord(fields={"content": image_to_html("/content/peacock.jpg")}),
]
ds_multi_modal.add_records(records)
# Push the dataset to Argilla
ds_multi_modal = ds_multi_modal.push_to_argilla("multi-modal-basic", workspace="admin")

Token Highlights

You can also add custom highlights to the text by using create_token_highlights and a custom color map.

from argilla.client.feedback.utils import create_token_highlights

tokens = ["This", "is", "a", "test"]
weights = [0.1, 0.2, 0.3, 0.4]
html = create_token_highlights(tokens, weights, c_map=custom_RGB) # 'viridis' by default

1.20.0 Changelog

Added

Added GET /api/v1/datasets/:dataset_id/records/search/suggestions/options endpoint to return suggestion available options for searching. (#4260)
Added metadata_properties to the __repr__ method of the FeedbackDataset and RemoteFeedbackDataset.(#4192).
Added get_model_kwargs, get_trainer_kwargs, get_trainer_model, get_trainer_tokenizer and get_trainer -methods to the ArgillaTrainer to improve interoperability across frameworks. (#4214).
Added additional formatting checks to the ArgillaTrainer to allow for better interoperability of defaults and formatting_func usage. (#4214).
Added a warning to the update_config-method of ArgillaTrainer to emphasize if the kwargs were updated correctly. (#4214).
Added argilla.client.feedback.utils module with html_utils (this mainly includes video/audio/image_to_html that convert media to dataURL to be able to render them in tha Argilla UI and create_token_highlights to highlight tokens in a custom way. Both work on TextQuestion and TextField with use_markdown=True) and assignments (this mainly includes assign_records to assign records according to a number of annotators and records, an overlap and the shuffle option; and assign_workspace to assign and create if needed a workspace according to the record assignment). (#4121)

Fixed

Fixed error in ArgillaTrainer, with numerical labels, using RatingQuestion instead of RankingQuestion (#4171)
Fixed error in ArgillaTrainer, now we can train for extractive_question_answering using a validation sample (#4204)
Fixed error in ArgillaTrainer, when training for sentence-similarity it didn't work with a list of values per record (#4211)
Fixed error in the unification strategy for RankingQuestion (#4295)
Fixed TextClassificationSettings.labels_schema order was not being preserved. Closes #3828 (#4332)
Fixed error when requesting non-existing API endpoints. Closes #4073 (#4325)
Fixed error when passing draft responses to create records endpoint. (#4354)

Changed

[breaking] Suggestions agent field only accepts now some specific characters and a limited length. (#4265)
[breaking] Suggestions score field only accepts now float values in the range 0 to 1. (#4266)
Updated POST /api/v1/dataset/:dataset_id/records/search endpoint to support optional query attribute. (#4327)
Updated POST /api/v1/dataset/:dataset_id/records/search endpoint to support filter and sort attributes. (#4327)
Updated POST /api/v1/me/datasets/:dataset_id/records/search endpoint to support optional query attribute. (#4270)
Updated POST /api/v1/me/datasets/:dataset_id/records/search endpoint to support filter and sort attributes. (#4270)
Changed the logging style while pulling and pushing FeedbackDataset to Argilla from tqdm style to rich. (#4267). Contributed by @zucchini-nlp.
Updated push_to_argilla to print repr of the pushed RemoteFeedbackDataset after push and changed show_progress to True by default. (#4223)
Changed models and tokenizer for the ArgillaTrainer to explicitly allow for changing them when needed. (#4214).

v1.19.0

7 months ago

🔆 Release highlights

🚨 Breaking changes

We have chosen to disable raining a ValueError during the FeedbackDataset.*_by_name(): FeedbackDataset.question_by_name(), FeedbackDataset.field_by_name() and FeedbackDataset.metadata_property_by_name. Instead, these methods will now return None when no match is found. This change is backwards compatible with previous versions of Argilla but might break your code if you are relying on the ValueError to be raised.

Add vectors to your `FeedbackDataset`

You can now add vectors to your Feedback dataset and records to enable similarity search.

To do that, first, you need to add vector settings to your dataset:

dataset = rg.FeedbackDataset(
    fields=[...],
    questions=[....],
    vector_settings=[
        rg.VectorSettings(
            name="my_vectors",
            dimensions=768,
            tite="My Vectors" #optional
        )
    ]
)

Then, you can add vectors to your records where the key matches the name of your vector settings and the value is a List[float]:

record = rg.FeedbackRecord(
    fields={...},
    vectors={"my_vectors": [...]}
)

⚠️ For vector search in OpenSearch, the filtering applied is using a post_filter step, since there is a bug that makes queries fail using filtering + KNN from Argilla. See https://github.com/opensearch-project/k-NN/issues/1286

Similarity search

If you have included vectors and vector settings in your dataset, you can use the similarity search features within that dataset.

In the Argilla UI, you can find records that are similar to each other using the Find similar button at the top right corner of the record card. Here's how to do it:

With a collapsed reference record: With an expanded reference record:

In the SDK, you can do the same like this:

ds = rg.FeedbackDataset.from_argilla("my_dataset", workspace="my_workspace")

# using another record
similar_records =  ds.find_similar_records(
    vector_name="my_vector",
    record=ds.records[0],
    max_results=5
)

# work with the resulting tuples
for record, score in similar_records:
    ...

You can also find records that are similar to a given text, but bear in mind that the dimensions of the resulting vector should be equal to that of the vector used in the dataset records:

similar_records =  ds.find_similar_records(
    vector_name="my_vector",
    value=embedder_model.embeddings("My text is here")
    # value=embedder_model.embeddings("My text is here").tolist() # for numpy arrays
)

`FeedbackDataset`

We added a show_progress argument to from_huggingface() method to make the progress bar for the parsing records process optional.

`RemoteFeedbackDataset`

We have added additional support for the pull()-method of RemoteFeedbackDataset. It is now possible to pull a RemoteFeedbackDataset with a specific max_records-argument. In combination with the earlier introduced filter_by and sorty_by this allows for more fine-grained control over the records that are pulled from Argilla.

`ArgillaTrainer`

The ArgillaTrainer class has been updated to support additional features. Hugging Face models can now be shared to the Hugging Face Hub directly from the ArgillaTrainer.push_to_huggingface-method. Additionally, we have included filter_by, sort_by, and max_records arguments to the `ArgillaTrainer '-initialisation method to allow for more fine-grained control over the records used for training.

from argilla import SortBy

trainer = ArgillaTrainer(
    dataset=dataset,
    task=task,
    framework="setfit",
    filter_by={"response_status": ["submitted"]},
    sort_by=[SortBy(field="metadata.my-metadata", order="asc")],
    max_records=1000
)

🎨 UI improvements

We have changed the layout of the filters for a slimmer and more flexible component that will host more filter types in the future without being disruptive.
We have fixed a small UI bug where larger svg-images were pushed out of the visible screen, leading to a bad user experience.
There is sorting support based on inserted_at and updated_at datetime fields.

1.19.0 Changelog

Added

Added POST /api/v1/datasets/:dataset_id/records/search endpoint to search for records without user context, including responses by all users. (#4143)
Added POST /api/v1/datasets/:dataset_id/vectors-settings endpoint for creating vector settings for a dataset. (#3776)
Added GET /api/v1/datasets/:dataset_id/vectors-settings endpoint for listing the vectors settings for a dataset. (#3776)
Added DELETE /api/v1/vectors-settings/:vector_settings_id endpoint for deleting a vector settings. (#3776)
Added PATCH /api/v1/vectors-settings/:vector_settings_id endpoint for updating a vector settings. (#4092)
Added GET /api/v1/records/:record_id endpoint to get a specific record. (#4039)
Added support to include vectors for GET /api/v1/datasets/:dataset_id/records endpoint response using include query param. (#4063)
Added support to include vectors for GET /api/v1/me/datasets/:dataset_id/records endpoint response using include query param. (#4063)
Added support to include vectors for POST /api/v1/me/datasets/:dataset_id/records/search endpoint response using include query param. (#4063)
Added show_progress argument to from_huggingface() method to make the progress bar for parsing records process optional.(#4132).
Added a progress bar for parsing records process to from_huggingface() method with trange in tqdm.(#4132).
Added to sort by inserted_at or updated_at for datasets with no metadata. (4147)
Added max_records argument to pull() method for RemoteFeedbackDataset.(#4074)
Added functionality to push your models to the Hugging Face hub with ArgillaTrainer.push_to_huggingface (#3976). Contributed by @Racso-3141.
Added filter_by argument to ArgillaTrainer to filter by response_status (#4120).
Added sort_by argument to ArgillaTrainer to sort by metadata (#4120).
Added max_records argument to ArgillaTrainer to limit record used for training (#4120).
Added add_vector_settings method to local and remote FeedbackDataset. (#4055)
Added update_vectors_settings method to local and remote FeedbackDataset. (#4122)
Added delete_vectors_settings method to local and remote FeedbackDataset. (#4130)
Added vector_settings_by_name method to local and remote FeedbackDataset. (#4055)
Added find_similar_records method to local and remote FeedbackDataset. (#4023)
Added ARGILLA_SEARCH_ENGINE environment variable to configure the search engine to use. (#4019)

Changed

[breaking] Remove support for Elasticsearch < 8.5 and OpenSearch < 2.4. (#4173)
[breaking] Users working with OpenSearch engines must use version >=2.4 and set ARGILLA_SEARCH_ENGINE=opensearch. (#4019 and #4111)
[breaking] Changed FeedbackDataset.*_by_name() methods to return None when no match is found (#4101).
[breaking] limit query parameter for GET /api/v1/datasets/:dataset_id/records endpoint is now only accepting values greater or equal than 1 and less or equal than 1000. (#4143)
[breaking] limit query parameter for GET /api/v1/me/datasets/:dataset_id/records endpoint is now only accepting values greater or equal than 1 and less or equal than 1000. (#4143)
Update GET /api/v1/datasets/:dataset_id/records endpoint to fetch record using the search engine. (#4142)
Update GET /api/v1/me/datasets/:dataset_id/records endpoint to fetch record using the search engine. (#4142)
Update POST /api/v1/datasets/:dataset_id/records endpoint to allow to create records with vectors (#4022)
Update PATCH /api/v1/datasets/:dataset_id endpoint to allow updating allow_extra_metadata attribute. (#4112)
Update PATCH /api/v1/datasets/:dataset_id/records endpoint to allow to update records with vectors. (#4062)
Update PATCH /api/v1/records/:record_id endpoint to allow to update record with vectors. (#4062)
Update POST /api/v1/me/datasets/:dataset_id/records/search endpoint to allow to search records with vectors. (#4019)
Update BaseElasticAndOpenSearchEngine.index_records method to also index record vectors. (#4062)
Update FeedbackDataset.__init__ to allow passing a list of vector settings. (#4055)
Update FeedbackDataset.push_to_argilla to also push vector settings. (#4055)
Update FeedbackDatasetRecord to support the creation of records with vectors. (#4043)
Using cosine similarity to compute similarity between vectors. (#4124)

Fixed

Fixed svg images out of screen with too large images (#4047)
Fixed creating records with responses from multiple users. Closes #3746 and #3808 (#4142)
Fixed deleting or updating responses as an owner for annotators. (Commit 403a66d)
Fixed passing user_id when getting records by id. (Commit 98c7927)
Fixed non-basic tags serialized when pushing a dataset to the Hugging Face Hub. Closes #4089 (#4200)

Contributors

@Racso-3141 Added a progress bar for parsing records process to from_huggingface() method with trange in tqdm.(#4132).

v1.18.0

7 months ago

🔆 Release highlights

💾 Add metadata properties to Feedback Datasets

You can now filter and sort records in Feedback Datasets in the UI and Python SDK using the metadata included in the records. To do that, you will first need to set up a MetadataProperty in your dataset:

# set up a dataset including metadata properties
dataset = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="prompt"),
        rg.TextField(name="response"),
    ],
    questions=[
        rg.TextQuestion(name="question")
    ],
    metadata_properties=[
        rg.TermsMetadataProperty(name="source"),
        rg.IntegerMetadataProperty(name="response_length", title="Response length")
    ]
)

Learn more about how to define metadata properties or adding or deleting metadata properties in existing datasets.

This will read the metadata in the records that match the name of the metadata property. Any other metadata present in the record not matching a metadata property will be saved but not available to use in the filtering and sorting features in the UI or SDK.

# create a record with metadata
record = rg.FeedbackRecord(
    fields={
        "prompt": "Why can camels survive long without water?",
        "response": "Camels use the fat in their humps to keep them filled with energy and hydration for long periods of time."
    },
    metadata={"source": "wikipedia", "response_length": 105, "my_hidden_metadata": "hidden metadata"}
)

Learn more about how to create records with metadata and how to add, modify or delete metadata from existing records.

🗃️ Filter and sort records using metadata in Feedback Datasets

In the Python SDK, you can filter and sort records based on the Metadata Properties that you set up for your dataset. You can combine multiple filters and sorts. Here is an example of how you could use them:

filtered_records = remote.filter_by(
    metadata_filters=[
        rg.IntegerMetadataFilter(
            name="response_length",
            ge=500, # optional: greater or equal to
            le=1000 # optional: lower or equal to
        ),
        rg.TermsMetadataFilter(
            name="source", 
            values=["wikipedia", "wikihow"]
        )
    ]
).sort_by(
    [
        rg.SortBy(
            field="response_length",
            order="desc" # for descending or "asc" for ascending
        )
    ]

In the UI, simply use the Metadata and Sort components to filter and sort records like this:

https://github.com/argilla-io/argilla/assets/126158523/6a5a7984-425d-4f1a-b0f7-7cc2bb7e4a0a

Read more about filtering and sorting in Feedback Datasets.

⚠️ Breaking change using SQLite as backend in a docker deployment

From version 1.17.0 a new argilla os user is configured for the provided docker images. If you are using the docker deployment and you want to upload to this version from versions older than v1.17.0 (If you already updated from v1.17.0 this step was already applied - see Release Notes), you should change permissions to the SQLite db file, before upgrading the version. You can do it with the following action:

docker exec --user root <argilla_server_container_id> /bin/bash -c 'chmod -R 777 "$ARGILLA_HOME_PATH"'

Note: You can find the docker container id by running:

docker ps  | grep -i argilla-server

713973693fb7   argilla/argilla-server:v1.16.0                "/bin/bash start_arg…"   11 hours ago   Up 7 minutes       0.0.0.0:6900->6900/tcp                           docker-argilla-1

Once the version is upgraded, we recommend to provided proper security access to this folder by setting the user and group to the new argilla user:

docker exec --user root <argilla_server_container_id>  /bin/bash -c 'chown -R argilla:argilla "$ARGILLA_HOME_PATH"'

1.18.0 Changelog

Added

New GET /api/v1/datasets/:dataset_id/metadata-properties endpoint for listing dataset metadata properties. (#3813)
New POST /api/v1/datasets/:dataset_id/metadata-properties endpoint for creating dataset metadata properties. (#3813)
New PATCH /api/v1/metadata-properties/:metadata_property_id endpoint allowing the update of a specific metadata property. (#3952)
New DELETE /api/v1/metadata-properties/:metadata_property_id endpoint for deletion of a specific metadata property. (#3911)
New GET /api/v1/metadata-properties/:metadata_property_id/metrics endpoint to compute metrics for a specific metadata property. (#3856)
New PATCH /api/v1/records/:record_id endpoint to update a record. (#3920)
New PATCH /api/v1/dataset/:dataset_id/records endpoint to bulk update the records of a dataset. (#3934)
Missing validations to PATCH /api/v1/questions/:question_id. Now title and description are using the same validations used to create questions. (#3967)
Added TermsMetadataProperty, IntegerMetadataProperty and FloatMetadataProperty classes allowing to define metadata properties for a FeedbackDataset. (#3818)
Added metadata_filters to filter_by method in RemoteFeedbackDataset to filter based on metadata i.e. TermsMetadataFilter, IntegerMetadataFilter, and FloatMetadataFilter. (#3834)
Added a validation layer for both metadata_properties and metadata_filters in their schemas and as part of the add_records and filter_by methods, respectively. (#3860)
Added sort_by query parameter to listing records endpoints that allows to sort the records by inserted_at, updated_at or metadata property. (#3843)
Added add_metadata_property method to both FeedbackDataset and RemoteFeedbackDataset (i.e. FeedbackDataset in Argilla). (#3900)
Added fields inserted_at and updated_at in RemoteResponseSchema. (#3822)
Added support for sort_by for RemoteFeedbackDataset i.e. a FeedbackDataset uploaded to Argilla. (#3925)
Added metadata_properties support for both push_to_huggingface and from_huggingface. (#3947)
Add support for update records (metadata) from Python SDK. (#3946)
Added delete_metadata_properties method to delete metadata properties. (#3932)
Added update_metadata_properties method to update metadata_properties. (#3961)
Added automatic model card generation through ArgillaTrainer.save (#3857)
Added FeedbackDataset TaskTemplateMixin for pre-defined task templates. (#3969)
A maximum limit of 50 on the number of options a ranking question can accept. (#3975)
New last_activity_at field to FeedbackDataset exposing when the last activity for the associated dataset occurs. (#3992)

Changed

GET /api/v1/datasets/{dataset_id}/records, GET /api/v1/me/datasets/{dataset_id}/records and POST /api/v1/me/datasets/{dataset_id}/records/search endpoints to return the total number of records. (#3848, #3903)
Implemented __len__ method for filtered datasets to return the number of records matching the provided filters. (#3916)
Increase the default max result window for Elasticsearch created for Feedback datasets. (#3929)
Force elastic index refresh after records creation. (#3929)
Validate metadata fields for filtering and sorting in the Python SDK. (#3993)
Using metadata property name instead of id for indexing data in search engine index. (#3994)

Fixed

Fixed response schemas to allow values to be None i.e. when a record is discarded the response.values are set to None. (#3926)
New Contributors

@splevine made their first contribution in https://github.com/argilla-io/argilla/pull/3832

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.17.0...v1.18.0

v1.17.0

7 months ago

☀️ Highlights

This release comes with a lot of new goodies and quality improvements. We added model card support for the ArgillaTrainer, worked on the FeedbackDataset task templates and added timestamps to responses. We also fixed a lot of bugs and improved the overall quality of the codebase. Enjoy!

🚨 Breaking change in updating existing Hugging Face Spaces deployments

The quickstart image startup script was changed from from /start_quickstart.sh to /home/argilla/start_quickstart.sh, which might cause existing Hugging Face Spaces deployments to malfunction. A fix was added for the Argilla template space via this PR. Alternatively, you can just create a new deployment.

⚠️ Breaking change using SQLite as backend in a docker deployment

From version 1.17.0 a new argilla os user is configured for the provided docker images. If you are using the docker deployment and you want to upload to this version, you should do some actions once update your container and before working with Argilla. Execute the following command:

docker exec --user root <argilla_server_container_id> /bin/bash -c 'chown -R argilla:argilla "$ARGILLA_HOME_PATH"'

This will change the permissions on the argilla home path, which allows it to work with new containers.

Note: You can find the docker container id by running:

docker ps  | grep -i argilla-server

713973693fb7   argilla/argilla-server:v1.17.0                "/bin/bash start_arg…"   11 hours ago   Up 7 minutes       0.0.0.0:6900->6900/tcp                           docker-argilla-1

💾 `ArgillaTrainer` Model Card Generation

The ArgillaTrainer now supports automatic model card generation. This means that you can now generate a model card with all the required info for Hugging Face and directly share these models to the hub, as you would expect within the Hugging Face ecosystem. See the docs for more info.

model_card_kwargs = {
    "language": ["en", "es"],
    "license": "Apache-2.0",
    "model_id": "all-MiniLM-L6-v2",
    "dataset_name": "argilla/emotion",
    "tags": ["nlp", "few-shot-learning", "argilla", "setfit"],
    "model_summary": "Small summary of what the model does",
    "model_description": "An extended explanation of the model",
    "model_type": "A 1.3B parameter embedding model fine-tuned on an awesome dataset",
    "finetuned_from": "all-MiniLM-L6-v2",
    "repo": "https://github.com/..."
    "developers": "",
    "shared_by": "",
}

trainer = ArgillaTrainer(
    dataset=dataset,
    task=task,
    framework="setfit",
    framework_kwargs={"model_card_kwargs": model_card_kwargs}
)
trainer.train(output_dir="my_model")
# or get the card as `str` by calling the `generate_model_card` method
argilla_model_card = trainer.generate_model_card("my_model")

🦮 `FeedbackDataset` Task Templates

The Argilla FeedbackDataset now supports a number of task templates that can be used to quickly create a dataset for specific tasks out of the box. This should help starting users get right into the action without having to worry about the dataset structure. We support basic tasks like Text Classification but also allow you to setup complex RAG-pipelines. See the docs for more info.

import argilla as rg

ds = rg.FeedbackDataset.for_text_classification(
    labels=["positive", "negative"],
    multi_label=False,
    use_markdown=True,
    guidelines=None,
)
ds
# FeedbackDataset(
#   fields=[TextField(name="text", use_markdown=True)],
#   questions=[LabelQuestion(name="label", labels=["positive", "negative"])]
#   guidelines="<Guidelines for the task>",
# )

⏱️ `inserted_at` and `updated_at` are added to responses

What are responses without timestamps? The RemoteResponseSchema now supports inserted_at and updated_at fields. This should help you to keep track of the time when a response was created and updated. Perfectly, for keeping track of annotator performance within your company.

1.17.0

Added

Added fields inserted_at and updated_at in RemoteResponseSchema (#3822).
Added automatic model card generation through ArgillaTrainer.save (#3857).
Added task templates to the FeedbackDataset (#3973).

Changed

Updated Dockerfile to use multi stage build (#3221 and #3793).
Updated active learning for text classification notebooks to use the most recent small-text version (#3831).
Changed argilla dataset name in the active learning for text classification notebooks to be consistent with the default names in the huggingface spaces (#3831).
FeedbackDataset API methods have been aligned to be accessible through the several implementations (#3937).
The unify_responses support for remote datasets (#3937).

Fixed

Fix field not shown in the order defined in the dataset settings. Closes #3959 (#3984)
Updated active learning for text classification notebooks to pass ids of type int to TextClassificationRecord (#3831).
Fixed record fields validation that was preventing from logging records with optional fields (i.e. required=True) when the field value was None (#3846).
Always set pretrained_model_name_or_path attribute as string in ArgillaTrainer (#3914).
The inserted_at and updated_at attributes are create using the utcnow factory to avoid unexpected race conditions on timestamp creation (#3945)
Fixed configure_dataset_settings when providing the workspace via the arg workspace (#3887).
Fixed saving of models trained with ArgillaTrainer with a peft_config parameter (#3795).
Fixed backwards compatibility on from_huggingface when loading a FeedbackDataset from the Hugging Face Hub that was previously dumped using another version of Argilla, starting at 1.8.0, when it was first introduced (#3829).
Fixed TrainingTaskForQuestionAnswering.__repr__ (#3969)
Fixed potential dictionary key-errors in TrainingTask.prepare_for_training_with_*-methods (#3969)

Deprecated

Function rg.configure_dataset is deprecated in favour of rg.configure_dataset_settings. The former will be removed in version 1.19.0

New Contributors

@osintalex made their first contribution in https://github.com/argilla-io/argilla/pull/3221
@kursathalat made their first contribution in https://github.com/argilla-io/argilla/pull/3756
@splevine made their first contribution in https://github.com/argilla-io/argilla/pull/3832

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.16.0...v1.17.0

v1.16.0

8 months ago

☀️ Highlights

This release comes with an auto save feature for the UI, an enhanced Argilla CLI app, new keyboard shortcuts for the annotation process in the Feedback Dataset and new integrations for the ArgillaTrainer.

💾 Auto save

Argilla UI Feedback Record getting auto saved

Have you been writing a long corrected text in a TextField for a completion given by an LLM and you have refreshed the page before submitting it? Well, since this release you are covered! The Argilla UI will save every few seconds the responses given in the annotation form of a FeedbackDataset. Annotators can partially annotate one record and then come back to finish the annotation process without losing the previous work.

👨🏻‍💻 More operations directly from the Argilla CLI

Argilla CLI displaying help information

The Argilla CLI has been updated to include an extensive list of new commands, from users and datasets management to training models all from the terminal!

⌨️ New keyboard shorcuts for the Feedback Dataset

Feedback dataset shortcuts

Now, you can seamlessly navigate within the feedback form using just your keyboard. We've extended the functionality of these shortcuts to cover all types of available questions: Label, Multi-label, Ranking, Rating and Text

QnA, Chat Completion with OpenAI and Sentence Transformers model training now in the `ArgillaTrainer`

The ArgillaTrainer doesn't stop getting new features and improvements!

A new TrainingTask has been added for Question and Answering (QnA)
Use a FeedbackDataset for fine-tuning an OpenAI model for Chat Completion
New integration with Sentence Transformers for fine-tuning a model for embedding generation

1.16.0

Added

Added ArgillaTrainer integration with sentence-transformers, allowing fine tuning for sentence similarity (#3739)
Added ArgillaTrainer integration with TrainingTask.for_question_answering (#3740)
Added Auto save record to save automatically the current record that you are working on (#3541)
Added ArgillaTrainer integration with OpenAI, allowing fine tuning for chat completion (#3615)
Added workspaces list command to list Argilla workspaces (#3594).
Added datasets list command to list Argilla datasets (#3658).
Added users create command to create users (#3667).
Added whoami command to get current user (#3673).
Added users delete command to delete users (#3671).
Added users list command to list users (#3688).
Added workspaces delete-user command to remove a user from a workspace (#3699).
Added datasets list command to list Argilla datasets (#3658).
Added users create command to create users (#3667).
Added users delete command to delete users (#3671).
Added workspaces create command to create an Argilla workspace (#3676).
Added datasets push-to-hub command to push a FeedbackDataset from Argilla into the HuggingFace Hub (#3685).
Added info command to get info about the used Argilla client and server (#3707).
Added datasets delete command to delete a FeedbackDataset from Argilla (#3703).
Added created_at and updated_at properties to RemoteFeedbackDataset and FilteredRemoteFeedbackDataset (#3709).
Added handling PermissionError when executing a command with a logged in user with not enough permissions (#3717).
Added workspaces add-user command to add a user to workspace (#3712).
Added workspace_id param to GET /api/v1/me/datasets endpoint (#3727).
Added workspace_id arg to list_datasets in the Python SDK (#3727).
Added argilla script that allows to execute Argilla CLI using the argilla command (#3730).
Added server_info function to check the Argilla server information (also accessible via rg.server_info) (#3772).

Changed

Move database commands under server group of commands (#3710)
server commands only included in the CLI app when server extra requirements are installed (#3710).
Updated PUT /api/v1/responses/{response_id} to replace values stored with received values in request (#3711).
Display a UserWarning when the user_id in Workspace.add_user and Workspace.delete_user is the ID of an user with the owner role as they don't require explicit permissions (#3716).
Rename tasks sub-package to cli (#3723).
Changed argilla database command in the CLI to now be accessed via argilla server database, to be deprecated in the upcoming release (#3754).
Changed visible_options (of label and multi label selection questions) validation in the backend to check that the provided value is greater or equal than/to 3 and less or equal than/to the number of provided options (#3773).

Fixed

Fixed remove user modification in text component on clear answers (#3775)
Fixed Highlight raw text field in dataset feedback task (#3731)
Fixed Field title too long (#3734)
Fixed error messages when deleting a DatasetForTextClassification (#3652)
Fixed Pending queue pagination problems when during data annotation (#3677)
Fixed visible_labels default value to be 20 just when visible_labels not provided and len(labels) > 20, otherwise it will either be the provided visible_labels value or None, for LabelQuestion and MultiLabelQuestion (#3702).
Fixed DatasetCard generation when RemoteFeedbackDataset contains suggestions (#3718).
Add missing draft status in ResponseSchema as now there can be responses with draft status when annotating via the UI (#3749).
Searches when queried words are distributed along the record fields (#3759).
Fixed Python 3.11 compatibility issue with /api/datasets endpoints due to the TaskType enum replacement in the endpoint URL (#3769).

As always, thanks to our amazing contributors

@sdiazlor made their first contribution in https://github.com/argilla-io/argilla/pull/3384

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.15.1...v1.16.0

v1.15.1

9 months ago

Changelog 1.15.1

Fixed

Fixed Text component text content sanitization behavior just for markdown to prevent disappear the text (#3738)
Fixed Text component now you need to press Escape to exit the text area (#3733)
Fixed SearchEngine was creating the same number of primary shards and replica shards for each FeedbackDataset (#3736).

v1.15.0

9 months ago

🔆 Highlights

Argilla 1.15.0 comes with an enhanced FeedbackDataset settings page enabling the update of the dataset settings, an integration of the TRL package with the ArgillaTrainer, and continues adding improvements to the Python client for managing FeedbackDatasets.

⚙️ Update `FeedbackDataset` settings from the UI

Update Feedback Dataset settings from the UI

FeedbackDataset settings page has been updated and now it allows to update the guidelines and some attributes of the fields and questions of the dataset. Did you misspell the title or description of a field or question? Well, you don't have to remove your dataset and create it again anymore! Just go to the settings page and fix it.

🤖 TRL integration with the `ArgillaTrainer`

ArgillaTrainer code snippet for training reward model with TRL

The famous TRL package for training Transformers with Reinforcement Learning techniques has been integrated with the ArgillaTrainer, that comes with four new TrainingTask: SFT, Reward Modeling, PPO and DPO. Each training task expects a formatting function that will return the data in the expected format for training the model.

Check this 🆕 tutorial for training a Reward Model using the Argilla Trainer.

🐍 Filter `FeedbackDataset` and remove suggestions

Using FeedbackDataset filter method

In the 1.14.0 release we added many improvements for working with remote FeedbackDatasets. In this release, a new filter_by method has been added that allows to filter the records of a dataset from the Python client. For now, the records can be only filtered using the response_status, but we're planning adding more complex filters for the upcoming releases. In addition, new methods have been added allowing to remove the suggestions created for a record.

1.15.0

Added

Added Enable to update guidelines and dataset settings for Feedback Datasets directly in the UI (#3489)
Added ArgillaTrainer integration with TRL, allowing for easy supervised finetuning, reward modeling, direct preference optimization and proximal policy optimization (#3467)
Added formatting_func to ArgillaTrainer for FeedbackDataset datasets add a custom formatting for the data (#3599).
Added login function in argilla.client.login to login into an Argilla server and store the credentials locally (#3582).
Added login command to login into an Argilla server (#3600).
Added logout command to logout from an Argilla server (#3605).
Added DELETE /api/v1/suggestions/{suggestion_id} endpoint to delete a suggestion given its ID (#3617).
Added DELETE /api/v1/records/{record_id}/suggestions endpoint to delete several suggestions linked to the same record given their IDs (#3617).
Added response_status param to GET /api/v1/datasets/{dataset_id}/records to be able to filter by response_status as previously included for GET /api/v1/me/datasets/{dataset_id}/records (#3613).
Added list classmethod to ArgillaMixin to be used as FeedbackDataset.list(), also including the workspace to list from as arg (#3619).
Added filter_by method in RemoteFeedbackDataset to filter based on response_status (#3610).
Added list_workspaces function (to be used as rg.list_workspaces, but Workspace.list is preferred) to list all the workspaces from an user in Argilla (#3641).
Added list_datasets function (to be used as rg.list_datasets) to list the TextClassification, TokenClassification, and Text2Text datasets in Argilla (#3638).
Added RemoteSuggestionSchema to manage suggestions in Argilla, including the delete method to delete suggestios from Argilla via DELETE /api/v1/suggestions/{suggestion_id} (#3651).
Added delete_suggestions to RemoteFeedbackRecord to remove suggestions from Argilla via DELETE /api/v1/records/{record_id}/suggestions (#3651).

Changed

Changed Optional label for * mark for required question (#3608)
Updated RemoteFeedbackDataset.delete_records to use batch delete records endpoint (#3580).
Included allowed_for_roles for some RemoteFeedbackDataset, RemoteFeedbackRecords, and RemoteFeedbackRecord methods that are only allowed for users with roles owner and admin (#3601).
Renamed ArgillaToFromMixin to ArgillaMixin (#3619).
Move users CLI app under database CLI app (#3593).
Move server Enum classes to argilla.server.enums module (#3620).

Fixed

Fixed Filter by workspace in breadcrumbs (#3577)
Fixed Filter by workspace in datasets table (#3604)
Fixed Query search highlight for Text2Text and TextClassification (#3621)
Fixed RatingQuestion.values validation to raise a ValidationError when values are out of range i.e. [1, 10] (#3626).

Removed

Removed multi_task_text_token_classification from TaskType as not used (#3640).
Removed argilla_id in favor of id from RemoteFeedbackDataset (#3663).
Removed fetch_records from RemoteFeedbackDataset as now the records are lazily fetched from Argilla (#3663).
Removed push_to_argilla from RemoteFeedbackDataset, as it just works when calling it through a FeedbackDataset locally, as now the updates of the remote datasets are automatically pushed to Argilla (#3663).
Removed set_suggestions in favor of update(suggestions=...) for both FeedbackRecord and RemoteFeedbackRecord, as all the updates of any "updateable" attribute of a record will go through update instead (#3663).
Remove unused owner attribute for client Dataset data model (#3665)

As always, thanks to our amazing contributors

@peppinob-ol made their first contribution in https://github.com/argilla-io/argilla/pull/3472
@eshwarhs made their first contribution in https://github.com/argilla-io/argilla/pull/3605

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.14.1...v1.15.0

v1.14.1

10 months ago

Changelog 1.14.1

Fixed

Fixed PostgreSQL database not being updated after begin_nested because of missing commit (#3567).

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.14.0...v1.14.1

v1.14.0

10 months ago

🔆 Highlights

Argilla 1.14.0 comes packed with improvements to manage Feedback Datasets from the Python client. Here are the most important changes in this version:

Code-snippet with a summary of the new workflows in Argilla

Pushing and pulling a dataset

Pushing a dataset to Argilla will now create a RemoteFeedbackDataset in Argilla. To make changes to your dataset in Argilla you will need to make those updates to the remote dataset. You can do so by either using the dataset returned when using the push_to_argilla() method (as shown in the image above) or by loading the dataset like so:

import argilla as rg
# connect to Argilla
rg.init(api_url="...", api_key="...")
# get the existing dataset in Argilla
remote_dataset = rg.FeedbackDataset.from_argilla(name="my-dataset", workspace="my-workspace")
# add a list of FeedbackRecords to the dataset in Argilla
remote_dataset.add_records(...)

Alternatively, you can make a local copy of the dataset using the pull() method.

local_dataset = remote_dataset.pull()

Note that any changes that you make to this local dataset will not affect the remote dataset in Argilla.

Adding and deleting records

How to add records to an existing dataset in Argilla was demonstrated in the first code snippet in the "Pushing and pulling a dataset" section. This is how you can delete a list of records using that same dataset:

records_to_delete = remote_dataset.records[0:5]
remote_dataset.delete_records(records_to_delete)

Or delete a single record:

record = remote_dataset.records[-1]
record.delete()

Add / update suggestions in existing records

To add and update suggestions in existing records, you can simply use the update() method. For example:

for record in remote_dataset.records:
    record.update(suggestions=...)

Note that adding a suggestion to a question that already has one will overwrite the previous suggestion. To learn more about the format that the suggestions must follow, check our docs.

Delete a dataset

You can now easily delete datasets from the Python client. To do that, get the existing dataset like demonstrated in the first section and just use:

remote_dataset.delete()

Create users with workspace assignments

Now you can create a user and directly assign existing workspaces to grant them access.

user = rg.User.create(username="...", first_name="...", password="...", workspaces=["ws1", "ws2"])

Changelog 1.14.0

Added

Added PATCH /api/v1/fields/{field_id} endpoint to update the field title and markdown settings (#3421).
Added PATCH /api/v1/datasets/{dataset_id} endpoint to update dataset name and guidelines (#3402).
Added PATCH /api/v1/questions/{question_id} endpoint to update question title, description and some settings (depending on the type of question) (#3477).
Added DELETE /api/v1/records/{record_id} endpoint to remove a record given its ID (#3337).
Added pull method in RemoteFeedbackDataset (a FeedbackDataset pushed to Argilla) to pull all the records from it and return it as a local copy as a FeedbackDataset (#3465).
Added delete method in RemoteFeedbackDataset (a FeedbackDataset pushed to Argilla) (#3512).
Added delete_records method in RemoteFeedbackDataset, and delete method in RemoteFeedbackRecord to delete records from Argilla (#3526).

Changed

Improved efficiency of weak labeling when dataset contains vectors (#3444).
Added ArgillaDatasetMixin to detach the Argilla-related functionality from the FeedbackDataset (#3427)
Moved FeedbackDataset-related pydantic.BaseModel schemas to argilla.client.feedback.schemas instead, to be better structured and more scalable and maintainable (#3427)
Update CLI to use database async connection (#3450).
Limit rating questions values to the positive range [1, 10] (#3451).
Updated POST /api/users endpoint to be able to provide a list of workspace names to which the user should be linked to (#3462).
Updated Python client User.create method to be able to provide a list of workspace names to which the user should be linked to (#3462).
Updated GET /api/v1/me/datasets/{dataset_id}/records endpoint to allow getting records matching one of the response statuses provided via query param (#3359).
Updated POST /api/v1/me/datasets/{dataset_id}/records endpoint to allow searching records matching one of the response statuses provided via query param (#3359).
Updated SearchEngine.search method to allow searching records matching one of the response statuses provided (#3359).
After calling FeedbackDataset.push_to_argilla, the methods FeedbackDataset.add_records and FeedbackRecord.set_suggestions will automatically call Argilla with no need of calling push_to_argilla explicitly (#3465).
Now calling FeedbackDataset.push_to_huggingface dumps the responses as a List[Dict[str, Any]] instead of Sequence to make it more readable via 🤗datasets (#3539).

Fixed

Fixed issue with bool values and default from Jinja2 while generating the HuggingFace DatasetCard from argilla_template.md (#3499).
Fixed DatasetConfig.from_yaml which was failing when calling FeedbackDataset.from_huggingface as the UUIDs cannot be deserialized automatically by PyYAML, so UUIDs are neither dumped nor loaded anymore (#3502).
Fixed an issue that didn't allow the Argilla server to work behind a proxy (#3543).
TextClassificationSettings and TokenClassificationSettings labels are properly parsed to strings both in the Python client and in the backend endpoint (#3495).
Fixed PUT /api/v1/datasets/{dataset_id}/publish to check whether at least one field and question has required=True (#3511).
Fixed FeedbackDataset.from_huggingface as suggestions were being lost when there were no responses (#3539).
Fixed QuestionSchema and FieldSchema not validating name attribute (#3550).

Deprecated

After calling FeedbackDataset.push_to_argilla, calling push_to_argilla again won't do anything since the dataset is already pushed to Argilla (#3465).
After calling FeedbackDataset.push_to_argilla, calling fetch_records won't do anything since the records are lazily fetched from Argilla (#3465).
After calling FeedbackDataset.push_to_argilla, the Argilla ID is no longer stored in the attribute/property argilla_id but in id instead (#3465).

As always, thanks to our amazing contributors

@plaguss made their first contribution in https://github.com/argilla-io/argilla/pull/3454

Full Changelog: https://github.com/argilla-io/argilla/compare/v1.13.3...v1.14.0

Rubrix Versions Save

v1.21.0

🔆 Release highlights

Draft queue

Improved shortcuts

New metrics module

Agreement metrics

Model metrics

List aggregation support for TermsMetadataProperty

Reindex from CLI

Added

Changed

Fixed

Removed

v1.20.0

🔆 Release highlights

Responses and suggestions filters

Utils module

Assign records

Multi-Modal DataURLs for images, video and audio

Token Highlights

Added

Fixed

Changed

v1.19.0

🔆 Release highlights

🚨 Breaking changes

Add vectors to your FeedbackDataset

Similarity search

FeedbackDataset

RemoteFeedbackDataset

ArgillaTrainer

🎨 UI improvements

Added

Changed

Fixed

Contributors

v1.18.0

🔆 Release highlights

💾 Add metadata properties to Feedback Datasets

🗃️ Filter and sort records using metadata in Feedback Datasets

⚠️ Breaking change using SQLite as backend in a docker deployment

Added

Changed

Fixed

New Contributors

v1.17.0

☀️ Highlights

🚨 Breaking change in updating existing Hugging Face Spaces deployments

⚠️ Breaking change using SQLite as backend in a docker deployment

💾 ArgillaTrainer Model Card Generation

🦮 FeedbackDataset Task Templates

⏱️ inserted_at and updated_at are added to responses

Added

Changed

Fixed

Deprecated

New Contributors

v1.16.0

☀️ Highlights

💾 Auto save

👨🏻‍💻 More operations directly from the Argilla CLI

⌨️ New keyboard shorcuts for the Feedback Dataset

QnA, Chat Completion with OpenAI and Sentence Transformers model training now in the ArgillaTrainer

Added

Changed

Fixed

As always, thanks to our amazing contributors

v1.15.1

Fixed

v1.15.0

🔆 Highlights

⚙️ Update FeedbackDataset settings from the UI

🤖 TRL integration with the ArgillaTrainer

🐍 Filter FeedbackDataset and remove suggestions

Added

Changed

Fixed

Removed

As always, thanks to our amazing contributors

New `metrics` module

List aggregation support for `TermsMetadataProperty`

Add vectors to your `FeedbackDataset`

`FeedbackDataset`

`RemoteFeedbackDataset`

`ArgillaTrainer`

💾 `ArgillaTrainer` Model Card Generation

🦮 `FeedbackDataset` Task Templates

⏱️ `inserted_at` and `updated_at` are added to responses

QnA, Chat Completion with OpenAI and Sentence Transformers model training now in the `ArgillaTrainer`

⚙️ Update `FeedbackDataset` settings from the UI

🤖 TRL integration with the `ArgillaTrainer`

🐍 Filter `FeedbackDataset` and remove suggestions