Hopsworks Versions Save

Hopsworks - Data-Intensive AI platform with a Feature Store

v3.7.0

2 months ago

Hopsworks

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-853 Model export fails in integration tests

HWORKS-862 Remote clients are not available on non das nodes

HWORKS-864 Istio doesn't use the docker images from the registry

HWORKS-865 NullPointerException when monitoring execution of a deleted job

HWORKS-869 Email validation regex doesn't support capital letters

HWORKS-872 Cannot disable oauth group mapping from cluster definition

HWORKS-878 Grafana uses wrong prometheus name

HWORKS-879 Upload will fail if the same file is uploaded again while the first upload is ongoing.

HWORKS-881 Validate SparkJob and Python job application file exists before launching job

HWORKS-882 Disallow exporting a model in the root path of a dataset

HWORKS-890 Restoring RonDB backup breaks because restore-data does not allow restoring in a table with unique indexes

HWORKS-898 PyTorch installation does not have GPU support

HWORKS-899 Broken image links in gh-pages documentation

HWORKS-901 Git installation support should not prepend git+

HWORKS-907 CloudManager should blacklist all instances from automatic removal

HWORKS-908 NPE in ProjectQouataController for a project that failed to get created

HWORKS-910 Endpoint identification alg missing from kafka get_default_config()

HWORKS-950 git pull fails if the user has multiple surnames

HWORKS-954 api_key for cert-operator is created empty

HWORKS-961 Do not turn on MySQL binlog if global replication is not enabled

HWORKS-974 Resource guard in ndb-chef for extracting RonDB does not work

HWORKS-975 OpenSearchApi in hopsworks-api should not rely on ELASTIC_ENDPOINT

HWORKS-976 Replacement of CREATE USER with CREATE USER IF NOT EXISTS is wrong in RonDB backup restore script

HWORKS-996 Removal of rotated one time JWT signing key should happen only on the Primary node

HWORKS-1001 Feature store sharing should also share data validation dataset

HWORKS-1029 Pin nbconvert in docker images

Task

HWORKS-135 Models backend should store metadata in tables instead of opensearch

HWORKS-302 Enhancement Request: Ability to disable access to Anaconda package repository and hide the functionality at a system defined level

HWORKS-707 Store descriptive statistics in NDB instead of HopsFS

HWORKS-802 Python 3.12 support for clients

HWORKS-808 add database/url to parameter in kafka cookbook

HWORKS-854 hops-system should use the minimal env definition without pydoop

HWORKS-855 Remove Kagent ability to do service key rotation

HWORKS-861 Disable Kafka alerting rules when bring your own kafka is enabled

HWORKS-868 Increase yarn.resourcemanager.rmappsecurity.jwt.validity to 1 hour

HWORKS-871 Hopsworks should be able to handle oauth claims in array format

HWORKS-873 Add flag to delete RonDB backup with the same backup-id

HWORKS-877 Helm chart for certs-operator

HWORKS-880 Model registry tests should account for if model is stored on hdfs and also run in PySpark

HWORKS-883 Fix Sonarqube issues in CommandsController

HWORKS-886 OnlineFS should be able to read Kafka configuration from environment variables

HWORKS-887 certs-operator deploy appropriate certificate for Strimzi operator

HWORKS-889 Chef flag to disable unattended upgrades

HWORKS-891 Add K8s GPUs monitoring to Grafana/Jupyter/Job UI

HWORKS-892 Use a single YarnClientWrapper to monitor all jobs

HWORKS-893 Nightly tests should run without installing the requirements.txt

HWORKS-894 Add pytest to python base environment

HWORKS-895 Upgrade xgboost to 2.0.3

HWORKS-897 Upgrade project python environment libraries

HWORKS-900 Upgrade to PyTorch 2.1.2

HWORKS-909 Add hopsworks-api opensearch and kafka workflow tests

HWORKS-912 Upload staging dir does not need to be configurable.

HWORKS-919 Create database on the online feature store on-demand

HWORKS-934 Use more appropriate data types for statistics

HWORKS-944 Pin upper pandas to 2.1.4

HWORKS-946 Global Chef attribute for arbitrary Systemd unit dependencies

HWORKS-949 Add git repo pull to workflow tests

HWORKS-962 Purge binlog files when Global replication is enabled

HWORKS-965 In some OSes DNS resolution does not work from within a Kubernetes Pod

HWORKS-973 Share feature store with project on creation

HWORKS-977 Mount hopsfs by default in jupyter notebooks and python jobs containers

HWORKS-980 Don't set DB storagePolicy for Hive warehouse location

HWORKS-992 Remove code for generating service Renewal JWTs

HWORKS-1050 Run expat as glassfish hopsfs user

Feature Store

Epic

FSTORE-612 Feature Monitoring

Bug

FSTORE-830 Error fetching feature statistics from feature view UI - but statistics exists with a different timestamp

FSTORE-856 Only one OnlineFS instance running

FSTORE-987 Failed to read data when there is a self-join

FSTORE-989 GCS connector-Encryption fields and secrets update issues

FSTORE-992 create_train_validation_test_split fails with unexpected keyword argument 'pit_query_asof'

FSTORE-998 Can't read from a shared feature store

FSTORE-1000 Tags parameter is missing in TrainingDataset class

FSTORE-1034 .select() method should not default to empty list

FSTORE-1035 No error if a user tries to create a feature view without features

FSTORE-1084 Cannot run multiple insert_stream query on the same project by default

FSTORE-1089 Remove copying of application code from databricks integration

FSTORE-1095 JDBC storage connector missing driver option in the documentation

FSTORE-1099 Nightly test test_append_feature failing on float type mistmatch

FSTORE-1100 Improve OnlineFS offset saving

FSTORE-1110 Recreation of Training dataset throwing exception due to NULL reference

FSTORE-1111 OnlineFS monitoring shows wrong clusterj session count

FSTORE-1118 Pandas arrow type dataframes cannot be inserted into feature group

FSTORE-1120 get_feature_vector(s) init fails if feature view contains complex feature

FSTORE-1122 QueryController issue when joining one feature group multiple times

FSTORE-1126 python mysql client sometimes failed when user name is too long

FSTORE-1138 Java client is not designed to work with shared feature store

FSTORE-1139 Get feature store requests can throw NullPointerException is the project is not properly initialized

FSTORE-1140 OnlineFS loads wrong kafka property files

FSTORE-1144 Hopsworks should not change the Access Mode of the Databricks cluster

FSTORE-1146 Appending lots of features results in error to commit activity update

FSTORE-1154 OnlineFS onPartitionsAssigned ConcurrentModificationException

FSTORE-1156 Fix FeatureView.clean() code snippet to use the static method

FSTORE-1161 Arrowflight server hangs when instantiating a FlightServer instance and cannot validate the certificates.

FSTORE-1174 Spark streaming workflow test failing

FSTORE-1181 Helper columns should return all columns if they have different names across feature groups

FSTORE-1184 Kafka storage connector confluent_options method uses wrong certificates.

FSTORE-1187 Execution failing with framework_failure causes insert to hang forever

FSTORE-1189 Arrowflight `tls` option parsing the wrong number of arguments

FSTORE-1193 HSFS python tests fail after moto has been updated

FSTORE-1199 Training dataset info objects cannot be retrieved within a job

FSTORE-1203 'FeatureGroup' object has no attribute 'time_travel_format'

FSTORE-1230 fg.select_all().read() on PySpark does not return latest version of a Feature Group

FSTORE-1233 AML tutorial replace append (removed from Pandas) and fix grammar errors

FSTORE-1252 In Neo4j tutorial. fix feature group order in query when creating feature view

FSTORE-1253 Create training test split without materialization fails when during computing statistics when Feature Group contains certain types.

FSTORE-1272 Statistics cleaner should not delete statistics during migration

Subtask

FSTORE-1132 Enable filtering in find_neighbors

FSTORE-1133 Enable deleting embedding index

FSTORE-1134 Enable updating embedding

FSTORE-1136 Add documentation for similarity search in hsfs

FSTORE-1164 Support composite primary key in similarity search

FSTORE-1276 Use a larger value of k in find_neighbors when using project index

Task

FSTORE-878 Integrate Docsbot into Docs

FSTORE-885 Catch no data error when fetching dataframe in feature monitoring

FSTORE-905 Add tests for training dataset statistics computation

FSTORE-951 Increase test coverage for feature monitoring

FSTORE-971 Do not compute statistics on in-memory training datasets

FSTORE-1021 Add a warning when using externally managed Kafka

FSTORE-1022 Missing data in featurestore benchmark

FSTORE-1042 Investigate increasing time to make hudi commits

FSTORE-1046 Improve online feature store metrics

FSTORE-1052 Update the docs to include Kafka config variables for throughput

FSTORE-1055 Support pandas 2.1.*

FSTORE-1060 Add load tests that read feature groups and feature views

FSTORE-1063 Hopsworks data preview should use arrow flight to retrieve data if available

FSTORE-1070 Tutorial for external Flink client

FSTORE-1074 Expand documentation on filter logic

FSTORE-1090 Concepts & Guides for helper columns and on-demand features

FSTORE-1097 Add user id to workflow test and unit test

FSTORE-1103 AML Tutorial

FSTORE-1104 Pin numpy < 2

FSTORE-1106 inconsistencies in training dataset documentation

FSTORE-1112 Support Similarity Search in the Feature Store v1 - OnlineFS

FSTORE-1113 Support Similarity Search in the Feature Store v1 - HSFS

FSTORE-1115 HSFS should be able to read data from Databricks Unity Catalog

FSTORE-1116 Parallelize PK lookups for get feature vectors

FSTORE-1119 Support Similarity Search in the Feature Store v1 - migration

FSTORE-1121 Support Similarity Search in the Feature Store v1 - onlinefs monitoring

FSTORE-1124 [OnlineFs] Subject id should add to black list if feature group is not found

FSTORE-1125 Add retry when vector db is empty in load test

FSTORE-1127 Fix log4j vulnerability

FSTORE-1128 Fallback to head node host in opensearch api

FSTORE-1130 Get schema from shared project

FSTORE-1131 DeltaStreamer job fails when feature group has complex feature

FSTORE-1142 Remove markupsafe<2.1.0 pinning

FSTORE-1143 Bump fastavro to 1.8.4 to install using provided wheel in python 3.12 environments

FSTORE-1147 Online feature store notification system

FSTORE-1148 Python to Kafka writing analysis and benchmarking

FSTORE-1152 Add support for listing training datasets metadata from a feature view object

FSTORE-1166 Add backend support for Delta as time travel format

FSTORE-1167 Add Api support for delta time travel format

FSTORE-1168 Update databricks instance configurator to configure cluster for delta

FSTORE-1175 Add FeatureGroup.read_changes(..) and FeatureGroup.as_of(..) workflow tests

FSTORE-1182 Extend spark-no-metastore engine to compute statistics and remove calling use database

FSTORE-1196 Neo4j tutorial

FSTORE-1213 Remove support for configuring databricks instances from hsfs API

FSTORE-1259 Add Change Notification for Feature Groups Example

v3.5.0

5 months ago

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-267 After fail checkout, branch is empty on pull

HWORKS-269 Don't show the same in file status and in UI table

HWORKS-309 Bugs in searching

HWORKS-331 Hopsworks metrics not showing after payara restart

HWORKS-374 opensearch_api.get_default_py_config() returns public IP of host, should return consul fqdn

HWORKS-386 Spark job does not accept space in arguments

HWORKS-469 Dependent conda commands not handled correctly in case of failure

HWORKS-475 hopsworks Python library can't connect in sklearn legacy deployments

HWORKS-492 Servings logs are not shown in Kibana

HWORKS-493 Request batching configuration is broken in the deployment creation form

HWORKS-496 Exception not raised if other than duplicated deployment entry

HWORKS-498 Transformer resources in dict format are not properly deserialized

HWORKS-499 Support for jupyter notebooks as predictor scripts

HWORKS-561 Hopsworks Kafka Authorizer does not allow IDEMPOTENT_WRITE operations

HWORKS-569 Filebeat cannot list nodes which is required for scraping container logs

HWORKS-602 HA cluster could not connect to live logs, rm UI, spark UI, monitor, onlineFS

HWORKS-630 Project creation doesn't fail on invalid project name

HWORKS-633 Failing deployments are shown as Starting when Kubernetes stops restarting pods

HWORKS-636 Inference logging default value is overwritten when updating a deployment using .save()

HWORKS-656 NP trying to force delete a partially created project

HWORKS-693 Server logs not collected when the predictor class fails to initialize

HWORKS-786 Storage connector specs failure for flag disabled case

HWORKS-787 Git repository current commit and branch not being updated

HWORKS-792 Hopsworks job schedule update doesn't update cron expression

HWORKS-809 Project creation fails when the serving api key k8s secret takes too long to be available

HWORKS-815 Judge service should be restarted when docker daemon restarts

HWORKS-817 Log level in RonDB replication scripts is wrong

HWORKS-819 The request URI port is absent in MultiRegionFilter

HWORKS-820 In MultiRegionFilter the Primary region is null if invoked too early

HWORKS-821 Kubernetes does not accept serving names starting with a number

HWORKS-824 Only Primary MySQLs should be able to update RonDB replication metadata table

HWORKS-833 Consul datacenter is hardcoded in Prometheus configuration

HWORKS-834 Undeploy hopsworks on upgrade

HWORKS-837 RonDB native backup script fails when there is no database to backup

HWORKS-848 AlertManager EJB does not take Datacenter into consideration in Multiregion clusters

HWORKS-851 Check in hops-hadoop-chef for creating RonDB undo and log files is wrong

HWORKS-865 NullPointerException when monitoring execution of a deleted job

Task

HWORKS-135 Models backend should store metadata in tables instead of opensearch

HWORKS-164 Add Airflow documentation

HWORKS-198 Remove hopsworks::image recipe

HWORKS-226 python api should throw informative error if user tries to download a folder

HWORKS-258 Add hopsworks python sdk dataset upload/download to workflow testing

HWORKS-262 Remove user email FK from hopsworks schema.

HWORKS-266 Checking out a branch that exist on remote repo does not work

HWORKS-284 Documentation to export cluster logs

HWORKS-302 Enhancement Request: Ability to disable access to Anaconda package repository and hide the functionality at a system defined level

HWORKS-314 Add support for git fetch and reset

HWORKS-422 Exclude old versions of the documentation from being indexed by search engines

HWORKS-491 Replace hsml with hopsworks in the code snippets shown in the UI

HWORKS-494 Support local script path in model.deploy()

HWORKS-511 Add number of GPUs to Python resource usage

HWORKS-520 Make oauth claims configurable

HWORKS-536 Move master encryption password to KMS

HWORKS-559 Users should be able to specify a directory when using the Hopsworks-api to download logs

HWORKS-565 Hopsworks api dataset upload should support parallel chunk upload

HWORKS-571 Run more integration tests in parallel

HWORKS-590 Make jenkins test use HA cluster

HWORKS-591 Show a better log for when pods are killed by Kubernetes with OOM

HWORKS-617 Make load balancer use https

HWORKS-631 Support info exchange between preprocess and postprocess methods in transformers

HWORKS-635 Add unit tests and loadtest for hsml deployments

HWORKS-641 Add loadtests for hsml and model registry

HWORKS-662 Fix versioning and build automation for hopsify

HWORKS-676 Supress CVE-2023-33265 in payara-embedded-web-5.2022.5.jar/META-INF/maven/com.hazelcast/hazelcast/pom.xml:

HWORKS-677 Supress multiple severe CVE in hive-storage-api-2.6.1.2.jar

HWORKS-679 Add option to set python environment

HWORKS-690 Airflow project dags sharing

HWORKS-697 Remove airflow JWT

HWORKS-707 Separate statistics from feature monitoring PR

HWORKS-729 Nightly loadtest cluster should have an admin user that is member of all projects

HWORKS-745 Support python 3.11 for the client APIs

HWORKS-746 Documentation for python environment history and custom commands

HWORKS-751 Add webhook support for Hopsworks alerts

HWORKS-756 Remove cmake dependency in hops-hadoop-chef

HWORKS-759 Show custom commands file that was used to build the environment in the environment history

HWORKS-764 Reduce JWT lifetime in nightly tests

HWORKS-774 Set hopsworks_rest_log_level to DEV in the nightly tests

HWORKS-775 Nightly tests should reload random module

HWORKS-777 Fix integration test

HWORKS-780 Increase number of concurrent load tests

HWORKS-783 Cleanup PIA functionality

HWORKS-784 Remove airflow jdbc metric from payara

HWORKS-789 Fix git tests

HWORKS-791 Add Python API support to schedule jobs

HWORKS-795 Hopsworks CA should generate Java keystores instead of a shell script on the client

HWORKS-798 Add **kwargs to python client libraries

HWORKS-800 Implement dataset.copy and dataset.move API

HWORKS-801 Export onlinefs/default/private_ips in metadata

HWORKS-803 Remove airflow from old UI

HWORKS-804 upgrade zookeeper 3.8.2 to 3.8.3 CVE-2023-44981

HWORKS-805 Environment history compares versions lexicographically instead of numerically per component

HWORKS-807 Add wait_until_finished method to execution object in the hopsworks python apis

HWORKS-829 Register model files without compressing them

HWORKS-830 Configurable glassfish tmpdir property

HWORKS-832 Set size limit in Consul log files

HWORKS-835 Download model artifact without compressing model files

HWORKS-841 Add 'See resource usage' button in the deployment overview page

HWORKS-847 RonDB Global replication alert

HWORKS-849 Check if dag has access_control is not None before syncing dag permissions in airflow

HWORKS-854 hops-system should use the minimal env definition without pydoop

Feature Store

Epic

FSTORE-473 Data Management

FSTORE-612 Feature Monitoring

FSTORE-1047 Support Similarity Search in the Feature Store

Bug

FSTORE-55 Backfill job gets stuck

FSTORE-537 Explicit provenance ClassCastException thrown

FSTORE-754 Feature Group creation during RonDB reconfig leaves behind broken feature group

FSTORE-755 Uninformative error when deleting feature group during RonDB reconfig

FSTORE-799 HSFS should reinstantiate sqlalchemy connectrion pool if MySQL node disappears

FSTORE-813 Unclear error when the feature group contains duplicated columns

FSTORE-830 Error fetching feature statistics from feature view UI - but statistics exists with a different timestamp

FSTORE-856 Only one OnlineFS instance running

FSTORE-892 FM jobs are not removed when the corresponding FM configuration is deleted

FSTORE-916 FeatureGroup.insert does not retry in case of a failure

FSTORE-932 Pyarrow dependency is required even if the python profile is not installed

FSTORE-943 NullPointerException in SchematizedTagHelper

FSTORE-944 Nested filter statements are not handled correctly when attached to a feature view

FSTORE-968 Add workflow test for get_batch_data

FSTORE-976 Feature group filter in PIT join gets pushed down on the temporary table that doesn't contain the feature group

FSTORE-987 Failed to read data when there is a self-join

FSTORE-989 GCS connector-Encryption fields and secrets update issues

FSTORE-992 create_train_validation_test_split fails with unexpected keyword argument 'pit_query_asof'

FSTORE-998 Can't read from a shared feature store

FSTORE-1000 Tags parameter is missing in TrainingDataset class

FSTORE-1005 Training datasets are written and read using a wrong path

FSTORE-1009 hsfs flink consumer doesn't work with parallelism greater than 1.

FSTORE-1011 Validation message during data ingestion to FG when there are no kafka topics configured

FSTORE-1017 Updating job schedule's cron expression doesn't work

FSTORE-1023 Mix up of subjectId and schemaId

FSTORE-1024 Concurrency can cause SQLIntegrityConstraintViolationException when creating topic

FSTORE-1026 Wrong td file path created by ArrowFlight

FSTORE-1029 Unexplained OnlineFS exceptions in logs

FSTORE-1034 .select() method should not default to empty list

FSTORE-1035 No error if a user tries to create a feature view without features

FSTORE-1037 ArrayIndexOutOfBounds in TrainingDatasetController

FSTORE-1038 Misleading exception for ArrowFlight when not waiting for ingestion job to finish

FSTORE-1044 Insert fails with Feature 'id': dtype 'O' (arrow_type 'null') not supported

FSTORE-1049 FlyingDuck gets NoneType in evaluate_filter_expression

FSTORE-1050 FlyingDuck gets Permission denied trying to read hoodie.properties file

FSTORE-1051 NotImplementedException when querying FeatureView based on external feature group

FSTORE-1053 Never ending materialization job

FSTORE-1057 Schema inconsistency in DuckDB after upsert

FSTORE-1058 if feature view query doesn't include event time feature add for get_batch_data(start_time, end_time)

FSTORE-1059 ArrowFlight server hangs after some requests

FSTORE-1062 Materialisation job does not work with new Hopsworks scheduler

FSTORE-1068 Fixed failed load test

FSTORE-1071 Flyingduck crashes when joining shared feature stores

FSTORE-1072 Connection timeout not set in the arrow_flight_client.py

FSTORE-1073 OnlineFS getting offsets fails when project name has upper case letters

FSTORE-1074 Expand documentation on filter logic

FSTORE-1075 Cannot get schema from shared project

FSTORE-1084 Cannot run multiple insert_stream query on the same project by default

FSTORE-1086 Broken query when adding a filter on the non-label feature group without selecting the feature

FSTORE-1087 feature_view.get_batch_data timing out when using ArrowFlight

FSTORE-1089 Remove copying of application code from databricks integration

FSTORE-1094 pyarrow._flight.FlightServerError: 'fileIdAndRelativePaths' when reading from ArrowFlight

FSTORE-1095 JDBC storage connector missing driver option in the documentation

Subtask

FSTORE-1007 Fix training dataset paths on the backend and adjust them on hsfs

Task

FSTORE-26 Print Warning that backfill job had to be started manually, when PySpark engine stream=True and using .insert_stream()

FSTORE-335 Track GIT commit that was used to create/insert into a feature group

FSTORE-362 Replace Deequ statistics with more lightweight module

FSTORE-363 Make HSFS API typesafe

FSTORE-367 Create internal guide for hsfs unit test

FSTORE-382 Feature store v2.5 documentation - broken links

FSTORE-394 Unify metadata update API in hsfs

FSTORE-404 Add rollback action for Fs job creation.

FSTORE-471 Allow tranformation functions for label features

FSTORE-550 Ensure type consistency between for online/offline/feature view reads in python and spark engine

FSTORE-665 Design log: Kafka topic deletion

FSTORE-722 Hopsworks should not retrieve metadata from the Hive metastore

FSTORE-725 Transformation Function unit test should test the default behaviour of the pre-registered function in the backend

FSTORE-738 UI should show commit timestamps in UTC format

FSTORE-753 Improve user error message for feature group creation and append during RonDB reconfig

FSTORE-790 Replace hsfs with hopsworks in the code snippets shown in the UI

FSTORE-823 Statistics Engine - Design Log

FSTORE-825 Deleting an online feature group doesn't revoke mysql privileges

FSTORE-826 Add test support for AWS and GCP connectors

FSTORE-828 Automatically add prefix if there are duplicate columns in the query join.

FSTORE-834 Relative path in absolute URI - External FG

FSTORE-845 Update Apache Hudi version to 0.14.0

FSTORE-870 Issues including None/Nan/Null values in queries on feature views

FSTORE-876 Remove Stats from Write Path

FSTORE-878 Integrate Docsbot into Docs

FSTORE-880 Improve error message when there is no match of primary key in get_feature_vector

FSTORE-881 Appending to feature groups after upgrade from 3.0 to 3.2 seems off

FSTORE-885 Catch no data error when fetching dataframe in feature monitoring

FSTORE-895 Capability Write Up - Online Inference Pipelines

FSTORE-896 Capability Write Up - Batch Inference Pipelines

FSTORE-897 Capability Write Up - Training Pipelines

FSTORE-898 Capability Write Up - Feature Engineering in Beam

FSTORE-900 Capability Write Up - Feature Engineering in SQL

FSTORE-901 Capability Write Up - Feature Engineering in Spark

FSTORE-905 Add tests for training dataset statistics computation

FSTORE-909 Capability Write Up - On-demand Feature Engineering

FSTORE-920 support for JDBC test connection

FSTORE-925 support for ADLS test connection

FSTORE-927 Upgrade great expectations to 0.15.12

FSTORE-930 Allow pyarrow object type in pandas 2

FSTORE-933 DBT Scheduling

FSTORE-941 Support versioning of Feature Monitoring configurations

FSTORE-949 FlyingDuck - Pandas 2.0 Pyarrow-backed Types support

FSTORE-950 Split feature descriptive statistics table into three tables

FSTORE-951 Increase test coverage for feature monitoring

FSTORE-954 Investigate how to apply transformation function in REST API

FSTORE-957 FS REST API benchmark

FSTORE-967 Bump intellij plugin version

FSTORE-971 Do not compute statistics on in-memory training datasets

FSTORE-973 JDBC connector - add driver upload and driver name fields

FSTORE-980 Helper, Primary key and event time columns with feature view

FSTORE-990 Incorporate Flying Duck in nightly tests

FSTORE-1003 Build locusts tests benchmarking Vertex AI- FS

FSTORE-1008 add java client to hsfs

FSTORE-1015 Add option to add Keytab to Kafka connector

FSTORE-1019 Single Kafka topic per project documentation

FSTORE-1021 Add a warning when using externally managed Kafka

FSTORE-1022 Missing data in featurestore benchmark

FSTORE-1025 Support datetime in training dataset time series split

FSTORE-1036 Tracking usage of hsfs libraries

FSTORE-1040 SageMaker Online benchmarking

FSTORE-1041 Databricks online benchmarking

FSTORE-1042 Investigate increasing time to make hudi commits

FSTORE-1043 Historical Data from Feature View

FSTORE-1046 Improve online feature store metrics

FSTORE-1052 Update the docs to include Kafka config variables for throughput

FSTORE-1055 Support pandas 2.1.*

FSTORE-1056 Increase sleep time in nightly tests after online inserts to handle heavy concurrent load

FSTORE-1060 Add load tests using concurrent clients that read feature groups and feature views

FSTORE-1061 Add feature group id to serving key prefix and documentation in get_feature_vector

FSTORE-1063 Hopsworks data preview should use arrow flight to retrieve data if available

FSTORE-1064 Improve Spine Group docs

FSTORE-1065 update streaming apis in hopsworks-tutorials

FSTORE-1066 Shared feature store workflows tests failing with permission denied

FSTORE-1067 Feast online scenarios

FSTORE-1069 Foreign key constraint

FSTORE-1070 Tutorial for external Flink client

FSTORE-1078 Support Similarity Search in the Feature Store v1.5

FSTORE-1080 Upgrade hudi to version 0.12.3

FSTORE-1085 Add profiles to HSFS to support multiple Spark version

FSTORE-1088 Bump recommended Databricks runtime version to 12.2

FSTORE-1090 Concepts & Guides for helper columns and on-demand features

FSTORE-1091 Feature monitoring Tutorial

FSTORE-1092 Helper Columns Tutorial

FSTORE-1097 Add user id to workflow test and unit test

v3.4.0

7 months ago

Hopsworks 3.4.0

Bug

HWORKS-372 explicit provenance missing community stereotype

HWORKS-397 Links to Model Serving API reference are broken

HWORKS-426 Latest versions of managed Kubernetes may not support first-party-jwt

HWORKS-538 dataset sharing acl bug

HWORKS-610 HazelcastSerializationException on JupyterJWTManager

HWORKS-632 Command.from_response_json in hopsworks-api should parse based on items field

HWORKS-638 Kafka 3 removed --zookeeper option

HWORKS-639 Template cert store passwords in onlinefs-kafka.properties

HWORKS-640 Nightly workflow and load tests should not run as admin user

HWORKS-654 extra constraints migration does not respect the order of the columns

HWORKS-663 remote_file does not support not_if condition

HWORKS-665 Bump Zookeeper dependency in Hopsworks to 3.8.2 to fix CVE-2023-34462

HWORKS-666 Configuring HA Glassfish fails when importing certificates on worker nodes

HWORKS-670 Set Flink job staging and checkpoint directories per user

HWORKS-675 Exclude logback dependencies from zookeeper library in Hopsworks

HWORKS-680 Karamel with recipe parallelism restrictions schedules tasks although a previous task has failed

HWORKS-687 ZooKeepers fail to start because they cannot resolve other ZK consul domain name

HWORKS-688 Python library installation error message not latin1_general_cs

HWORKS-689 oauth logout uri is set to redirect uri

HWORKS-711 User deletion fails with FK constraint

HWORKS-731 Model deployment with a wrong predictor script path fails with a non-intuitive error

HWORKS-735 Chef attribute to configure ldap_groups_search_filter

HWORKS-736 ndb-chef download backup should skip host key verification

HWORKS-750 Pin traitlets in project python environment

Task

HWORKS-248 Publish all images to the internal docker registry

HWORKS-343 Add support for accessing a shared feature store using project.get_feature_store()

HWORKS-351 Add support for running generic Docker commands

HWORKS-354 Remove python_kernel variable

HWORKS-489 Register test code using register_assert_rewrite

HWORKS-544 Opensearch docs creation moved to Hopsworks

HWORKS-545 Move Tags and Keyword to Hopsworks db

HWORKS-546 Feature components - remove xattr - use Hopsworks db

HWORKS-547 featurestore search - static reindex

HWORKS-572 Add uptime to status-all-local-service.sh

HWORKS-576 Test upgrade from HA cluster to HA

HWORKS-589 Remove metastore foreign key from cached/streaming feature groups

HWORKS-618 Make integration test use HA cluster def

HWORKS-627 Log more information during Project creation

HWORKS-628 Skip project creation failure alert if a project with the same name exists

HWORKS-637 Add Release Candidate version to spark-chef for hsfs

HWORKS-642 Online feature group logs

HWORKS-646 Move airflow dags to hopsfs

HWORKS-647 Replace Chef with Cinc

HWORKS-667 MySQL X plugin socket should not be at /tmp

HWORKS-672 Remove angular file browser in old UI

HWORKS-673 Remove Hopsworks Kagent monitor

HWORKS-678 Allow configuration of RDRS with user supplied certificates

HWORKS-681 Flink apis for hopsworks python library

HWORKS-682 Include return type and description for model.download()

HWORKS-684 Separate job scheduler from feature monitoring PR

HWORKS-685 Add HOPS_SERVICE_USER account type to Python Environment endpoints

HWORKS-686 Zip and push airflow image tar to master in jenkins pipeline

HWORKS-696 Add documentation on project quota management

HWORKS-698 Precreate the project root dir in hopsfs

HWORKS-699 Update docker daemon security configuration

HWORKS-700 Remove kagent_disable check in conda::default and template kagent_utils there

HWORKS-701 Add date filter to python environment history

HWORKS-702 Automate setup of RonDB Global replication

HWORKS-703 Add consul/enabled flag to disable consul operations if consul is not available

HWORKS-704 hopsworks::install should create the hopsmon user if it doesn't exists

HWORKS-705 Chef should not pre-register the hosts

HWORKS-706 Switch opensearch-dashboard to use consul instead of IPs

HWORKS-708 Use git api to delete git repository in the hopsworks python SDK

HWORKS-709 hops-hadoop-chef should not overwrite the tls/enabled parameter if hopsworks::default is missing

HWORKS-710 Hopsworks database should create the ts_1 tablespace for the certificates

HWORKS-712 Add option to tag RonDB management node for onlinefs

HWORKS-713 Create a symlink to hopsfs-mount directory in the default working directory in jupyter terminal

HWORKS-715 Add hops::docker dependency to rdrs and onlinefs recipes

HWORKS-716 Add append2ChefTrustAnchors to elastic::default

HWORKS-717 Remove consul dependency in hops::client

HWORKS-718 Switch elastic clients to use consul fqdn and not ips

HWORKS-719 Add chef code to configure WAN federated consul

HWORKS-720 Remove Kafka waiter in Onlinefs systemd script if external kafka

HWORKS-723 Add rondb_replication to the list of forbidden project names

HWORKS-724 Implement multiregion controller in Hopsworks

HWORKS-726 Delete Chef cached attributes json in beginning of new installation

HWORKS-727 Bump Hops version to 3.2.0.11-RC0

HWORKS-728 Add secondary region attribute in Chef

HWORKS-734 Remove jdbc pool when initializing online feature store database for project

HWORKS-737 Deleted project leads to NPE when processing SearchFSCommand

HWORKS-738 Recipe to setup Judge service

HWORKS-740 Expat - disable provenance on existing projects

HWORKS-741 Upgrade Grafana to version 9.3.16

HWORKS-749 Remove chef-ulimit dependency from hopsworks-chef Berksfile

HWORKS-752 Flink workflow tests should get latest release candidate dynamically

Feature Store

Epic

FSTORE-709 Rest API to retrieve data from the online feature store

Bug

FSTORE-728 Improve error msgs when wrong API key provided for hopsworks.login()

FSTORE-824 Shows undefined in accept shared feature store prompt

FSTORE-842 Setting fg.insert(storage="online") adds data to offline as well

FSTORE-857 Appending to an online feature group fails after upgrade

FSTORE-879 Fix consecutive slashes (//) in file selector component

FSTORE-906 Training data shouldn't be recreated when recomputing stats

FSTORE-907 Recomputing statistics on feature groups without time-travel enabled fails

FSTORE-921 Resolve the set of serving key in get_feature_vector

FSTORE-928 When hitting limit of number of projects that one user can create, deleting a project doesn't work as expected

FSTORE-942 ADLS connector path issues

FSTORE-946 Orphan statistics are not removed from the DB

FSTORE-965 fastavro fails to build when installing HSFS

FSTORE-966 Removing spark options on update does not work on bigquery connector

FSTORE-969 Scheduled jobs are not triggering when expected

FSTORE-993 HSFS Java should use UTF-8 when creating the StringEntity

FSTORE-999 Cannot get feature vector from a shared feature view

FSTORE-1012 Hopsworks-api got an unexpected keyword argument 'feature_store_topic'

FSTORE-1016 Onlinefs schema issue

Task

FSTORE-50 Tutorial how to write custom transformation function for feature view

FSTORE-449 Use Caffeine cache for schemas

FSTORE-576 Tutorial for transformation functions in sklearn

FSTORE-654 Should throw proper error if data format is not provided.

FSTORE-672 Add shared feature store workflows tests

FSTORE-742 Statistics API

FSTORE-803 Beam support for HSFS java client

FSTORE-839 Single Kafka topic per project

FSTORE-840 Use existing Kafka cluster

FSTORE-860 EJB timer to delete orphan statistics

FSTORE-862 Allow naming of Kafka topics for data ingestion

FSTORE-865 Update online feature store logs

FSTORE-866 Add option to deprecate a feature group

FSTORE-877 Create Training Datasets in Flyingduck

FSTORE-884 Flying Duck - Write End-User Documentation

FSTORE-887 Compute statistics on both untransformed and transformed data when creating a Training Dataset

FSTORE-888 Upgrade flink 1.14 to flink 1.17

FSTORE-891 SQL and expat migrations for Statistics

FSTORE-899 Capability Write Up - Feature Engineering in Flink

FSTORE-903 External Warehouse Access Flyingduck

FSTORE-904 Flying Duck - Write Blog Article

FSTORE-908 Upgrade scheduler to near real-time standard and cron frequencies

FSTORE-926 Hopsworks BigQuery Storage Connector can't create and write data to external FG from Dataproc cluster

FSTORE-938 Remove enabled flag on Feature Monitoring Configuration to only use scheduler flag

FSTORE-939 Change name of Stream Feature Group's DeltaStreamer Job

FSTORE-940 Constraints for job and scheduler deletion

FSTORE-947 Tutorials Streamlit Fix

FSTORE-948 Add ASOF Joins to Hopsworks Query Constructor

FSTORE-952 Single Kafka topic per project

FSTORE-956 FS REST API logging and metric

FSTORE-958 Rename and split tutorial notebooks as feature, training and inference pipelines

FSTORE-961 tutorial for bytewax / hsfs integration

FSTORE-964 Exclude extra data from get all feature groups api

FSTORE-970 Include schedule in the job DTO

FSTORE-977 FlyingDuck - Slides Pydata

FSTORE-978 Hopsworks recommender system tutorial

FSTORE-979 Flink hsfs client workflow test

FSTORE-981 Make clear that from python env only stream=True is supported

FSTORE-982 Adopt serving key in feature store Rest API

FSTORE-984 FlyingDuck BigQuery Connector

FSTORE-988 Add PIT join tests to workflow/load tests

FSTORE-994 Use non-ascii characters in user first and last name in nightly tests

FSTORE-995 Return feature value in get_feature_vectors when entry is str instead of int

FSTORE-996 FS REST API Alert

FSTORE-1010 Don't leave orphaned subjects when deleting online enabled fg

FSTORE-1013 Hopsworks-api allow to specify feature_store_topic when creating project

FSTORE-1014 [Online FS REST API] Failed to return features when only primary key is selected

FSTORE-1020 Unable to delete a FeatureGroup with a FeatureView with filters

FSTORE-1027 FS docs fix

FSTORE-1028 POC for prospect customer

FSTORE-1031 Update Colab links in docs

v3.3.0

9 months ago

Hopsworks 3.3.0

Bug

HWORKS-224 hopsworks python SDK opensearch link broken

HWORKS-267 After fail checkout, branch is empty on pull

HWORKS-269 Don't show the same in file status and in UI table

HWORKS-309 Bugs in searching

HWORKS-319 User active projects not updated

HWORKS-331 Hopsworks metrics not showing after payara restart

HWORKS-374 opensearch_api.get_default_py_config() returns public IP of host, should return consul fqdn

HWORKS-379 LDAP users cannot download certificates

HWORKS-386 Spark job does not accept space in arguments

HWORKS-397 Links to Model Serving API reference are broken

HWORKS-459 Audit log caller should be consistent

HWORKS-469 Dependent conda commands not handled correctly in case of failure

HWORKS-475 hopsworks Python library can't connect in sklearn legacy deployments

HWORKS-492 Servings logs are not shown in Kibana

HWORKS-493 Request batching configuration is broken in the deployment creation form

HWORKS-496 Exception not raised if other than duplicated deployment entry

HWORKS-498 Transformer resources in dict format are not properly deserialized

HWORKS-499 Support for jupyter notebooks as predictor scripts

HWORKS-501 Collect logs in Flink UI and distribute custom libraries

HWORKS-521 Resolve ruby test failures

HWORKS-524 HSML ignores knative_domain parameter

HWORKS-525 Python resource monitor for K8s counts all nodes

HWORKS-528 Agent users should never be blocked

HWORKS-533 Nightly jenkins pipeline reporting 500 failures even though we have only 40 tests

HWORKS-539 Storage initializer broken after release of requests 2.29.0

HWORKS-540 Bump guava version to 27.0-jre

HWORKS-541 update CVE suppression - hops updated dependencies

HWORKS-558 Old UI getting 404 trying to get dela client

HWORKS-561 Hopsworks Kafka Authorizer does not allow IDEMPOTENT_WRITE operations

HWORKS-562 Serving hashcode method should handle null cases

HWORKS-563 Bump commons-configuration2 in expat

HWORKS-566 User account creation fails if the serving api key is not available within 3 seconds

HWORKS-568 Expat fails to start with logging class not found exception

HWORKS-569 Filebeat cannot list nodes which is required for scraping container logs

HWORKS-570 Flatbuffer class not found exception when running arrow internal spark

HWORKS-577 Upgrade 3.2 to 3.3 failing on kzookeeper::default

HWORKS-599 Execution status is not being updated

HWORKS-600 Python library installation failure on non DAS nodes

HWORKS-602 HA cluster could not connect to live logs, rm UI, spark UI, monitor, onlineFS

HWORKS-604 Collect glassfish node logs in logstash

HWORKS-605 GitTimeoutCommandsMonitor not working in HA

HWORKS-606 Spark and Flink configuration files are not present in Glassfish config nodes

HWORKS-608 Git commit message might be bigger than VARCHAR(1000)

HWORKS-609 Alertmanager config missing route.repeat_interval

HWORKS-610 HazelcastSerializationException on JupyterJWTManager

HWORKS-611 ApiKey filter should check if user is blocked

HWORKS-612 hops-hadoop-chef should automatically set HopsYARN HA attributes

HWORKS-614 hopsworks.login on a project with shared feature store

HWORKS-616 Upgrade fails on fuse_mnt

HWORKS-619 Python jobs are always set to FAILED even after completion

HWORKS-622 hsml connection fails to close

HWORKS-624 Race condition in LibraryInstaller CondaCommand execution

HWORKS-625 Kafka storage connector fails migration if inode is null

HWORKS-634 Revisit RonDB restore process in Chef recipes

HWORKS-638 Kafka 3 removed --zookeeper option

HWORKS-639 Template cert store passwords in onlinefs-kafka.properties

HWORKS-643 Revoke Glassfish internal certificate when restoring a backup

HWORKS-653 Publish hopsworks-service-discovery module on archiva

HWORKS-654 extra constraints migration does not respect the order of the columns

HWORKS-655 elkadmin certificate is not generated when restoring a backup

HWORKS-657 Setting current_version attribute with patch version will template all the DDL SQL files

HWORKS-660 Consul recipe should install bc package

HWORKS-661 Incorrect glassfish certificates after upgrade to HA environment

HWORKS-663 remote_file does not support not_if condition

HWORKS-666 Configuring HA Glassfish fails when importing certificates on worker nodes

HWORKS-670 Set Flink job staging and checkpoint directories per user

HWORKS-674 Feature store job migration doesn't work if the spark user is not spark

Task

HWORKS-217 Alerting needs to update alertmanager config in local file system

HWORKS-218 Fix upload for clustered payara

HWORKS-225 Create Deployment Groups in payara5 (clustering)

HWORKS-226 python api should throw informative error if user tries to download a folder

HWORKS-258 Add hopsworks python sdk dataset upload/download to workflow testing

HWORKS-266 Checking out a branch that exist on remote repo does not work

HWORKS-284 Documentation to export cluster logs

HWORKS-314 Add support for git fetch and reset

HWORKS-409 cleaner timer bean are too verbose and pollute logs

HWORKS-415 Add project name to the audit logs

HWORKS-422 Exclude old versions of the documentation from being indexed by search engines

HWORKS-428 Add sleep to loadtest to avoid running into issue due to parallelizing test

HWORKS-437 Upgrade Kafka to version 3.4.0

HWORKS-446 Fix programmatic timers for HA

HWORKS-447 Merge ObjectMapper singletons

HWORKS-448 Fix singletons with concurrency management

HWORKS-449 Fix cache singletons for HA

HWORKS-452 Fix persistent timers for HA

HWORKS-486 Remove hdfs_inodes foreign key from Project table

HWORKS-487 Remove hdfs_inodes foreign key from Dataset table

HWORKS-491 Replace hsml with hopsworks in the code snippets shown in the UI

HWORKS-494 Support local script path in model.deploy()

HWORKS-503 Remove inode foreign key from git repositories

HWORKS-508 Remove DML undo files

HWORKS-510 Archive hops-util Java

HWORKS-511 Add number of GPUs to Python resource usage

HWORKS-512 Docker cgroup parent should be configurable

HWORKS-515 Remove inode foreign key from feature store code

HWORKS-516 Integration test VMs not shut down properly when test is aborted or fails early

HWORKS-520 Make oauth claims configurable

HWORKS-523 Remove inode foreign key from transformation_function table

HWORKS-526 Remove inode foreign key from validation_report table

HWORKS-527 Fix Jupyter JWT manager for HA

HWORKS-529 Upgrade base environment to Python 3.10

HWORKS-530 Kafka advertise listeners should use consul domain name

HWORKS-531 Memory alerts for HMS and HS2

HWORKS-536 Move master encryption password to KMS

HWORKS-542 Docker Library Installer refactor

HWORKS-548 Remove inode foreign key from training datasets

HWORKS-551 UserDTO polluting logs

HWORKS-552 Add org.apache.commons:commons-pool2:jar:2.6.2 to Spark classpath

HWORKS-555 Jupyter on Hopsfs-mount

HWORKS-556 Remove bijection-avro_2.12 - CVE-2017-15288

HWORKS-557 Send alert through Alertmanager if project creation fails

HWORKS-560 Add an alert when approaching the maximum number of shards in a cluster

HWORKS-564 Bump Grafana version to address security vulnerabilities

HWORKS-565 Hopsworks api dataset upload should support parallel chunk upload

HWORKS-573 Only set python engine in project.get_feature_store() if connected to app

HWORKS-574 Remove hdfs_inodes FK from hdfs_command_execution

HWORKS-575 Remove reusing load/workflow test environment from run script

HWORKS-584 Metrics exception due to uninitialised executor pool

HWORKS-585 Change the download url for maven in karamel-chef

HWORKS-586 Upgrade great expectations to 0.14.13

HWORKS-587 Remove inode foreign key from feature views

HWORKS-588 Remove inode foreign key from external feature groups

HWORKS-592 Documentation for enabling hopsfs mount for Jupyter

HWORKS-603 Switch to using libmamba solver in Anaconda

HWORKS-607 Remove inode foreign key for statistics

HWORKS-620 Bump Hops version to 3.2.0.10-RC0

HWORKS-623 Bump hops-git dev version to 0.5.0

HWORKS-628 Skip project creation failure alert if a project with the same name exists

HWORKS-648 Change default ports for the alertmanager

HWORKS-649 consul should log to a file instead of stdout/systemd

HWORKS-650 onlinefs should not log to stdout

HWORKS-651 Airflow should not log to stdout

HWORKS-658 Hops certificate localization directory should not be at /tmp

HWORKS-659 PID files should not be under /tmp

HWORKS-664 Hive temporary directories should be under java.io.tmpdir and not under /tmp

HWORKS-667 MySQL X plugin socket should not be at /tmp

HWORKS-668 Release new version of Karamel

Feature Store

Bug

FSTORE-647 GCP connector path issue

FSTORE-683 Editing BigQuery (and maybe other connectors) It's not possible to set arguments field to ‘null’/‘undefined’/[]

FSTORE-719 Kafka connectors with plaintext as security protocol can't be retrieved by hsfs

FSTORE-752 Online type consistency issue

FSTORE-798 Beam integration tutorial improvement

FSTORE-800 Kafka Topic creation times out after 3 seconds

FSTORE-801 Onlinefs should not try and write rows to tables that have been deleted

FSTORE-812 Enable Kafka topic auto creation

FSTORE-814 StackOverflowException if you provide statistics columns in the statistics config

FSTORE-816 Add Kafka Datasource Jar file to spark deployment

FSTORE-843 Online-enabled External Feature Groups are offline after .save

FSTORE-844 Shade avro dependency in hsfs

FSTORE-846 Feature Group commit_details fails in certain cases

FSTORE-847 Spark Pandas UDFs not working on 3.1 and non-cloud installations

FSTORE-853 Flyingduck HSFS Client: Support Features without Featuregroup

FSTORE-854 Flyginduck HSFS Client: Support datetime objects for date filters

FSTORE-857 Appending to an online feature group fails after upgrade

FSTORE-863 Spark integration documentation doesn't have correct hive metastore jar option

FSTORE-864 Downgrade avro version in hsfs to 1.8.2

FSTORE-890 Statistics are not filtered properly for row percentages other than 1

FSTORE-894 mkdocs-minify-plugin missing from setup.py

FSTORE-902 Recomputing training dataset statistics should overwrite existing statistics

FSTORE-910 OnlineFS set NaN to null

FSTORE-912 Use a recursive approach to infer schema of pyarrow struct

FSTORE-917 Update metadata for expectation suite expect decamelize dict

FSTORE-918 Ensure alert are sent only once

FSTORE-919 Dataframe row upload to kafka progress bar is broken on hsfs master

FSTORE-922 Python join deserialization reads join_type instead of type

FSTORE-929 Enable self join in REST API

FSTORE-934 Flying Duck / HSFS - JVM Error Investigation

FSTORE-936 Canary NPE when ingesting

FSTORE-955 get_batch_data failed when there is no training dataset

FSTORE-962 Topics for feature store do not account for default replication configuration

FSTORE-963 feature group expectation_suite setter doesn't handle properly ExpectationSuite type objects

FSTORE-965 fastavro fails to build when installing HSFS

Subtask

FSTORE-808 Statistics API new schemas, CRUD and filtering

FSTORE-859 Loadtest and integration tests for Statistics API

FSTORE-868 Support for Feature View statistics based on both commit times and event times

Design Log

FSTORE-625 Design Log for FM persistence

FSTORE-733 Write initial design document for feature store REST API

Task

FSTORE-544 Flying Duck v1

FSTORE-574 streaming feature computation pipeline example using Flink

FSTORE-577 Add options to AWS S3 storage connector

FSTORE-626 Boilerplate FM persistence

FSTORE-674 Modify project and FG alert to add new status for FM

FSTORE-675 Hopsworks backend for FM alert

FSTORE-684 Job scheduling PoC

FSTORE-690 Setup Online Feature Store Benchmark with Locust

FSTORE-712 Add Feature Monitoring Job to FsJobManagerController

FSTORE-713 Add endpoint to get feature monitoring config by name + enable/disable

FSTORE-721 Sketch prototype run_feature_monitoring job

FSTORE-736 Export Consumer Group status from onlinefs to prometheus for onlinfs health monitoring

FSTORE-737 Show archived commits in Feature Group activity page

FSTORE-740 Support for Date type columns as primary key on online storage

FSTORE-744 Flying Duck - HSFS Integration

FSTORE-748 Flying duck: Monitoring

FSTORE-749 Move OnlineFS Kafka offsets in RonDB

FSTORE-750 Feature flag for Feature Monitoring

FSTORE-751 Flink support for HSFS java client

FSTORE-768 Integrate scheduler with Feature monitoring service

FSTORE-769 Input Validation and Exception in Feature Monitoring

FSTORE-771 UX proposal for python client feature monitoring

FSTORE-778 Add more type hint to hsfs

FSTORE-779 Simplify OnlineFS

FSTORE-802 Resolve OnlineFS memory consumption

FSTORE-815 Auto-reset Feature Group Kafka offset after topic recreation/upgrade/restore

FSTORE-817 Integrate alerts in feature monitoring

FSTORE-821 Add feature store name to feature store table for REST API

FSTORE-829 Test ASOF Joins in DuckDB

FSTORE-835 Investigation: HSFS Pandas 2.0 Compatability

FSTORE-836 HSFS "Querying..." Printline

FSTORE-837 Flyingduck Karamel Integration

FSTORE-838 Asnychronous fg.insert

FSTORE-841 Relax confluent-kafka dependency to support python 3.10

FSTORE-848 Add support for transformation function to Feature Monitoring

FSTORE-849 Support Spine Dataframes for batch data and training datasets

FSTORE-850 DBT tutorial with BigQuery

FSTORE-851 Increase Kafka List Topics Timeout in HSFS

FSTORE-852 Increase Flyingduck Python Version

FSTORE-855 HSFS Client: Query object .get_feature(), features that returns all features, not just the left side of join

FSTORE-858 Pandas 2.0 Compatibility

FSTORE-861 hsfs client should fetch external broker address for spark/pyspark

FSTORE-867 End to end workflows feature monitoring.

FSTORE-869 Fix typo data validation doc

FSTORE-871 Round event_time to hour in monitoring

FSTORE-873 Add fetch dataset stats support based on specific id + bug fix

FSTORE-874 Notebook feature monitoring demo

FSTORE-875 Design Log External FG Access in Python/Flyingduck

FSTORE-886 Persist feature view statistics computed on both transformed and untransformed data

FSTORE-889 Documentation for Flink/Beam compute engines and feature engineering

FSTORE-893 Get hopsworks certificates for external Flink client

FSTORE-911 Failed to ingest nullable int/long data

FSTORE-913 Add support for MEDIUM/LONG BLOB/TEXT

FSTORE-914 Test HA Hackathon Ticket

FSTORE-915 Flying duck not being used for non-admin users

FSTORE-931 Add support for creating feature group metadata through save method

FSTORE-935 add return type to get_feature_vector(s)

FSTORE-945 Bump OnlineFS version to 1.6

FSTORE-947 Tutorials Streamlit Fix

FSTORE-953 Add feature store REST API doc

v3.2.0

1 year ago

Release notes - Feature Store - 3.2.0

Epic

FSTORE-615 HSFS JavaDoc

Bug

FSTORE-22 Failed to insert data to streaming FG

FSTORE-313 Throw intelligible error message when calling save on unsupported external feature groups in python engine

FSTORE-456 Validation time in the future

FSTORE-516 the 'transformationFunction" fields empty in feature_view.json()

FSTORE-540 Fix shared fs integration tests

FSTORE-549 Uploading a large number of rows to onlinefs fails with TOPIC_AUTHORIZATION_FAILED

FSTORE-566 Feature store online metrics page returns dashboard not found

FSTORE-602 get_or_create_feature_group does not validate feature group name

FSTORE-607 Investigate hsfs java resource leaks

FSTORE-636 ExpectationSuite sonarkube smartUpdate transaction

FSTORE-639 Pin SQLalchemy version or upgrade to new 2.0.0 version

FSTORE-653 get_feature_vector is broken

FSTORE-660 Docs: Syntax issue for creating external feature group

FSTORE-661 Creating external feature group in Jupyter Notebook didn't work

FSTORE-682 Fix fg save_expectation_suite

FSTORE-689 Shallow dataframe copy hides TZ bug

FSTORE-693 Insert with validation suite throws date format exception

FSTORE-701 Fix loadtest datavalidation

FSTORE-730 Fix Multiple backfill job started in dataval loadtest

FSTORE-760 Fix the printing of log in the stack trace for loadtest jobs

FSTORE-763 Onlinefs handling avro exception after append

FSTORE-776 Floating Point Issue with Train Test Split Ratios

FSTORE-780 Documentation doens't mention type limitation for event_time

FSTORE-781 Documentation for extenral feature groups still refers to them as on-demand feature groups

FSTORE-794 Offline fg backfill fails due to the DelataStreamer config Class path(s) in HudiEngine.java are not set correctly

FSTORE-804 Resolve OnlineFS getAndCastValue NullPointerException

FSTORE-812 Enable Kafka topic auto creation

FSTORE-814 StackOverflowException if you provide statistics columns in the statistics config

Design Log

FSTORE-559 Test connection feature for storage connectors

FSTORE-572 Design Log: HSFS Javadocs API

FSTORE-609 Design Log for FM Extend Alerts API

FSTORE-610 Design Log for FM UI / UX

Task

FSTORE-41 From the storage connector; there is no way to know if you are actually connected

FSTORE-44 From a notebook the connector does not say either if it worked or failed

FSTORE-343 Separate HSFS java client build for different Frameworks

FSTORE-349 Add advanced tutorials to documentation

FSTORE-353 Extend the README.md files for the electricity prices and NYC Taxi tutorials

FSTORE-373 Remove job when a fg or fv is deleted.

FSTORE-405 Add entity type to fs job name

FSTORE-434 Improve Kafka message delivery callback error handling

FSTORE-483 Hiveserver2 query optimization config

FSTORE-487 Handle each feature group in its own retry loop

FSTORE-490 Add more assert to Feature Store load test

FSTORE-491 Backfilling user story Design Document

FSTORE-492 Change notebooks that use sklearn models to XGBoost and save them as JSON files, instead of as pickled python files

FSTORE-493 Add load test for online feature store to test data integrity and volume

FSTORE-495 CItibike Tutorial

FSTORE-506 update hopsworks tutorials for 3.1 APIs

FSTORE-528 Add single expectation API and 3.1 changes to data validation tutorials

FSTORE-546 Update several Tutorials to 3.1 hopsworks API

FSTORE-547 Update Tutorials to 3.1

FSTORE-554 cleanup training dataset statistics

FSTORE-557 Simplify getRowClass method

FSTORE-558 Attach transformation function to self join query

FSTORE-561 Add load test for concurrent creation and insertion of online feature groups

FSTORE-562 Add load test to integrate benchmark for vector lookup

FSTORE-568 Add meaningful exceptions to the tutorials, that use API keys

FSTORE-569 New version of Bitcoin Tutorial

FSTORE-570 Design log: Kafka Authoriser Update

FSTORE-571 3.0 Some tutorials Fix

FSTORE-573 documentation/guide about on-demand feature computation pipeline

FSTORE-575 write streaming feature computation pipeline example

FSTORE-581 Review feature view documentation

FSTORE-588 Kafka Authoriser Update

FSTORE-589 Document behaviour of get_feature_vector(s) for missing PK values

FSTORE-591 Fix for hops-kafka-authorizer Sonarqube issues

FSTORE-597 Fix for online-fs Sonarqube issues

FSTORE-611 Investigate how to visualise Feature Monitoring

FSTORE-613 Make Kafka Topic partition default configurable for APP upgrade

FSTORE-614 Auto Kafka Topic recreation needs to provide better error message in Hopsworks for APP upgrade

FSTORE-617 Fix java docstrings to build javadocs

FSTORE-618 Expand current java docstrings to cover new hsfs functionalities

FSTORE-619 make the changes to the github action, pom.xml and mkdocs.yml to publish javadocs

FSTORE-621 Make unit tests run on hsfs branch-3.1

FSTORE-623 Flying Duck: Fix integration into deployment process

FSTORE-624 Add examples from hops-examples repo to hopsworks-tutorials repo

FSTORE-627 Create dockerfile and python entrypoint script-snowflake connector

FSTORE-628 Hopsworks backend logic, bash script to start the container and REST API-snowflake

FSTORE-629 Add CI/CD, set up Jenkins to deploy docker image to docker registry and pull on new clusters-snowflake

FSTORE-658 Incomplete doc string of create_external_feature_group

FSTORE-666 Fix modeling notebook in Bitcoin Price Prediction tutorial

FSTORE-667 Rerun and Refactor all tutorials

FSTORE-669 Enable GCS and BigQuery storage connector feature flags by default

FSTORE-670 Add provenance workflow tests

FSTORE-678 Flying Duck - Logging / Monitoring Configuration in Server

FSTORE-679 Upgrade Air Quality Tutorial

FSTORE-680 Optimize HSFS for single row writes

FSTORE-681 Fix blockers in hopsworks noticed by sonarqube

FSTORE-685 Improvments to onlinefs

FSTORE-687 Pin numpy in nightly-build

FSTORE-694 Users should have control over the size of the connection pool for online feature view

FSTORE-695 Flying Duck - Integrate Monitoring in Server and Chef

FSTORE-697 Handle timezones in validation report timestamp

FSTORE-698 Write Hudi performance best practices guide

FSTORE-704 Flying Duck - Hudi Table Reader Implementation

FSTORE-706 HSFS java make FeatureGroupBase into abstract class

FSTORE-708 Flying duck: Logging

FSTORE-714 Bump onlinefs version

FSTORE-724 Separate workflow and load tests

FSTORE-726 when computing pandas statistics check if mean is numeric type

FSTORE-735 Kafka authorizer Sonarqube fix

FSTORE-739 Add batch scoring examples to all non-online tutorials

FSTORE-741 Resolve outstanding onlinefs sonarqube issues

FSTORE-743 Add online support for External Feature Groups

FSTORE-757 Investigate Describe requests on all kafka topics by users in workflow tests

FSTORE-758 Fix failed workflow test (failed to serialise timestamp)

FSTORE-759 Relax fastavro requirements in hsfs

FSTORE-762 Cherrypick FSTORE-467 to hopsworks 3.0

FSTORE-765 Simplify initialization of OnlineFS

FSTORE-772 BigQuery ExternalFG tutorial

FSTORE-774 Move gcsfs constraint to docker image

FSTORE-775 Improve OnlineFS markdown file

FSTORE-777 Remove MapValue from onlinefs

FSTORE-782 Allow resetting kafka offsets for stream feature groups after backup/restore

FSTORE-786 Pin PyMysql version in hsfs to < 1.0.3

FSTORE-787 FraudBatch Fix

FSTORE-788 FraudBatch Fix for dev branch

FSTORE-789 multi modular hsfs java client

FSTORE-795 Provide flink dependencies in Hopsowrks deployment and not in hsfs.

FSTORE-805 Bump OnlineFS version to 1.5

FSTORE-811 Bump python version in testconnector image

FSTORE-818 Flying Duck - HSFS Cloud Integration

FSTORE-820 FlyingDuck - Memory-Aware Result Streaming

FSTORE-822 Flying Duck - HSFS External Client Integration

FSTORE-831 Upgrade citibikes tutorials

Release notes - Hopsworks - 3.2.0

Bug

HWORKS-29 Remove "Logical Clocks" from the feed back page

HWORKS-315 Copy and move get interrupted if run for too long

HWORKS-320 Pypi library versions not sorted correctly

HWORKS-360 Upgrade org.owasp.dependency-check-maven to 7.4.4

HWORKS-368 Fix serving tests

HWORKS-372 explicit provenance missing community stereotype

HWORKS-379 LDAP users cannot download certificates

HWORKS-380 PKI.java class contains resource leaks

HWORKS-381 hopsworks-api connection.close() should clean up hsfs and hsml connections

HWORKS-382 Provenance input links are missing for models trained on python kernels

HWORKS-390 app_id field in execution table is too short

HWORKS-407 Setting the maximum number of Projects per user to 0 let the user create unlimited Projects

HWORKS-410 Pin jupyter-console in Sparkmagic

HWORKS-414 dockerfile:experimental should be pushed to our docker registry

HWORKS-425 Update Grafana to address CVE-2022-39328, CVE-2022-39307 and CVE-2022-39306

HWORKS-431 Git image version shoud not be hardcoded in Hopsworks settings

HWORKS-444 Log aggregation reports incorrect failed status under load

HWORKS-445 Increase polling time for log aggregation to finish in execution.wait_until_finished()

HWORKS-481 hopsworks-api create_job fails on app_path missing for flink jobs

HWORKS-483 Add CREATING status for deployments

HWORKS-484 Consecutive python uninstall commands uses incorrect tag

HWORKS-507 kagent_param for kube hopsmon certificate break the deployment

HWORKS-524 HSML ignores knative_domain parameter

Subtask

HWORKS-370 Upgrade TensorFlow to 2.11.0

HWORKS-373 Bump ray[serve] dependency in kserve package

HWORKS-400 Upgrade Jupyter(Lab) to address CVE-2021-32797

HWORKS-430 OnlineFS CVEs

Task

HWORKS-126 Add idempotent methods for creating deployments

HWORKS-219 Fix Proxy Servlet to use consul instead of localhost

HWORKS-220 Fix zip/unzip for clustered payara

HWORKS-253 Refactor loadtest project creation

HWORKS-313 Include model.deploy() method in the docs

HWORKS-333 Fix container vulnerabilities

HWORKS-352 Make model input_example consistent across data types

HWORKS-361 Increase TransactionInactiveTimeout to 30000

HWORKS-362 Update spark to depend on latest version of hopsfs

HWORKS-363 Bump Hops version to 3.2.0.9-SNAPSHOT on hopsworks master

HWORKS-364 Update flatbuffers dependency in Hive to address CVE-2020-36632

HWORKS-366 Remove unused ClusterUtili price multiplicator singleton

HWORKS-367 create zip/unzip job

HWORKS-378 Metrics for consul nodes

HWORKS-391 Add utility function to create directory using hopsworks python SDK

HWORKS-394 Remove cron job to copy logs to HopsFS

HWORKS-395 Remove admin UI

HWORKS-398 OpenSearch Dashboard - elastic cluster status is wrong

HWORKS-399 Improve project admin rest api

HWORKS-402 Add monitoring rules

HWORKS-403 Parallelize workflow and load tests

HWORKS-405 Installing library using hopsworks python SDK should wait for ongoing environment sync operation

HWORKS-411 CVE-2023-25194 - Kafka Connect Worker Vulnerability

HWORKS-416 Upgrade data science libraries in Project environment

HWORKS-420 Remove links from images

HWORKS-423 Fix links for public slack on docs and community edition

HWORKS-435 Enable ACL for Zookeeper

HWORKS-436 Helper for loadtest to show execution logs in case of failure

HWORKS-441 Bump RonDB version to 21.04.10

HWORKS-442 Add log aggregation load test

HWORKS-443 Disable Hiveserver2 Web UI

HWORKS-451 Upgrade payara to 5.2022.5 to fix CVE-2022-42920

HWORKS-455 Remove old tours

HWORKS-457 Create users_groups view outside flyway

HWORKS-462 Remove Gemfile.lock file from community tests

HWORKS-463 Bump activerecord dependency in hopsworks-IT to 6.0.5.1

HWORKS-464 Update Python job documentation to reflect the fact that a K8s cluster is needed

HWORKS-467 Disable SSL/TLS Diffie-Hellman keys less that 2048 bits

HWORKS-470 Bump NVIDIA driver version to support CUDA 11.8

HWORKS-471 Remove usage of docker/dockerfile:experimental

HWORKS-472 Workflows/load tests python dependencies should be defined in a requirements.txt

HWORKS-474 Remove hdfs_user foreign key from JupyterProject

HWORKS-476 Remove hdfs_user_id foreign key from TensorBoard

HWORKS-478 Add webhook configuration support for alertmanager

HWORKS-480 Remove inode foreign key from storage connectors

HWORKS-485 Do not create linux user/group in git container

HWORKS-488 Add support for storing docker registry artifacts on S3

HWORKS-497 Remove homebew dependency from java cookbook

HWORKS-502 Add silence for CVE-2023-28462

HWORKS-505 Upgrade GPU components to support Cgroups v2/Rhel8/Ubuntu22.04

HWORKS-506 Bump Hops version to 3.2.0.9-RC0

HWORKS-513 Add MLOps Dictionary link to documentation

HWORKS-514 loadtest should run in its own anaconda environment

v3.1.0

1 year ago

Release notes - Feature Store - 3.1.0

Epic

FSTORE-472 Metadata Management

FSTORE-474 Feature Flags

FSTORE-475 Storage Connectors

Task

FSTORE-33 Improve HSFS parameter description

FSTORE-35 Permission matrix

FSTORE-42 Not very helpful doc "you can do it via the UI" (for external FG)

FSTORE-43 Documentation sample data has syntax issue (missing comma)

FSTORE-52 Fix typo in GE tutorial

FSTORE-309 Add support for like operator in query filter

FSTORE-311 Support datetime64[ns, TZ] typed columns

FSTORE-315 Upgrading GE version to 0.14.12 to support Jinja2 version 3.0.0

FSTORE-316 Add Timestamp/Timezone Guide

FSTORE-322 Enable update of feature view metadata in python client

FSTORE-325 Make driver field optional in Redshift connector configuration

FSTORE-328 throw intelligible error message if user set primary and partition keys doesn't exist

FSTORE-330 Adapt python client to the change introduced for the single expectation api

FSTORE-332 implement method for get_or_create_feature_view

FSTORE-334 explicit provenance - feature groups, feature views, training datasets

FSTORE-336 Improve error message in get_batch_data

FSTORE-337 Single expectation API & Expectation Suite Smart Update

FSTORE-339 Add Great Expectations Validation Activity

FSTORE-340 Add validation for great expectations kwars for feature names

FSTORE-341 Validation History for an expectation

FSTORE-342 Add java tests to github pipeline

FSTORE-345 Update documentation to reflect supported methods in hsfs engines

FSTORE-346 Fail to create feature view with query having `as_of`

FSTORE-347 Make documentation build strict

FSTORE-348 Refactor Tutorials documentation page

FSTORE-350 Clean up Key Files for Storage Connectors on Delete

FSTORE-352 Clean up hudi delta streamer job config files in resources

FSTORE-354 Write Home Credit Default Risk hopsworks tutorial

FSTORE-358 Add icon to external links in documentation navigation

FSTORE-359 Add return types to transformation functions in 3.1

FSTORE-360 Add "Client install guide" to docs

FSTORE-364 Fail to run DeltaStreamer job

FSTORE-369 Remove feature group details endpoint

FSTORE-371 Cannot modify a FG on the UI

FSTORE-379 Add documentation on how to share a feature store store and how to access it.

FSTORE-380 Expand query documentation

FSTORE-381 Label does not consider prefixed feature name

FSTORE-385 Handle same instance of transformation function for multiple features properly

FSTORE-387 Add feature_store_id and feature_group_id in expectation suite DTO

FSTORE-397 Fail to get batch query in java client

FSTORE-398 Add validation history method to python client

FSTORE-406 Document time related parameter in feature view

FSTORE-407 API Doc should include full package name for the RestAPIError

FSTORE-408 Fix feature group integration tests

FSTORE-411 Sharing featurestore fails

FSTORE-416 Problem with get_training_data

FSTORE-422 PIT join does not return event time column of right feature group

FSTORE-425 throw meaningful error if user tries to set time_travel_format=None in a Python kernel

FSTORE-429 Add data validation tutorial to loadtest

FSTORE-430 Add return type and code example in feature view python client

FSTORE-431 Fix typo in user guide for data validation

FSTORE-433 Accept list in event_time

FSTORE-441 Enable shortening of expectation_config field in ValidationResult

FSTORE-442 Increase kwargs varchar field in expectation table

FSTORE-446 user guide update for 3.1 data validation

FSTORE-447 Improve documentation for Tags

FSTORE-448 Feature view documentation problems

FSTORE-451 Bump onlinefs version to 1.4-SNAPSHOT on master

FSTORE-452 Show entire stack trace when exception in engine occurs

FSTORE-460 The code part of the Data Validation is confusing

FSTORE-467 Make it easier to start the backfill job from python

FSTORE-470 Add tests for running workflows with hudi commits

FSTORE-477 Change Online FS ALTER table algorithm

FSTORE-478 Add workflow tests for pyspark fraud batch and fraud online

FSTORE-481 Unify python and pyspark test code in load tests

FSTORE-482 Workflow test for onlinefs append feature

FSTORE-484 Storage connector feature flag

FSTORE-485 Update Hive Grafana dashbaord to reflect new GC configuration

FSTORE-488 Ensure type consistency between write/read/online/offline python/spark

FSTORE-489 Bump spark driver memory to 2GB and executor memory to 4GB

FSTORE-494 add documentation how to add new filters and/or join new FGs to feature view query

FSTORE-496 Test improvements

FSTORE-497 Change integration test after validation time in future bug fix

FSTORE-498 Throw error when filter feature not exist

FSTORE-499 add python code snippets to API docs

FSTORE-505 Remove old feature store tour files

FSTORE-509 Update expectation suite metadata in backend on setter call.

FSTORE-510 Implement reliable ingestionResult for Validation Report

FSTORE-513 Make .parquet file default format for training datasets

FSTORE-515 Prevent deleting Expectation Suite activity on deletion of the suite

FSTORE-519 Redshift storage connector documentation is out of sync

FSTORE-523 Improve user messages for forbidden actions.

FSTORE-526 Don't cast all types when reading from feature view

FSTORE-529 Documentation for Permission matrix

FSTORE-530 Add override logic to save_expectation_suite

FSTORE-541 Investigate missing data in onlinefs load test

FSTORE-542 Add server version check to python client to make correct expectation suite call

FSTORE-543 Add get_batch_data and get_feature_vector functions to workflows

FSTORE-545 Add 3.1 new data validation methods to loadtest

FSTORE-548 Remove deprecated np. types from tests and type conversions

FSTORE-554 cleanup training dataset statistics

FSTORE-555 Add explicit provenance documentation

FSTORE-556 Add examples about include_primary_key and include_event_time

FSTORE-562 Add load test to integrate benchmark for vector lookup

FSTORE-567 Fix data validation loadtest to avoid uploading reports with same timestamp.

FSTORE-579 Tutorials 3.1 Update

FSTORE-581 Review feature view documentation

FSTORE-589 Document behaviour of get_feature_vector(s) for missing PK values

FSTORE-600 Fix training_dataset creation in Electricity project

FSTORE-606 Upgrade Fraud Batch/Online Tutorials

FSTORE-613 Make Kafka Topic partition default configurable for APP upgrade

FSTORE-621 Make unit tests run on hsfs branch-3.1

FSTORE-658 Incomplete doc string of create_external_feature_group

FSTORE-662 Deployment error. 3.1.

FSTORE-669 Enable GCS and BigQuery storage connector feature flags by default

FSTORE-700 Improve Bitcoin Tutorial

FSTORE-726 when computing pandas statistics check if mean is numeric type

Bug

FSTORE-4 FGs with 3 rows - when creating a TD, get an error creating statistics

FSTORE-5 Snowflake connector documentation assumes using the old classic snowflake UI

FSTORE-7 Deleting a Storage Connector does not work

FSTORE-8 Snowflake connector: trying to read with no query throw an error

FSTORE-10 Snowflake storage connector Warehouse is NOT optional

FSTORE-11 Feature group create documentation points to dev (3.1.0-SNAPSHOT)

FSTORE-13 Fraud Online Tutorial Step 7: Create Training Dataset. FeatureStore exception

FSTORE-15 Weight & Biases Colab failed

FSTORE-16 Filtering null values on a decimal feature failed

FSTORE-20 UI changes iris ML tutorial

FSTORE-21 Cannot create streaming feature group without primary key

FSTORE-23 Hopsworks Tutorials - Colabs

FSTORE-24 Fraud retail Colab failed

FSTORE-25 Query difference between external/internal Hive clients

FSTORE-27 Failed to get training data on a feature view if multiple splits with non standard names

FSTORE-29 Problem with Online Feature Group creation

FSTORE-31 Fraud Online Tutorial error

FSTORE-32 Data Validation with GE Tutorial

FSTORE-34 Weights & Biases Tutorial

FSTORE-36 Documentation not updated for train_validation_test_split method

FSTORE-37 api-key documentation broken image and screenshots from old UI

FSTORE-40 Creating online feature without online type

FSTORE-45 Missing entries when uploading large number of rows

FSTORE-53 apply_transformation_function provided Training Dataset instead of Transformation Functions

FSTORE-56 Fail to delete a FV in clients if some FG are deleted.

FSTORE-312 Multiple minor documentation issues in Redshift Feature Ingestion Tutorial

FSTORE-314 Fix td -> fv migration

FSTORE-318 Provenance NullPointerException for a feature group

FSTORE-319 replace license for ImportJobConf to Hopsworks instead of LC

FSTORE-320 Feature group documentation in feature group list points to nowhere

FSTORE-323 Resolve OnlineFS ClusterJ Session usage issue

FSTORE-326 Bigquery connector removed after updating it through the UI

FSTORE-327 Incorrect filling of default kwargs in great expectations table

FSTORE-329 CommitDTO type field is not returned anymore after Payara5 merge

FSTORE-331 BigQuery connector doesn't work if the application is executed on multiple nodes

FSTORE-333 Featurestore insert in Hudi table does not support certificate rotation

FSTORE-351 epipe - provenance index - handle resource folders correctly

FSTORE-366 Shorten result field is throwing error

FSTORE-370 NPE when time_travel_format=None

FSTORE-372 User Guide typos

FSTORE-376 Error in getting batch query when label is the only column selected from a feature group.

FSTORE-377 Corrupted FGs in Hopsworks after deleting a FG

FSTORE-378 Fixing pre-registered great expectations to fix the UI

FSTORE-383 Feature View method documentation

FSTORE-388 Weird values in Online FS, after using .append_feature

FSTORE-389 Activity logs reports activities from other users

FSTORE-390 Exception thrown when query feature group with appended features with default value

FSTORE-391 Ingesting dataframe with nullable=False column causes Exception in Hudi/Hive/Spark

FSTORE-392 Change GCS authentication properties

FSTORE-393 Fix bug introduced by single expectation API

FSTORE-395 great_expectation_engine in python client called with outdated arg list

FSTORE-396 Registered Great Expectations Kwargs set in json should be list

FSTORE-400 Integration Tests: trainingdataset_spec

FSTORE-401 Integration Tests: featuregroup_spec

FSTORE-402 Integration Tests: feature_store_activity_spec

FSTORE-403 Integration Tests: featurestore_code_spec

FSTORE-410 create feature view join.prefix is NoneType

FSTORE-412 json ignore on `isTimeTravel()` in Query.java

FSTORE-413 query prefix throws TypeError

FSTORE-414 feature_view.create_train_test_split always returns empty df

FSTORE-417 Problem with create_train_validation_test_split

FSTORE-418 Write options have no effect with .insert method in Python engine

FSTORE-420 Feature Group insertion job from Python does not apply spark resources correctly

FSTORE-423 expectation meta field not parsed on init

FSTORE-426 Errors/out of date info in the Data Validation documentation

FSTORE-427 Reading training data from HopsFS stopped working with pyarrow 10.0.0

FSTORE-432 Fix event_time in hopsworks-tutorial

FSTORE-435 Fetching report which have results with deleted expectation throws null pointer exception.

FSTORE-436 HSFS Java doesn't serialize JSONs correctly

FSTORE-437 Key error 'type' in inode.from_response_json

FSTORE-438 Methods get_train_validation_test_split and train_validation_test_split not returning same column type

FSTORE-443 Do not allow sending feature group feature offline type to be null

FSTORE-444 Data validation minor bug fixes for python client

FSTORE-445 Minor fix to tutorials

FSTORE-453 Fix convert_event_time_to_timestamp for datetimes with millisecond precision (hudi timestamps)

FSTORE-454 Show warning in output when data validation fails, right now it silently moves on

FSTORE-455 Default great expectation configuration not working

FSTORE-456 Validation time in the future

FSTORE-457 Data Validation code example is broken

FSTORE-459 Wrong FS name in API example for sharing FG

FSTORE-461 Append feature default value not working for online feature store

FSTORE-463 Import data to feature group job failure

FSTORE-465 Can't assign filter to feature view query

FSTORE-468 Cannot set timestamp type as partition_key in feature group

FSTORE-476 Schema not updated for hsfs java client after append

FSTORE-480 Create new Dynamic Object for each row in transaction

FSTORE-501 Can't insert streaming dataframe in streaming feature group using `insert_stream` method

FSTORE-507 figure out features and their transformation functions when attaching transformation functions with prefixes

FSTORE-508 NPE when creating feature view with labels

FSTORE-512 Typo in expectation_type leads to failure to execute validate job

FSTORE-518 Problem with Feature Groups Join

FSTORE-520 Feature view - explicit provenance if parent feature group is deleted

FSTORE-522 HSFS get redshift storage connector fails

FSTORE-527 Fix deletion issue with expectation suite due to activity table constrain

FSTORE-532 Redshift connector NPE when using Instance Role for authentication

FSTORE-534 Redshift Connector IAM Role authentication jackson issue with expiration field

FSTORE-538 Redshift connector read throws validation error for query argument inspite of table name specified on connector

FSTORE-551 Fix loadtest incorrect assert

FSTORE-552 Fix validate for pyspark when providing pandas dataframe

FSTORE-560 In onlinefs configuration, Hopsworks domain name is not configured by chef

FSTORE-639 Pin SQLalchemy version or upgrade to new 2.0.0 version

FSTORE-640 Fix QuickStart and Credit_Scores tutorials

FSTORE-653 get_feature_vector is broken

FSTORE-657 in dev branch tutorials imports are missing

FSTORE-660 Docs: Syntax issue for creating external feature group

FSTORE-661 Creating external feature group in Jupyter Notebook didn't work

FSTORE-663 Model training tutorial:Error on predictions after model deployment

FSTORE-682 Fix fg save_expectation_suite

FSTORE-689 Shallow dataframe copy hides TZ bug

FSTORE-691 Train test splits contain NaN column values

FSTORE-693 Insert with validation suite throws date format exception

FSTORE-730 Fix Multiple backfill job started in dataval loadtest

Release notes - Hopsworks - 3.1.0

Bug

HWORKS-1 Documentation link at /account/api should point to https://docs.hopsworks.ai/3.0/user_guides/projects/api_key/create_api_key/ and not https://docs.hopsworks.ai/3.0/user_guides/integrations/databricks/api_key/

HWORKS-5 When adding member to project it is always set to data scientist

HWORKS-11 Github documentation expect you to know the platform

HWORKS-26 Clone repo documentation could be more hand holding

HWORKS-36 Deactivate the "Save" button on "Edit profile" page if there are no changes

HWORKS-39 python env - wrong docs link

HWORKS-45 Confusing paragraph in jobs and git

HWORKS-47 Python and Docker job should show reason why it crashed also when not caused by the application program

HWORKS-93 Fix serving tests

HWORKS-94 Upgrade from 3.0.0 to master fails when applying dml sql files

HWORKS-96 Feature store connection does not reset when calling hopsworks.logout()

HWORKS-97 Ensure a default value for serving_tool in new deployments

HWORKS-101 Models saved with the latest version of joblib (1.2.0) cannot be loaded in Hopsworks

HWORKS-110 Link to Kibana logs from deployment page is wrong in App

HWORKS-114 The password recovery email send to the old ui

HWORKS-115 Deploying cluster with Managed Kubernetes fails

HWORKS-116 Cluster deployment fails when overriding Hopsworks port in the cluster definition

HWORKS-117 Doc Bug: Auth Login Page

HWORKS-118 Fix failing git tests

HWORKS-122 Revoking certificate with exact X509 Name does not work

HWORKS-123 hopsworks.login() should hide API Key after it was supplied as input

HWORKS-140 Kafka authorizer incorrectly parses x509 Subject

HWORKS-143 .condarc still templating deleted attribute

HWORKS-150 Fix failing serving tests

HWORKS-154 Execution success variable not set correctly

HWORKS-156 Deleting projects with certain names may delete hopsworks docker images from the registry

HWORKS-158 Stuck in project creation when creating a project with the same name as a recently delete project

HWORKS-159 Payara5 fixes for hopsworks-api

HWORKS-160 Can't open airflow if payara is not restarted.

HWORKS-168 Customize consul health check error exit code

HWORKS-175 Make SELinux disabling and modifying resolv.conf optional in consul-chef

HWORKS-184 onlinefs username and monitoring port not templated in onlinefs-site.xml

HWORKS-185 Template Kafka listeners even if IDs are explicitly set

HWORKS-186 Fix blocker bugs reported by sonarqube analysis

HWORKS-194 Kafka external listener port collides with alertmanager

HWORKS-203 dnsmasq systemd unit is not enabled by default in CentOS

HWORKS-205 Fix RESTCodes compilation

HWORKS-208 mgm-server-start check for existing port is too wide

HWORKS-211 Deleting a python environment in a cluster with managed kube can delete another project images.

HWORKS-212 ExperimentResultsDTO throws IllegalAnnotationException

HWORKS-214 Reorder operations to start docker

HWORKS-242 Deployment fails to start in a non-kubernetes cluster

HWORKS-249 redshift connector save fails for default driver

HWORKS-250 HSML 3.0.* does not fully work with Hopsworks 3.1

HWORKS-254 onlinefs user is hardcoded in Hopsworks

HWORKS-260 Update LDAP configuration via the rest api not working

HWORKS-268 Karamel fail during dnsmasq installation

HWORKS-271 hopsify may time out when submitting a CSR

HWORKS-273 Docker containers total memory seems wrong on grafana dashboard

HWORKS-295 Project creation error message not informative

HWORKS-296 Spark job resource docs are wrong

HWORKS-297 Add retries to package Chef resource

HWORKS-301 Connection reset is not handled by LdapRealm injected resource

HWORKS-332 LDAP Group Project mapping UI changes the name of the ldap group when the mapping is selected

HWORKS-340 Member is removed from Project in LDAP auth where no LDAP groups Project mappings are defined

HWORKS-350 Add numpy constraint to KServe Python library

HWORKS-356 HDFS copyFromLocal and chown operations eventually fail during zipping/unzipping

HWORKS-357 Fix upgrade from Glassfish 4 to Glassfish 5

HWORKS-365 Remove hardcoded yarnapp user in jupyter notebook convert script

HWORKS-371 Prometheus database alerts do not trigger if the metric is absent

HWORKS-372 explicit provenance missing community stereotype

HWORKS-383 Documentation does not mention python kernel feature flag

HWORKS-410 Pin jupyter-console in Sparkmagic

HWORKS-425 Update Grafana to address CVE-2022-39328, CVE-2022-39307 and CVE-2022-39306

Subtask

HWORKS-400 Upgrade Jupyter(Lab) to address CVE-2021-32797

Task

HWORKS-2 API Keys guide should link to hopsworks login

HWORKS-8 Which permission for the github token

HWORKS-34 Make git repositories private

HWORKS-37 git workflow docs

HWORKS-71 Increase DockerResourcesConfiguration to 2GB

HWORKS-82 Code snippet in docs should use hopsworks.login() and not hopsworks.connection()

HWORKS-95 Show resource usage in JupyterLab

HWORKS-121 Skip adding application certificates to CRL

HWORKS-125 Improve status management for deployments

HWORKS-132 Push onlinefs docker image to internal docker registry

HWORKS-134 All integration tests should use expect_status_details instead of expect_status

HWORKS-136 Docker Cgroup cpu.cfs_quota_us should account for number of cores

HWORKS-137 Add chef parameter to overwrite location of /tmp directory

HWORKS-141 Add sample yaml for Alertmanager config

HWORKS-144 Bump RonDB version to 21.04.9

HWORKS-145 Remove ROCM Code

HWORKS-146 Docker image anaconda python environment should use python 3.8

HWORKS-155 Mention hops python library as deprecated from 3.0 in docs

HWORKS-161 Add final status to Python Jobs

HWORKS-162 Don't show "service user" role if it's not app

HWORKS-163 Hopsworks python jobs api should not check if the job already exists

HWORKS-165 Disable ee_ml_pipeline spec

HWORKS-166 conda-chef ignores tmp_directory attribute when installing miniconda

HWORKS-167 Remove dev/SNAPSHOT version from hsfs, hsml and hopsworks API docs

HWORKS-169 Bump version of commons-text due to CVE-2022-42889

HWORKS-171 model version should be downloaded in temp directory instead of current working directory

HWORKS-172 Do not overwrite mysql server log files

HWORKS-173 Explicitly set JAVA_HOME when calling hash.sh in elastic-chef

HWORKS-190 Hopsworks-chef should allow installation of custom certificate for Hopsworks UI

HWORKS-191 Run unit tests, license checker and vulnerability checker on pull request review

HWORKS-192 Fix master to use latest version of HopsHadoop 3.2.0.8-SNAPSHOT

HWORKS-199 Should be possible to get python library version using .__version__

HWORKS-201 Investigate git clone failing on multi-VM setup

HWORKS-202 Add project under creation flag

HWORKS-204 Support for installing .whl and requirements.txt using hopsworks python SDK

HWORKS-207 Add Docs on how to configure LDAP from Payara UI

HWORKS-210 jupyter threads fill up executor pool

HWORKS-213 hopsworks python SDK API doc code samples should use hopsworks.login()

HWORKS-223 Remove username prefix from cloned git repository folder name

HWORKS-243 Documentation on how to export audit logs to third party platform

HWORKS-244 Documentation on how to export metrics to third party platform

HWORKS-245 Documention that new github fine-grained tokens are not supported yet

HWORKS-256 [hopsworks-chef] Deploy user supplied intermediate certificate for Glassfish

HWORKS-257 Improve data validation testing in loadtest

HWORKS-259 Sync ldap groups upon user login

HWORKS-263 Make LDAP groups Hopsworks Project synchronization more granular that hour

HWORKS-289 Add enable_git_pull_only flag

HWORKS-290 Add enable_data_science_profile flag

HWORKS-291 Update opensearch version to 1.3.6

HWORKS-292 Apply SPARK-40801 to address CVE-2022-42889

HWORKS-293 Add cookbook flag to enable remote JMX

HWORKS-299 Add enable_jupyter_python_kernel_non_k8s flag

HWORKS-303 Grafana should show all resource managers not only active ones

HWORKS-310 Disable TLS1.1 for Opensearch dashboard

HWORKS-311 Disable TLS1.0 and TLS1.1 for Docker registry

HWORKS-312 Disable insecure chipers in Opensearch

HWORKS-318 Apply HIVE-24636 to remove the memory leak

HWORKS-322 CVE-2022-1471

HWORKS-324 .hw_api_key should be saved in home folder, or fall back to temp

HWORKS-326 Update and fix prometheus alerting rules

HWORKS-327 Drop commons-net dependency

HWORKS-330 Create a cluster overview dashboard in Grafana

HWORKS-335 Fix failing tests

HWORKS-339 Ensure the hopsworks tutorials work with 3.1 release

HWORKS-341 Add missing Audience.JOB annotation for full hopsworks API compatibility

HWORKS-342 Check major/minor version compatibility on hopsworks.login()

HWORKS-345 Switch Livy to use log4j 1.2 bridge

HWORKS-347 Configure Spark to use Log4j2 configuration

HWORKS-349 hsml v3.1.0 fails when backend is 3.0.4

HWORKS-353 In case of an upgrade initialize Hopsworks CAs in hopsworks-ca instead of Expat

HWORKS-361 Increase TransactionInactiveTimeout to 30000

HWORKS-362 Update spark to depend on latest version of hopsfs

HWORKS-393 Enable HSFS in default deployments (without KServe)

HWORKS-413 Description for model evaluation metrics is missing

HWORKS-423 Fix links for public slack on docs and community edition

v3.0.0

1 year ago

Release notes - Hopsworks - Version 3.0.0

Bug

HOPSWORKS-3255 elastic search - highlight fragment bug

HOPSWORKS-3253 Feature Group Type Enum breaks with Stream Feature Group after upgrade

HOPSWORKS-3252 Disable Kafka Topic deletion for feature group ingestion topics

HOPSWORKS-3246 New kafka topic names not included in the response after updating a deployment

HOPSWORKS-3236 Improve model inference error responses

HOPSWORKS-3235 Use Hudi snapshot query when users doesn't specify as_of in the query param

HOPSWORKS-3233 Timestamp incompatibility Spark/Hive/Hudi

HOPSWORKS-3232 Deleting a model used in existing deployments should not be possible

HOPSWORKS-3228 Bump scikit-learn to v1.1.1

HOPSWORKS-3227 Return valid inference endpoints only

HOPSWORKS-3221 Spark dist files do not respect HopsFS replication factor

HOPSWORKS-3220 HopsKafkaAuthorizer retry and refresh ACLs cache if cannot find any ACLs

HOPSWORKS-3212 Remove pickle file validation in storage-initializer

HOPSWORKS-3209 expectation_suite update is failing

HOPSWORKS-3207 Unexported service log viewer Chef attributes

HOPSWORKS-3204 Feature View can't get query when one of the feature groups was deleted

HOPSWORKS-3197 Feature Store quota enforcement fail with NPE for non-cached Feature Groups

HOPSWORKS-3193 Creating Feature Group from UI without Expectation Suite fails

HOPSWORKS-3191 OAuth not removing group if mapping is empty

HOPSWORKS-3187 Inference with Istio using HSML not enabled in transformers

HOPSWORKS-3185 Api key scopes do not cascade on delete

HOPSWORKS-3183 Online feature vectors are not sorted according to the user input

HOPSWORKS-3171 Rename PYTHON scope to PYTHON_LIBRARIES

HOPSWORKS-3149 Consul should only register the active resource manager

HOPSWORKS-3142 Transformation function does not handle null value

HOPSWORKS-3141 Query filter does not handle DATE type properly

HOPSWORKS-3133 Hopsify log file is in wrong directory

HOPSWORKS-3129 Consul does not detected a failed MySQLd

HOPSWORKS-3126 Should flush new Serving entity to get id

HOPSWORKS-3124 ruby test - always wait for started executions to stop

HOPSWORKS-3121 [FeatureView] Bug report

HOPSWORKS-3111 Hopsworks K8s certificate expiration should not be hardcoded

HOPSWORKS-3097 Make OAuth provider display name should not be mandatory

HOPSWORKS-3093 FileSaver.js package was outdated from Teleborder repository

HOPSWORKS-3090 CVE-2022-26612 - flink client vulnerability

HOPSWORKS-3087 Feature group data preview returns 500 when previewing streaming feature groups

HOPSWORKS-3070 Install custom PyTorch and Torchvision in base python environment

HOPSWORKS-3059 Latest version for monitored library should be null if not available

HOPSWORKS-3058 Hopsfs-mount chown's root directory ownership during upgrade

HOPSWORKS-3048 Python library installation using APIKey fails to get the user of the request

HOPSWORKS-3038 Add lock management in Python search singletons

HOPSWORKS-3031 Uglify failing hopsworks-web build

HOPSWORKS-3019 Filebeats data directory is wrong

HOPSWORKS-3015 flyway validation_rule table broken for 2.3 -> 2.4 -> 2.5 upgrade

HOPSWORKS-3013 Storage Connector Update not working

HOPSWORKS-2994 ModuleNotFoundError when saving code

HOPSWORKS-2993 Update git provider secrets throws an secret already exists exception

HOPSWORKS-2958 Hopsify should use internal https port to register with Hopsworks

HOPSWORKS-2957 Call hops::default from hopsworks::default

HOPSWORKS-2953 init_prepared_statement throws NoneType exception

HOPSWORKS-2952 update hsfs examples with train_split property

HOPSWORKS-2564 Alert manager config backup timer fails

HOPSWORKS-2100 Cannot enable model serving when Models folder already exists

HOPSWORKS-1932 feature store sharing ignores online feature store

Improvement

HOPSWORKS-3251 Add numFeatureViews to FeaturestoreDTO for front-end

HOPSWORKS-3248 Rename On Demand Feature groups to external feature groups in hsfs

HOPSWORKS-3242 Improve handling of different data types in feature groups across online/offline

HOPSWORKS-3240 Stream Feature Group online enable exception only when trying to change the setting

HOPSWORKS-3231 Support custom external IP for istio ingressgateway

HOPSWORKS-3226 Hops Kafka Authorizer improve synchonization

HOPSWORKS-3225 Throw a warning instead of error while creating empty data featuregroup

HOPSWORKS-3222 Improve Logstash CPU utilization

HOPSWORKS-3217 Set default values for Inference batching and add access to Hopsworks var for Job audience

HOPSWORKS-3216 hops-hadoop-chef hops/nn/heap_size should set only the NN heap

HOPSWORKS-3211 Add endpoint for deployment logs

HOPSWORKS-3206 bump hopshadoop version 3.2.0.7 RC0

HOPSWORKS-3195 Refactor inference endpoints details

HOPSWORKS-3190 Upgrade OpenSearch to 1.3.3

HOPSWORKS-3186 [FeatureView] Feature view training dataset API redesign

HOPSWORKS-3184 feature view search

HOPSWORKS-3181 OAuth update user profile not working

HOPSWORKS-3176 Change the format of the Hopsworks audit logs to JSON.

HOPSWORKS-3173 Kafka store number of partitions and replicas in the database

HOPSWORKS-3169 [hops-hadoop-chef] Expose attribute to configure user limits in Yarn Capacity Scheduler

HOPSWORKS-3161 Hive connection should return better error message when accessing an unshared feature group

HOPSWORKS-3158 Print Job and Feature Store URLs within Hopsworks with correct Hostname

HOPSWORKS-3154 Make nodemanager-graceful-decommission-timeout-secs configurable

HOPSWORKS-3147 Make Kafka clusterinfo endpoint return also the external kafka broker endpoints

HOPSWORKS-3146 Fix delete user end-point

HOPSWORKS-3136 Migrate IAM role chaining docs

HOPSWORKS-3131 there can be only one fg.insert() or fg.save()

HOPSWORKS-3128 Iam role mapping fixes for new UI

HOPSWORKS-3125 [FeatureView] Merge feature view dev branch to master

HOPSWORKS-3123 [FeatureView] remove label from feature view table

HOPSWORKS-3119 Register OAuth user as admin

HOPSWORKS-3118 Create new role for hopsworks as a service user

HOPSWORKS-3114 Enable HTTP 1.0 in istio ingress gateway

HOPSWORKS-3113 Enable Hive session for all Jobs

HOPSWORKS-3112 Documentation for managing services

HOPSWORKS-3109 Allow creation of AWS storage connector without IAM role passthrouh

HOPSWORKS-3104 Fix error messages for services ui

HOPSWORKS-3103 Connect to Hive externally

HOPSWORKS-3102 Support custom domain name in the istio ingress gateway

HOPSWORKS-3100 [FeatureView] Migrate training dataset to feature view

HOPSWORKS-3099 [FeatureView] Java client

HOPSWORKS-3098 [FeatureView] Python client

HOPSWORKS-3096 Multiple expectation suite + remove great expectation matching

HOPSWORKS-3091 Allow users to configure clusterj connection pool/max transactions in onlinefs

HOPSWORKS-3086 change hopsfs small files default to disabled

HOPSWORKS-3084 Add Documentation for Okta group claim

HOPSWORKS-3081 Make OAuth redirect uri provider specific

HOPSWORKS-3079 Support fixed node port for istio-ingressgateway service

HOPSWORKS-3078 Refactor tags sub-resource

HOPSWORKS-3072 CVE-2021-43090

HOPSWORKS-3071 allow utf-8 chars for user names/surnames

HOPSWORKS-3069 Allow StreamFeatureGroup in backend to be available offline only

HOPSWORKS-3056 Add hopsworks to monitored libraries

HOPSWORKS-3055 Add chef attribute to limit the size of datanode storage

HOPSWORKS-3052 Add a flag to skip namespace creation when creating a project

HOPSWORKS-3051 Add new Python extra to HSFS for python engine to replace hive

HOPSWORKS-3047 Add chef attribute to control resource manager HTTPs ports

HOPSWORKS-3046 Enable TLS by default

HOPSWORKS-3042 Kerberos fixes for new UI

HOPSWORKS-3041 Bump up RonDB version to 21.04.4

HOPSWORKS-3040 Use informers in the model-serving-authenticator server

HOPSWORKS-3036 Kerberos and LDAP Docs

HOPSWORKS-3034 Move azure OAuth2 doc from cloud to admin

HOPSWORKS-3030 Add hopsworks debug flag

HOPSWORKS-3028 Extend JWT token lifetime

HOPSWORKS-3016 Improve Logstash Resource Utilization Configurability

HOPSWORKS-3014 [consul-chef] Attribute to change Consul client bind interface

HOPSWORKS-3012 Add dummy file in HopsFS for External Training Datasets to support Keywords/Tags

HOPSWORKS-3011 Decrease logstash pipeline workers for each pipeline

HOPSWORKS-3008 Upgrade Fink to 1.14.3 release

HOPSWORKS-3007 Limit TLS versions used by elastic

HOPSWORKS-3006 Add Expectation Suite Logic between common and persistence layer

HOPSWORKS-3005 Duplicate Expectation persistence logic to ExpectationSuite objects

HOPSWORKS-3004 Duplicate Expectation API logic to ExpectationSuite objects

HOPSWORKS-2999 HSFS Hive engine should throw proper errors for all unsupported methods

HOPSWORKS-2998 Hopsfs-mount fails in git container on VM with TLS disabled

HOPSWORKS-2996 Add git integration documentation

HOPSWORKS-2995 Add job configuration documentation

HOPSWORKS-2992 Jupyter update endpoint should update the entire settings

HOPSWORKS-2985 Add ssl_verify flag in condarc file

HOPSWORKS-2974 HSML support in non-kubernetes installations

HOPSWORKS-2970 Fix dependency-check

HOPSWORKS-2966 LDAP and Kerberos fixes for new UI

HOPSWORKS-2960 Migrate User management docs

HOPSWORKS-2956 Move/clean up hive certificates when using HSFS external client

HOPSWORKS-2955 default train_split property to "train"

HOPSWORKS-2949 Add Client IP and User Agent to the Hopsworks audit logs

HOPSWORKS-2921 Migrate Alert docs

HOPSWORKS-2918 Register built-in transformations by defaut for every project in backend

HOPSWORKS-2897 Remove old feature store UI

HOPSWORKS-2832 Support serving custom predictor scripts without model files

HOPSWORKS-2715 Support for 'requested' and 'limited' pod resources configuration

HOPSWORKS-2543 add support to insert_stream() to ingest data into offline fs

HOPSWORKS-2496 Split up JDBC Storage Connector arguments from string into class attributes

HOPSWORKS-2032 Remove unused parameters for small files from hops-hadoop-chef

New Feature

HOPSWORKS-3215 Make Proxy use JWT

HOPSWORKS-3214 [hops-hadoop-chef ]fix wrong param name in hdfs-site.xml

HOPSWORKS-3203 Add training dataset support for GCS connector

HOPSWORKS-3199 Add loging page overwrite option

HOPSWORKS-3120 Quotas enforcement

HOPSWORKS-3054 Stream Feature Group Python Engine ingestion

HOPSWORKS-3010 Add Google BigQuery Connector

HOPSWORKS-2988 REST API Great Expectations DTO

HOPSWORKS-2987 Add create GE store method to ValidationPipeline class

HOPSWORKS-2986 Data Validation Hopsworks3

HOPSWORKS-2947 [FeatureView] Implement PrepareStatementResource

HOPSWORKS-2946 [FeatureView] Implement TransformationResource

HOPSWORKS-2945 [FeatureView] Implement activity endpoints

HOPSWORKS-2944 [FeatureView] Implement keyword related endpoints

HOPSWORKS-2943 [FeatureView] Implement QueryResource

HOPSWORKS-2942 [FeatureView] Implement TagResource

HOPSWORKS-2941 [FeatureView] Implement TrainingDatasetResource

HOPSWORKS-2844 Add autoscaling information to the /metrics endpoint

HOPSWORKS-2741 Add Kafka storage connector

Sub-task

HOPSWORKS-1851 Show feature store info in search

Task

HOPSWORKS-3256 Remove old feature store tour job files

HOPSWORKS-3218 Support custom KServe TensorFlow version

HOPSWORKS-3179 Add catboost and xgboost libraries to base environment

HOPSWORKS-3178 Remove Deequ based data validation in favour of GE

HOPSWORKS-3177 Bump confluent-kafka to 1.8.2

HOPSWORKS-3175 [FeatureView] Add expansion on keywords and tags when getting td by version

HOPSWORKS-3174 Bump-up RonDB version to 21.04.6 and Bump Hops version in Chef to use 3.2.0.7-SNAPSHOT

HOPSWORKS-3153 Rebuild Spark 3.1 after hops dependency change

HOPSWORKS-3145 Chef enabled online feature store by default

HOPSWORKS-3085 Suppress CVE-2021-28165 from zookeeper dependency

HOPSWORKS-3083 Add hopsworks python SDK examples

HOPSWORKS-3076 [hops-hadoop-chef] Bump up Hops ndb version

HOPSWORKS-3018 Set executions_per_job_limit variable in tests

HOPSWORKS-2984 Add expansions to Git API

HOPSWORKS-2976 NullPointerException when logging requests containing queryDTOs

HOPSWORKS-2975 Remove feature store integration test spec

HOPSWORKS-2969 Add [rsa] profile to MySQL dependency in hsfs

HOPSWORKS-2881 Clean old Git support

HOPSWORKS-2879 Upgrade hudi to 0.10.0 release

HOPSWORKS-2673 Upgrade TensorFlow to 2.9 and PyTorch to 1.12

v2.5.0

2 years ago

v2.4.0

2 years ago

Release notes - Hopsworks - Version 2.4.0

Bug

HOPSWORKS-2770 Feature Group clear content endpoint rethinking

HOPSWORKS-2748 Kibana and Grafana consul health checks are wrong

HOPSWORKS-2738 Artifact files should not have glassfish as owner

HOPSWORKS-2730 Proxy servlet is leaking resources

HOPSWORKS-2717 Avro fails with schema names starting with numeric characters

HOPSWORKS-2710 Wrong permissions for Airflow subdirectories

HOPSWORKS-2696 Cannot store features of tinyint offline type as avro serialization fails

HOPSWORKS-2682 node_exporter is not added to kagent services file if kagent is not enabled

HOPSWORKS-2669 Nullpointerexception when trying to get an environment that doesn't exists

HOPSWORKS-2665 Use Hopsworks internal port when calling generate_service_jwt in chef

HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

HOPSWORKS-2662 Fix angular-material version

HOPSWORKS-2659 [hopsworks-chef] Chef guard has incorrect syntax

HOPSWORKS-2656 Incorrect payload when creating sklearn servings with new artifacts

HOPSWORKS-2644 Non HOPS_ADMIN users should not be able to access the cluster settings.

HOPSWORKS-2642 EditableSelect listening to keyboard event on window.

HOPSWORKS-2641 Project page not loading after login

HOPSWORKS-2640 Cannot start Jupyter if Kafka is offline

HOPSWORKS-2638 elastic clear cached scrolling contexts

HOPSWORKS-2620 Add shared by and accepted by to dataset sharing requests

HOPSWORKS-2213 feature group commit endpoints doesn't fall back to pagination defaults

Improvement

HOPSWORKS-2768 Add private_ips attribute in consul-chef metadata.rb

HOPSWORKS-2736 Add versioning for KFServing tars in the archive

HOPSWORKS-2731 On-demand feature group should return primary key information

HOPSWORKS-2720 KfServing storage initializer should use Hopsworks internal port

HOPSWORKS-2712 Add support for IN condition in query filter

HOPSWORKS-2709 Add endpoint to retrieve the most recent ran jupyter notebooks

HOPSWORKS-2708 Add option to disable Jupyter shutdown

HOPSWORKS-2701 improve Jobs REST API filters

HOPSWORKS-2695 Replace creator with UserDTO in FeaturestoreEntityDTO

HOPSWORKS-2692 Disable Yarn application retry

HOPSWORKS-2683 Anaconda environment should be exported after every operation

HOPSWORKS-2681 Profiling optimization

HOPSWORKS-2667 Disable projects with no feature store

HOPSWORKS-2661 Sort Feature Group columns by hive metastore index

HOPSWORKS-2660 No user feedback when jupyter is not launched due to cpu/memory constraints

HOPSWORKS-2654 Attaching an expectation without features should try to use all the FG features

HOPSWORKS-2653 Flyway dir already exists error in hopsworks::install

HOPSWORKS-2606 compute training dataset statistics for train,test and validation separately

HOPSWORKS-2598 Add application field in snowflake storage connector

HOPSWORKS-2453 Feature validations - turn on by default

HOPSWORKS-1832 [ePipe] Handle connection errors with NDB

New Feature

HOPSWORKS-2518 Point-In-Time Joins

Task

HOPSWORKS-2765 Remove Sqoop

HOPSWORKS-2754 Bump hops dependencies to 3.2.0.5-RC0

HOPSWORKS-2753 Install ca-certificates in base image as required by nodejs

HOPSWORKS-2737 Pre-install flask in airflow-chef

HOPSWORKS-2723 Remove AUDITOR role

HOPSWORKS-2705 Deterministic UID/GID

HOPSWORKS-2693 Bump minor version of service proxy

HOPSWORKS-2689 Increase max file descriptors for mysql unit

HOPSWORKS-2684 Remove ndb_backup from Hopsworks

HOPSWORKS-2650 Bump hops dependencies to 3.2.0.5-SNAPSHOT

HOPSWORKS-2155 Set up multi-repo documentation for docs.hopsworks.ai

v2.3.0

2 years ago

Release notes - Hopsworks - Version 2.3.0

Bug

HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

HOPSWORKS-2639 NPE with no Databricks Clusters in Feature Store Integration

HOPSWORKS-2634 Feature order not respected in training dataset prepared statement

HOPSWORKS-2631 SYNC_BASE_ENV operation should also show error logs in case of failure

HOPSWORKS-2628 OnlineFS schema of feature groups with appended features misses old features

HOPSWORKS-2626 MySQL timezone issue

HOPSWORKS-2625 Add status to dataset projectsSharedWith

HOPSWORKS-2619 Add missing JVM imports in Livy for Spark 3

HOPSWORKS-2618 NPE when force deleting a Project

HOPSWORKS-2614 Training Dataset from Python with default connector fails

HOPSWORKS-2609 Unshare a pending feature store fails

HOPSWORKS-2594 Attaching Jupyter config when running notebook with Spark or Python kernel with git backend fails

HOPSWORKS-2583 Experiment mnist notebook should work in airgapped environment

HOPSWORKS-2582 Spark driver cores is incorrectly set to number of configured executor cores

HOPSWORKS-2577 Python Feature Group Ingestion fails with additional write arguments

HOPSWORKS-2574 Feature validation rule Level.Warning defaults to Error

HOPSWORKS-2569 Allow capital letters keywords

HOPSWORKS-2567 Nullpointerexception when retrieving validations for on-demand feature groups

HOPSWORKS-2556 Jupyter should use default DockerJobConfiguration for Experiments and Spark

HOPSWORKS-2555 KAFKA_BROKERS not set in Python kernel

HOPSWORKS-2554 oauth2 get user bug

HOPSWORKS-2553 Fix kfserving airgapped deployment

HOPSWORKS-2546 Add RonDB services to kagent even if disabled

HOPSWORKS-2538 Secrets API not setting projectId and visibility in certain cases

HOPSWORKS-2536 Online FS Avro schema constructor decimals should take precision and scale

HOPSWORKS-2526 add fg to prepared statement only if it contains features other than pk and targets.

HOPSWORKS-2520 Installing python libraries with no version and from git should use --no-cache docker build option

HOPSWORKS-2517 Job executions not filtered by submission time

HOPSWORKS-2515 Set spark.serializer to Kryo for Databricks integration

HOPSWORKS-2510 Updating validationType on feature group response contains old value

HOPSWORKS-2503 Pin containerd version in ubuntu

HOPSWORKS-2495 HSFS connection cannot not read JWT in Python jobs

HOPSWORKS-2490 Disable Python Library Updates Monitor for snapshot versions

HOPSWORKS-2386 Decimal types are converted to varbinary on the online feature grouip

HOPSWORKS-2349 Hive InodeHelper doesn't work if path contains slahs (/)

HOPSWORKS-2343 Validate storage connector type before creating a training dataset

Improvement

HOPSWORKS-2653 Flyway dir already exists error in hopsworks::install

HOPSWORKS-2645 Support feature validation for on-demand FGs in hsfs

HOPSWORKS-2643 Add option to enable hopsfs/rm audit logs

HOPSWORKS-2636 Add option to specify Kubernetes node labels from cluster definition

HOPSWORKS-2615 Add option to redirect stdout and stderr of Docker jobs

HOPSWORKS-2607 Set NotValidBefore of issued X.509 a few minutes earlier

HOPSWORKS-2605 Configuration parameter for Kubernetes maximum retry number

HOPSWORKS-2604 add option to specify additional insecure docker registry

HOPSWORKS-2597 Add hops Python helper methods for model and dataset modules

HOPSWORKS-2584 Increase Schema Registry database schema column size

HOPSWORKS-2578 Use a separate Data directory to store Hopsworks state

HOPSWORKS-2575 Make Online FS Kafka topic partition number configurable

HOPSWORKS-2560 Support Memory, Cores and GPU as a resource for model serving

HOPSWORKS-2559 Add input paths option for Docker jobs and multiple commands

HOPSWORKS-2552 Send monitoring URLs as part of execution dto

HOPSWORKS-2551 Bump hops dependencies to 3.2.0.4-SNAPSHOT

HOPSWORKS-2529 Package dockerfile:experimental in the base.tar for

HOPSWORKS-2528 Add support to provide PIP extra-index-url in the cluster definition

HOPSWORKS-2523 Use own java cookbook instead of upstream

HOPSWORKS-2506 Feature Validation Improvements

HOPSWORKS-2502 Add JOB audience to delete endpoint for training datasets

HOPSWORKS-2492 Improve readiness check of KFServing inference services

HOPSWORKS-2356 Alerting for hopsworks services

HOPSWORKS-2167 Add inference logging sidecar to KFServing deployments

New Feature

HOPSWORKS-2381 Project level configuration for Jupyter notebooks and Jobs

HOPSWORKS-2334 Support custom transformers in KFServing deployments

HOPSWORKS-2319 Add API to interact with the secret store

HOPSWORKS-2104 Model Serving with KFServing

Task

HOPSWORKS-2637 Bump hops dependencies to 3.2.0.4-RC0

HOPSWORKS-2622 pin six to 1.15.0

HOPSWORKS-2621 Bump pip in hops-system to last version that supports python 2.7

HOPSWORKS-2601 Update swagger code to be compatible with the latest swagger hub and enable for demo cluster

HOPSWORKS-2498 Remove filebeat beam recipes

HOPSWORKS-2424 Attach transformation function to training dataset

HOPSWORKS-2420 Bump hops dependencies to 3.2.0.3-SNAPSHOT

HOPSWORKS-1942 Upgrade to Spark 3.x