Skip to content

feat: Implementing online_read for MilvusOnlineStore#4996

Merged
franciscojavierarceo merged 6 commits intomasterfrom
milvus-onlin-read
Feb 1, 2025
Merged

feat: Implementing online_read for MilvusOnlineStore#4996
franciscojavierarceo merged 6 commits intomasterfrom
milvus-onlin-read

Conversation

@franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Jan 31, 2025 •
edited
Loading

What this PR does / why we need it:

This pull request includes several updates and improvements to the Feast project, focusing on the integration and enhancement of the Milvus online store. Below is a detailed summary of the changes made:

  1. Examples Update

    • README.md: Modified the materialization steps for features into the online store by switching to Python code snippets.
    • milvus-quickstart.ipynb: Format change.

  2. Milvus Online Store Enhancements

    • milvus.py:
      • Removed unused imports.
      • Added VALUE_TYPE_TO_PROTO_VALUE_MAP for mapping value types.
      • Introduced a new method _get_composite_key_name to generate composite key names.
      • Updated methods to utilize _get_composite_key_name for consistency.
      • Implemented the online_read method to handle reading from the Milvus online store.
      • Improved the _extract_proto_values_to_dict method to handle more numeric types and serialization.

  3. Type Map Enhancements

    • type_map.py:
      • Added VALUE_TYPE_TO_PROTO_VALUE_MAP for better type conversion.

  4. Feast Types Enhancements

    • types.py:
      • Introduced from_feast_type method to convert Feast types to ValueType.

  5. Example Repository Updates

    • example_feature_repo_1.py:
      • Updated the schema for document_embeddings to include vector index and search metric attributes.

  6. Unit Tests Enhancements

    • test_online_retrieval.py:
      • Modified test_get_online_features for better readability.
      • Added comprehensive tests for Milvus online features retrieval.
      • Added skipping marker for test_local_milvus due to CI issues.

Testing

  • Comprehensive unit tests have been added and modified to ensure the functionality of the changes made.
  • Manual testing was performed to validate the integration with the Milvus online store.

Which issue(s) this PR fixes:

Another for #4364

Misc

  • The test_local_milvus has been marked to be skipped due to CI struggles. Further investigation is needed to address this.

@franciscojavierarceo
feat: Implementing online_read for MilvusOnlineStore
f448b2c 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
updated tests and implementation to handle for missing entities
7aa8e77 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
ignoring local milvus test
28b05c5 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo franciscojavierarceo marked this pull request as ready for review February 1, 2025 01:50
@franciscojavierarceo franciscojavierarceo requested a review from a team as a code owner February 1, 2025 01:50
shuchu
shuchu reviewed Feb 1, 2025
View reviewed changes
lokeshrangineni
lokeshrangineni reviewed Feb 1, 2025
View reviewed changes
sdk/python/tests/unit/online_store/test_online_retrieval.py Outdated
full_feature_names=False,
)

# TODO: Need to fix these tests to actually run corecctly.
Copy link
Contributor

@lokeshrangineni lokeshrangineni Feb 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this needs to revisit now or later?

Copy link
Member Author

@franciscojavierarceo franciscojavierarceo Feb 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was going to handle that in a follow up PR too. Trying to get a release out soon was the main reason.

Copy link
Member Author

@franciscojavierarceo franciscojavierarceo Feb 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

@franciscojavierarceo
adding milvus md and incorporating feedback from Lokesh
edba589 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
including Milvus in summary page
8c605c1 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
@franciscojavierarceo
linter
cb71592 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
shuchu
shuchu approved these changes Feb 1, 2025
View reviewed changes
Copy link
Collaborator

@shuchu shuchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@franciscojavierarceo franciscojavierarceo merged commit 92dde13 into master Feb 1, 2025
25 checks passed
@dmartinol
Copy link
Contributor

dmartinol commented Feb 3, 2025

amazing example notebook, thank you! 💯

franciscojavierarceo pushed a commit that referenced this pull request Feb 4, 2025
@feast-ci-bot
chore(release): release 0.44.0
aa7224a 
# [0.44.0](v0.43.0...v0.44.0) (2025-02-04)

### Bug Fixes

* Adding periodic check to fix the sporadic failures of the operator e2e tests.  ([#4952](#4952)) ([1d086be](1d086be))
* Adding the feast-operator/bin to the .gitignore directory. Somehow it… ([#5005](#5005)) ([1a027ee](1a027ee))
* Changed Env Vars for e2e tests ([#4975](#4975)) ([fa0084f](fa0084f))
* Fix GitHub Actions to pass authentication ([#4963](#4963)) ([22b9138](22b9138)), closes [#4937](#4937) [#4939](#4939) [#4941](#4941) [#4940](#4940) [#4943](#4943) [#4944](#4944) [#4945](#4945) [#4946](#4946) [#4947](#4947) [#4948](#4948) [#4951](#4951) [#4954](#4954) [#4957](#4957) [#4958](#4958) [#4959](#4959) [#4960](#4960) [#4962](#4962)
* Fix showing selected navigation item in UI sidebar ([#4969](#4969)) ([8ac6a85](8ac6a85))
* Invalid column names in get_historical_features when there are field mappings on join keys ([#4886](#4886)) ([c9aca2d](c9aca2d))
* Read project data from the 'projects' key while loading the registry state in the Feast UI ([#4772](#4772)) ([cb81939](cb81939))
* Remove grpcurl dependency from Operator ([#4972](#4972)) ([439e0b9](439e0b9))
* Removed the dry-run flag to test and we will add it back later. ([#5007](#5007)) ([d112b52](d112b52))
* Render UI navigation items as links instead of buttons ([#4970](#4970)) ([1267703](1267703))
* Resolve Operator CRD bloat due to long field descriptions ([#4985](#4985)) ([7593bb3](7593bb3))
* Update manifest to add feature server image for odh ([#4973](#4973)) ([6a1c102](6a1c102))
* Updating release workflows to refer to yml instead of yaml ([#4935](#4935)) ([02b0a68](02b0a68))
* Use locally built feast-ui package in dev feature-server image ([#4998](#4998)) ([0145e55](0145e55))

### Features

* Added OWNERS file for OpenshiftCI ([#4991](#4991)) ([86a2ee8](86a2ee8))
* Adding Milvus demo to examples ([#4910](#4910)) ([2daf852](2daf852))
* Adding retrieve_online_documents endpoint ([#5002](#5002)) ([6607d3d](6607d3d))
* Adding support to return additional features from vector retrieval for Milvus db ([#4971](#4971)) ([6ce08d3](6ce08d3))
* Creating/updating the stable branch after the release. ([#5003](#5003)) ([e9b53cc](e9b53cc))
* Implementing online_read for MilvusOnlineStore ([#4996](#4996)) ([92dde13](92dde13))
* Improve exception message for unsupported Snowflake data types ([#4779](#4779)) ([5992364](5992364))
* Operator add feast ui deployment ([#4930](#4930)) ([b026d0c](b026d0c))
* Updating documents to highlight v2 api for Vector Similarity Se… ([#5000](#5000)) ([32b82a4](32b82a4))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lokeshrangineni lokeshrangineni lokeshrangineni left review comments

@shuchu shuchu shuchu approved these changes

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@franciscojavierarceo @dmartinol @shuchu @lokeshrangineni