Python: Introducing Keyword Hybrid Search and Lambda Filters for Azure AI Search #11482
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
markwallace-microsoft
added
python
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
e4baa12 to
9e4b565
Compare
Python Unit Test Overview
|
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
590746f to
bb5d361
Compare
eavanvalkenburg
changed the title
eavanvalkenburg
changed the title
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
a1924ac to
b0f60ae
Compare
roji
reviewed
Apr 15, 2025
python/semantic_kernel/connectors/memory/azure_ai_search/azure_ai_search_collection.py
Outdated
Show resolved
Hide resolved
|
If I understand correctly, there aren't any integration tests for this, right? I'm seeing what looks like unit tests only for now? |
|
@roji indeed int tests still to be done, made a note |
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
a8fc179 to
32aa44c
Compare
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
b201298 to
0b08684
Compare
eavanvalkenburg
force-pushed
the
hybrid_search
branch
2 times, most recently
from
3548400 to
0aa68d5
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR introduces support for keyword hybrid search and adds lambda filter functionality across multiple memory connectors while updating filter property names. Key changes include:
- Adding support for Callable filters (lambda filters) in methods such as _build_filter with corresponding parameter type updates.
- Renaming data field properties from is_filterable/is_full_text_searchable to is_indexed/is_full_text_indexed in connectors and samples.
- Removing obsolete Azure AI Search connector files to consolidate functionality.
Reviewed Changes
Copilot reviewed 52 out of 52 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| python/semantic_kernel/connectors/memory/pinecone/_pinecone.py | Updated filter handling with lambda support and renaming of filter properties. |
| python/semantic_kernel/connectors/memory/mongodb_atlas/* | Updated field property checks and filter dictionary creation. |
| python/semantic_kernel/connectors/memory/in_memory/in_memory_collection.py | Added support for Callable filters and updated error handling for get without keys. |
| python/semantic_kernel/connectors/memory/chroma/chroma.py | Introduced lambda filter support and updated type imports. |
| python/semantic_kernel/connectors/memory/azure_cosmos_db/* | Updated filter property usage and query building logic. |
| python/semantic_kernel/connectors/memory/azure_ai_search/* | Removed obsolete Azure AI Search connector files as part of consolidation. |
| python/samples/concepts/memory/* | Updated sample data model definitions and search/upsert methods to reflect rename changes. |
| python/samples/concepts/chat_history/, python/samples/concepts/caching/ | Updated field attribute names for filtering and indexing. |
Comments suppressed due to low confidence (2)
python/semantic_kernel/connectors/memory/pinecone/_pinecone.py:465
- [nitpick] The variable name 'filter' shadows the built-in filter function. Consider renaming it (e.g. 'flt' or 'built_filter') to avoid potential confusion.
if options.filter and (filter := self._build_filter(options.filter)):
python/semantic_kernel/connectors/memory/azure_ai_search/utils.py:1
- The entire Azure AI Search connector has been removed. Please ensure that any dependent modules or documentation are updated accordingly and that this change is intentional.
# File removed entirely
eavanvalkenburg
force-pushed
the
hybrid_search
branch
from
632489d to
4605301
Compare
TaoChenOSU
reviewed
Apr 18, 2025
| else: | ||
| raise VectorStoreOperationException("No lambda expression found in the filter.") | ||
|
|
||
| def _lambda_parser(self, node: ast.AST) -> str: |
There was a problem hiding this comment.
Will this be used in the future by other memory connectors? Should this live in a common module? Given the complexity of the logic, it may be beneficial to move it to its own module for test purposes.
| ) -> Sequence[Any] | None: | ||
| if not keys: | ||
| if options is not None: | ||
| raise NotImplementedError("Get without keys is not yet implemented.") |
There was a problem hiding this comment.
This is only raised when options is not None and key is empty, but the error message doesn't give the full context.
There was a problem hiding this comment.
This happens in other connectors too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Hybrid
Adds a new abstraction for keyword hybrid search
Implements this for Azure AI Search
Adds search tests for Azure AI Search
Filters
Adds support for a Callable (lambda func) as a filter in VectorSearchOptions
Implements a ast walker to go from a lambda to a string for the filter
Adds a set of test parameter that can be extended
This further does some work on the vector abstractions:
is_filterabletois_indexedandis_full_text_searchabletois_full_text_indexed: Closes Port dotnet feature: .Net: Rename IsFilterable to IsIndexed and IsFullTextSearchable to IsFullTextIndexed #11598local_embeddingfrom embedding field, addshas_local_embeddingto data field, this allows one to create a data model without local embedding, in the old model that required a vector field anyway, now that can be set on the data field that needs to be embedded by the service.does_collection_existsanddelete_collectionmethods to VectorStore: Closes Port dotnet feature: .Net: Add delete and exists methods for IVectorStore #11597Relates to #10561
Description
Contribution Checklist