What this PR does / why we need it:

This PR adds a Document Embedder capability to Feast, allowing users to go from raw documents to embeddings stored in the online vector store in a single step. It handles chunking, embedding generation, and writing the results to the online store — providing an end-to-end ingestion pipeline for RAG workflows within Feast.

What changed:

sdk/python/feast/chunker.py

Defines the document chunking layer. Provides:

• A BaseChunker abstract class that users can extend with custom chunking strategies
• A built-in TextChunker that splits plain text by word count with configurable size, overlap, and thresholds
• A chunk_dataframe() convenience method to chunk all documents in a DataFrame

Currently only basic text chunking is implemented. There is room for improvement — future iterations can support more advanced strategies like semantic chunking, sentence-aware splitting, or format-specific chunkers (PDF, HTML, etc.).

sdk/python/feast/embedder.py

Defines the embedding generation layer. Provides:

• A BaseEmbedder abstract class with a modality-routing registry for extensibility
• A MultiModalEmbedder with built-in support for text (via sentence-transformers) and image (via CLIP) embeddings
• Lazy model loading and configurable batch processing

sdk/python/feast/doc_embedder.py

The high-level orchestrator that coordinates chunking, embedding, and storage. Provides:

• A DocEmbedder class that runs the full pipeline: chunk -> embed -> logical layer -> write to online store
• A user-defined logical layer function to map output to the FeatureView schema
• Auto-generation of a default FeatureView if the user does not define their own (controlled by create_feature_view). This is a basic implementation intended as a starting point

sdk/python/feast/init.py

Updated to export DocEmbedder, LogicalLayerFn, BaseChunker, TextChunker, ChunkingConfig, BaseEmbedder, MultiModalEmbedder, and EmbeddingConfig as part of Feast's public API.

Which issue(s) this PR fixes:

Create DocEmbedder class along with RAGRetriever #5426

Misc