What is Fennel?
Fennel is a platform designed to help users author, compute, store, serve, monitor, and govern both real-time and batch machine learning pipelines. It utilizes a CDC-aware engine built in Rust for automatic incremental computation, optimizing efficiency and ensuring data freshness. The system avoids DSLs, relying instead on standard Python and Pandas, thereby minimizing the learning curve.
Features include automatic backfills, fully-managed infrastructure, and a feature repository for reusability. Fennel also incorporates robust data quality tooling, including strong typing, immutability, versioning, unit testing, compile-time validation, and data expectation checks to prevent errors and maintain data integrity.
Features
- Incremental Computation: Automatically computes only the changes in data, improving efficiency.
- Python-Based: Uses standard Python and Pandas, eliminating the need for DSLs or specialized job configurations.
- Automatic Backfills: Pipelines automatically backfill on declaration.
- Fully-Managed Infrastructure: Manages all necessary infrastructure components.
- Feature Repository: Allows users to write standardized features once and reuse them across different applications.
- Strong Typing: Catches typing bugs at compile time and at the source of data.
- Immutability & Versioning: Features are immutable and versioned to avoid discrepancies between offline and online environments.
- Unit Testing: Supports unit testing for both batch and real-time pipelines.
- Compile Time Validation: Validates lineage end-to-end at compile time to minimize runtime errors.
- Data Expectation: Allows users to define expected data distributions.
- Online / Offline Skew: single definition of feature.
Use Cases
- Developing and deploying real-time machine learning models.
- Managing and monitoring batch processing of machine learning features.
- Creating and maintaining a centralized feature store.
- Ensuring data quality and consistency in machine learning pipelines.
Related Queries
Helpful for people in the following professions
Featured Tools
Join Our Newsletter
Stay updated with the latest AI tools, news, and offers by subscribing to our weekly newsletter.
