refinery
The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like a software artifact.
- 1.5k
- Python
- Apache-2.0
- 24 mo ago
code-kern-ai
Building data-centric open-source tools for NLP
Built from code-kern-ai's public GitHub profile, Eichwalde, Brandenburg. Not on Take My Repo yet, so there are no posts or updates here.
The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like a software artifact.
CLI-based tool to automatically build ML models from training data into a servable Docker container
Since the twitter algorithm has been open-sourced, the issues section of their repository is being polluted. Let's try to fix that.
Containing examples of projects you can use to test refinery. Please select the use case from the branches.
With sequence-learn, you can build models for named entity recognition as quickly as if you were building a sklearn classifier.
With embedders, you can easily convert your texts into sentence- or token-level embeddings within a few lines of code. Use cases for this include similarity search between texts, information extraction such as named entity recognition, or basic text classification.
With weak-nlp, you can integrate heuristics like labeling functions and active learners based on weak supervision. Automate data labeling and improve label quality.
Neural search for refinery. Manages similarity search powered by Qdrant and outlier detection, both based on vector representations of the project records.
This is a demo of how we can leverage embeddings and LLMs to quickly build a chatbot so you can "chat" with a documentation.
Data model for refinery. Manages entities and their access for multiple services, e.g. the gateway.
UI for refinery. Used to interact with the whole system; to find out how to best work with the UI, check out our docs.
Tokenizer for refinery. Manages the creation and storage of spaCy tokens for text-based record attributes and supports multiple language models. It is used by the gateway.
Submodule which contains the requirements of the different parent images of refinery.
Gateway for refinery. Manages incoming requests and holds the workflow logic. To interact with the gateway, the UI or Python SDK can be used.
Embedder for refinery. Manages the creation of document- and token-level embeddings using the embedders library.
Configuration of refinery. Manages amongst others endpoints and available language models for spaCy.
Overview of recurring functions you can use for your projects