zer0int
I π CLIP ~ π€π«Άπ€
Built from zer0int's public GitHub profile. Not on Take My Repo yet, so there are no posts or updates here.
Text Encoders finally matter π€π₯ - scale CLIP & LLM influence! + a Nerdy Transformer Shuffle node
Vision Transformers Needs Registers. And Gated MLPs. And +20M params. Tiny modality gap ensues!
For SDXL, SD1.5, Flux. Nuke T5 and let CLIP guide Flux.1 on its own! Or let let random guide Flux.1! Or load a CLIP crazy opinion embedding about your image and let that guide the AI!
CLIP GUI - XAI app ~ explainable (and guessable) AI with ViT & ResNet models
CLIP Interrogator, fully in HuggingFace Transformers π€, with LongCLIP & CLIP's own words and / or *your* own words!
Example scripts for using [my] fine-tuned CLIP models with HuggingFace π€
What do CLIP Vision Transformers learn? Feature Visualization can show you!
Get CLIP ViT text tokens about an image, visualize attention as a heatmap.
Comfy Nodes (and a CLI script) for shuffling around layers in transformer models, creating a curious confusion.
Shuffle LLama-3.2 layers and have it prompt an image. Node works with any model - Flux, SD3, SDXL...
Use CLIP to create matching texts + embeddings for given images; useful for XAI, adversarial training
Shuffle GPT-2's layers and have it prompt an image. Node works with any model - Flux, SD3, SDXL...
Workflows to implement fine-tuned CLIP Text Encoders with ComfyUI / SD, SDXL, SD3
Experimental removal / shuffling of layers in CLIP ViT + Text Transformer
No Hopper architecture (RTX 5090, etc.) required! <16 GB VRAM, Windows.
A simple CLIP tokenizer encode / decode script. For when you need to know CLIP's tokenization and / or token IDs for some reason.
Projected Gradient Descent (PGD), inverted and amplified -> prompt & generate images with CLIP