Rapidata

No one actually knows if a model is good. OpenAI just released their new 4o image model (wtf is that naming, just call it Dall-E 4 already) The only way to truly judge a genAI model is with shit tons of humans. Luckily I am in the "shit ton of humans" business. In just one day we collected data from 200'000 humans on how this new models stacks up. We present the first ever independent large scale benchmark of the new model! They crushed! Black Forest Labs Flux model has finally been dethroned!

🚀 First Benchmark of OpenAI's 4o Image Generation Model! We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive! In our tests, OpenAI 4o image generation absolutely crushed  leading competitors, including Black Forest Labs, Google, xAI, Ideogram, Recraft, and DeepSeek AI, in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark! The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images: - 5 hours to generate 1000 images (no API available yet) - Just 10 minutes to set up and launch the benchmark - Over 200,000 responses rapidly collected While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the OpenAI 4o model proactively refused to generate certain images: 🚫 Styles of living artists: completely blocked 🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!

Perhaps this shows that there is a space for more specialized AI models next to foundation models.

OpenAI Is Falling? The Unthinkable Shift in AI Dominance It happened in the span of a single hour: over 30,000 human annotations poured in, revealing the truth ... The Runway’s Gen3 Alpha are now outperforming OpenAI’s own Sora in style and coherence But ... what does this really mean? For an AI model to rise to the top, it needs more than brilliant engineering and computational power Most critically is access to high-quality data which is currently a bottleneck in AI, as even OpenAI Co-Founder Ilya Sutskever said ... Even though Harvard University’s Institutional Data Initiative (IDI) has released a trove of nearly 1 million public-domain books, it might not be enough to fully satisfy the ever-growing hunger for diverse Yes, this helps, but how much? Rapidata helps AI companies to laverage more than 25M human anotators, delivering answers and insights within minutes. As AI companies scramble to refine their models, Rapidata’s on-demand data annotation could be the key to unlocking new levels of performance And with the AI landscape shifting faster than ever, whoever masters the data pipeline may well decide the fate of AI’s next chapter. Hold on tight ...

We benchmarked xAI 's Aurora model, as far as we know the first public evaluation of the model at scale. We collected 401k human annotations in over the past ~2 days for this, we have uploaded all of the annotation data on Hugging Face with a fully permissive license https://lnkd.in/eFjsXX8h

Three important datasets recently shared on the @huggingface Hub: **Bespoke-Stratos-17k: Reasoning Distillation at Scale** from Mahesh (Maheswaran) Sathiamoorthy What: A synthetic reasoning dataset (17k examples) with questions, reasoning traces, and answers. Built by refining the Berkeley Sky-T1 pipeline using DeepSeek-R1. Why It Matters: Powers two high-performing models (32B https://lnkd.in/e4NfEABj and 7B https://lnkd.in/et23TwHB) that outperform predecessors like Sky-T1-32B on benchmarks like MATH500 and LiveCodeBench. Innovation: Generated in 1.5 hours using the Bespoke Curator https://lnkd.in/eKZYzz8K tool, with a 73% retention rate for correct solutions via GPT-4o-mini filtering. Showing again that data is 🔑 🔗 https://lnkd.in/eMC_wCTY **Rapidata Video Generation Preference Dataset** from Rapidata What: AI-generated video dataset with Likert scale ratings (1-5) from ~6,000 human evaluators assessing visual appeal. Evaluators rated from "Strongly Dislike" (1) to "Strongly Like" (5), blind to generation prompts. Why It Matters: Creates benchmark for human preference data in video generation models. Scores incorporate evaluator userScore for weighted ratings. Innovation: Rapid data collection via Rapidata Python API. Videos accessible as full MP4s under Files with GIF previews in dataset viewer. Open API enables scalable annotation projects. 🔗 https://lnkd.in/eq9PqW9y *Polite Guard: Politeness Classification Benchmark* from Intel Corporation What: 100k synthetic samples (50k few-shot, 50k Chain-of-Thought) plus 200 annotated corporate training samples for classifying text into polite, somewhat polite, neutral, and impolite categories. Why It Matters: First benchmark for context-aware politeness classification. Provides defense against adversarial inputs and enhances customer service AI interactions. Innovation: Multi-model generation using Llama 3.1, Gemma 2, and Mixtral. Complete reproducibility with code, training pipeline, and fine-tuning guide at https://lnkd.in/egZgZH5k. 🔗 https://lnkd.in/e5-S2SNb

What a view ... (Trending page of Huggingface Image Datasets)

Massive human feedback dataset for text-to-image models from Rapidata - 1.5M human responses from 152K participants - Evaluates image coherence, style & prompt alignment - Includes detailed error heatmaps - Covers DALL-E, Midjourney, Imagen outputs Dataset available on Hugging Face Hub

Our vision seems aligned with one of the brightest minds in AI (Ilya Sutskever). That is one hell of a stamp of approval (and especially nice if you aren't competing).

🚨 Google crushes the competition: In our updated text-to-image benchmark, Imagen 3 beats ALL other models*** ***at generating a person under a car. In the rest of the examples not so much. We just collected 1,250,000 additional human preferences to include Imagen 3 and Flux1.1 in our leaderboard of best text-to-image AI models. The results show that Google have stepped up their game a bit with Imagen 3, however, they are still behind both the current and last generation of Flux. Interestingly enough, the latest version of Flux, 1.1, also performs worse than the previous generation. Perhaps the model regressed slightly with the improvement in efficiency? We have made all the images and preference data freely available on our Huggingface page - check it out through the links in the comments!