Training AI to operate in the physical world requires enormous amounts of data, but many situations autonomous systems need to recognise are rare, dangerous, or hard to capture in the real world.
Hungarian startup DiffuseDrive is tackling that data gap by generating synthetic training data for physical AI systems, helping companies expose their models to scenarios their existing datasets don’t contain.
I spoke with team members Gabor Vecsei, VP of AI Engineering, and Daniel Schmid, AI Research Lead, to learn more.
Vecsei leads the engineering of the company’s AI systems and the transition of research into scalable production technology. Before joining DiffuseDrive, Vecsei spent several years working in machine learning and AI engineering, including leading an ML R&D organisation that grew from a small team into a department of more than 30 people. He studied at the Budapest University of Technology and Economics, specialising in applied computer science.
Schmid is AI Research Lead at DiffuseDrive, working at the intersection of the company’s research and production engineering. Originally from Germany, he has specialised in machine learning since 2020. Before joining DiffuseDrive, he worked on several academic projects, including research experience in Korea and a collaboration with the University of Oxford. He completed his master’s degree at the Technical University of Munich.
Why Physical AI has a data problem
DiffuseDrive aims to solve the problem of data scarcity in physical AI across applications including defence, aerospace, mining, and autonomous vehicles.
According to Schmid, three things are incredibly important for AI: data, data, and data.
“With language models, there is an enormous corpus of textual data available on the internet.
Physical AI is very different. Data is scarce and difficult to collect, and the scenarios you’re interested in can be extremely hard — or dangerous — to reproduce, particularly in defence environments.”
DiffuseDrive has built a system that can generate those missing data points and inject them directly into customers’ datasets, helping improve downstream applications such as autonomous drones, vehicles, and mining systems.
Take a defence company with a perception system monitoring a coastline. Its data and models are inevitably incomplete because simply pointing a camera at the coastline won’t expose the system to every scenario it needs to recognise.
According to Schmid, those rare scenarios are often precisely the ones defence organisations need to be prepared for.
DiffuseDrive’s platform analyses a company’s existing solution and identifies gaps in the data.
It can then generate data to fill those gaps.
“One of the areas where we’ve seen the most success is not only generating the missing data, but helping customers identify which gaps exist in the first place,” explained Schmid.
Finding the edge cases that matter
To ensure a synthetic edge case accurately represents something that could happen in the real world, the company usually starts with the customer.
Vecsei shared:
“They might have 100 images or data points, or perhaps a few thousand, and typically already have a trained neural network that they’ve evaluated. That means they already know some of the core shortcomings of their system.”
DiffuseDrive works with them to understand how the system was trained and what data they have. It analyses part or all of the dataset, identifies what’s there, and extrapolates what might be missing but could occur in the real world.
He explained:
“
There are potentially huge numbers of edge cases, so we have to prioritise. For an autonomous mining application, for example, wildlife or cattle might be among the first things you need to address.”
It’s an iterative process. The platform analyses the current dataset, the downstream task, and its shortcomings. It then generates data addressing those shortcomings, adds it to the dataset, retrains, and repeats that loop for as long as necessary.
To keep generated data realistic and matched to the customer’s domain, DiffuseDrive adapts the system to the customer’s camera characteristics, sensors and data types.
Vecsei explained:
“We actually want some forms of bias — towards a particular sensor family, camera type, or customer requirement — because we’re generating data for a very specific environment.”
For biases coming from the underlying foundation models, the company conducts a pre-analysis and builds mechanisms into the system to mitigate them. It can also use humans in the loop to inspect datasets or automated quality assurance that checks whether generated images or videos meet the requirements provided to the system.
Schmid admits that generative AI can, of course, produce unrealistic scenarios, “so we also have a quality-assurance component that analyses the system’s output.
It determines whether an image is realistic, whether it makes sense, and whether to discard a particular data point.”
Controlling generative AI for real-world edge cases
Another technical challenge for DiffuseDrive is controllability: having precise control over what generative AI produces rather than simply generating a broadly realistic scene.
“If you use a diffusion or generative model out of the box, you have some control over the output, but not necessarily enough for these applications,” said Schmid.
For example, a conventional image-generation model can be prompted to place a balloon in an image. But specifying that the balloon should appear far in the distance and occupy just 10 pixels is much harder.
“We’ve developed a dedicated solution that lets us control where objects appear and their size.”
That level of control becomes important when generating training data for perception systems, particularly in defence applications where an object may be small, distant, or positioned in an unusual part of the camera’s field of view.
“You need a perception system to recognise an object even if it appears far away in the top-left corner of an image.”
From autonomous driving to defence
The company started with autonomous-driving companies because its founders came from Bosch’s vehicle automation division.
However, it then saw growing demand in defence, particularly in markets such as Germany, and realised its system was very well suited to those applications.
The founders started building relationships in the industry, and today they’re working with global automotive suppliers, leading autonomous-systems and defence companies, and some of the world’s largest enterprise technology organisations.
Generating new training data in hours
If a customer discovers a new failure or gap, DiffuseDrive can generate useful training data in hours.
Vecsei explained:
“If we receive the data in an agreed format, the system is ready to go. By that stage, we’ll usually already have a dataset adapted to the customer’s use case, so it largely becomes a question of compute.
If they have a large GPU cluster we can use, it’s straightforward.
If they only have one GPU, we’re constrained by the hardware. But if the infrastructure is there, we’re talking about hours.”
The company has also created an air-gapped system customers can deploy on their own infrastructure, so their machine-learning or data engineers can use it without interacting with the team.
It can also connect to external APIs when required, depending on the customer’s needs.
Does synthetic data actually improve performance?
DiffuseDrive has reported performance gains of more than 10 per cent when synthetic and real data are combined.
Schmid asserts that the biggest value is making customers’ prediction systems more robust and reliable when they encounter edge cases.
“A system might perform well against its test data, but that doesn’t necessarily mean it will perform reliably in the real world. We provide data that helps customers make their systems robust against edge cases they can anticipate.”
However, one of the hardest things to communicate is that data volume doesn’t necessarily correlate with downstream performance.
“I could drive around in my car and collect millions of images, but they might not contain anything remotely interesting for the customer. Filling the gaps is far more important, as is preparing for new scenarios they might not yet be aware of,” shared Vecsei.
Curiously, the company’s biggest competitors are often its own customers. They already have internal workflows and pipelines, so during onboarding DiffuseDrive has to demonstrate why they should do something differently.
“We’re essentially competing against an internal system we can’t see; we can only see its outputs,” explained Vecsei.
Another challenge is evaluation. With the first batch of generated images, people naturally look at them and judge them with the human eye.
“But an image that looks impressive to a human isn’t necessarily the image that’s most useful for training.
We need customers to evaluate the data based on its effect on the neural network or perception system downstream.”
Simulation companies are another competitor. They use physical equations to simulate the real world, but Schmid asserts that those systems don’t necessarily provide the visual realism they can achieve.
“Our goal is to deliver images that look as though they were captured in the real world.”
DiffuseDrive raised $3.5 million in seed funding in 2025. It’s now looking to generate data for robotics companies, a rapidly growing field.
According to Vecsei,
“We want to go beyond generating visual data and also make sense of existing robotic experiments. These episodes may include video, sensor streams and other metadata, but often contain very little information about what is actually happening.
Our goal is to understand those recordings and produce dense annotations capturing the actions, events and state changes throughout an episode.”
Smaller, specialised models could be Physical AI’s next step
On where physical AI and synthetic data are heading, Schmid predicts that foundation models and pretrained predictors are becoming increasingly capable and accurate, and he expects that to continue.
Vecsei believes companies will increasingly realise that spending heavily on very large foundation models might provide short-term gains but doesn’t necessarily make economic sense over the long term.
“I expect we’ll move towards more specialised models and specialised solutions for individual companies.
Businesses will want smaller versions of what’s available today, both to reduce compute costs and to optimise models for their particular use cases.
I think that’s especially relevant in Europe.
We don’t necessarily have the same compute capacity as the US or China, so specialised models offer another path forward.”
