2.1.

Approaches to Generating Synthetic Data

For computer vision tasks, such as image classification, object detection, and semantic segmentation, the different approaches to generating synthetic datasets can be classified as follows:

Cut and paste approach. It involves cutting and pasting foreground objects on to background scenes, thus creating numerous combinations of synthetic images.¹⁷

Dvornik et al.¹⁸ used a dedicated convolutional neural network (CNN) to choose potential bounding boxes where an appropriate object can be placed based on an object score for each box in a given image. This resulted in an improved performance in VOC12.¹⁹

Wang et al.²⁰ utilized a simpler approach, where foreground objects of a class were carefully pasted on background images where a similar instance of that same class was removed. For example, a teddy bear instance from one image is pasted on another image where a similar teddy bear was removed. They called this instance switching, and the advantages were that context, shape and sometimes the scale also can be preserved to a certain extent. However, the major limitations of the cut and paste approach, such as inconsistent lighting (between foreground and background), and the creation of boundary artifacts remain unresolved. One exception is a complex pipeline involving geometrically consistent cut and paste methodology combined with 3D-specific image perturbation that improves upon state-of-the-art results in monocular 3D depth estimation²¹ on the nuscenes dataset.²²

Realistic synthetic environment approach. A 3D environment with well-placed 3D objects can provide a testing ground for various applications, particularly navigation and mapping. There have been several works that created outdoor environments for training self-driving cars^23–26 and unmanned aerial vehicles.^27–29

A notable dataset created using this approach is VKITTI,⁴ modeled after KITTI,³⁰ an urban self-driving dataset with ground-truth annotations of bounding boxes and semantic segmentation masks.

A popular dataset for urban semantic segmentation is SYNTHIA,³¹ which has data from a virtual New York city for 13 urban classes, such as roads, buildings, and pedestrians. Both SYNTHIA³¹ and VKITTI⁴ used the Unity game engine,³² whereas there are other works including DeepDrive³³ and VIVID,³⁴ which use the Unreal Engine to create their virtual environment.

The last couple of years has also seen a rise in high quality synthetic data generation pipelines,^35,36 which can produce realistic synthetic scenes, albeit with the limitation that they require 3D scans of objects to work.

Hybrid approach. It combines synthetically generated 3D foreground objects layered on background images taken from the real world. A good example is presented in Ref. 9, where the synthetic images are composed by combining 3D models with random textures, against a background of random images taken from the Flikr 8k.³⁷

A 2022 approach called photorealistic neural domain randomization (PNDR)³⁸ utilizes a neural rendering technique, which learns a combination of modular neural networks to generate high-quality renderings, randomizing different aspects of a scene including lighting and materials while still preserving realism.

In our work, we use a simpler hybrid approach, as will be described in Sec. 3. Apart from using only a limited set of nine images from Google Images, a distinguishing feature of our approach is how we convert the 2D images into decal surfaces so that they integrate into the environment in a more realistic fashion than Ref. 9, plus properties, such as surface brightness of objects, can be varied to introduce greater diversity into the dataset.

2.2.

Combination of Synthetic and Real Images

It has been shown in numerous instances³⁹ that synthetic data alone is not useful to train general-purpose models. Earlier, synthetic datasets were used to complement real datasets so that models trained on such combined datasets can generalize well to real-world test data. This addition of synthetic datasets for training, helped outperform models trained on real datasets alone.

In 2014, virtual human images from the video-game Half-Life 2 were used in conjunction with a real dataset called INRIA⁴⁰ for the task of pedestrian detection, and it was shown on several benchmark datasets that this outperformed training with just INRIA.⁴¹ Synthetic humans were generated with the help of 3D templates from Ref. 5 and shape information from Ref. 42 to generate a synthetic dataset called SURREAL.⁴³ The authors beat the state-of-the-art (approaches trained on real data only) on multiple tasks including body pose estimation and depth estimation, by training with a combination of real and virtual data.

Synthetic data have been used in object detection starting from 2015, when Peng et al.⁶ used CAD images to fine-tune an RCNN object detector⁴⁴ and showed that synthetic images of objects are useful when there is limited real data available, as in the Office dataset,⁴⁵ where the performance was better when trained on synthetic CAD images compared to webcam images of the same objects.

The state-of-the-art dataset generator called Kubric⁴⁶ released in 2022, built using Blender⁴⁷ and PyBullet,⁴⁸ can flexibly generate synthetic data for 11 different types of computer vision tasks from 3D-NeRF⁴⁹ to optical flow estimation. Even Kubric⁴⁶ with all the flexibility and scalability of its pipeline only generates synthetic datasets, which beat the state-of-the-art, when used in conjunction with real data.

2.3.

Pure Synthetic Data

Data from a virtual world were used to train a multiclass object detector⁵⁰ that outperformed state-of-the-art methods at the time, deformable parts model (DPM)⁵¹ and aggregate channel features (ACF)⁵² on the PETS09-S2L1 challenge.⁵³ They achieved the highest score on the precision metric even without using a pretrained network backbone and got competitive results on other metrics, such as recall and false positive count. McCormac et al.,⁵⁴ using their SceneNet dataset, showed that pretraining purely on synthetic data for semantic segmentation resulted in an improvement over pretraining on ImageNet when the final transfer learning task is segmentation on real datasets.

Tremblay et al.⁵⁵ achieved state of the art in pose estimation with six degrees of freedom, using only synthetic images, utilizing a combination of domain-randomized, and photorealistic images to train their pose estimator.

Similarly, Hinterstoisser et al.⁵⁶ demonstrated that training on pure synthetic data can outperform training on real data. The detection works on a set of 64 retail objects under various poses, heavy background clutter, partial occlusion, and illumination changes. However, their work relies on obtaining high-quality 3D scans of objects to train an object detector. In 6D object detection, PNDR,⁵⁷ mentioned in Sec. 2.1 achieves state-of-the-art results using purely synthetic data.

2.4.

ObjectNet

Popular datasets, such as ImageNet,⁵⁸ have been a major driving factor in the advancement of algorithms and network architectures. One of the newer ones is ObjectNet,² which was designed to be a difficult test-only dataset (as opposed to being used for both testing and training), and to challenge standard practices for transfer learning, which do not work well on it. Some samples from the ObjectNet paper² are shown in Fig. 1.

Fig. 1

Sample chair images from (a) ImageNet and (b) ObjectNet (column 2), taken from Ref. 2 against (c), (d) our synthetic chair images. This highlights the challenging aspects of ObjectNet (which are incorporated into our synthetic data): rotation, background, and viewpoints, compared to the much simpler ImageNet images of chairs.

The leave-one-out contrastive learning (LooC)⁵⁹ constructs separate embedding spaces for each invariant feature. When tested on a subset of ObjectNet containing 13 classes, their LooC network achieves 32.6% top-1 accuracy over a base benchmark of 30.9% using a supervised linear classifier.

In context-gated-convolution (CGC),⁶⁰ context-aware CNNs are created, where the weights are modified based on a global context, allowing an improved extraction of representative local patterns. Testing on ObjectNet, on a subset of 113 classes that are in common with ImageNet classes, their CGC architecture improves on the baseline Resnet50 (29.35%) by 2.18%, achieving a top-1 accuracy of 31.53% on the 113 ObjectNet classes. This is comparable to the same number of classes tested in the original ObjectNet paper.²

Big transfer (BiT)⁶¹ achieves 58.1% top-1 accuracy on ObjectNet, using large-scale pretraining and their internal dataset called JFT-300M, which has more than 1 billion labels spread over more than 300 million images. As their dataset has not been made public, one cannot directly compare against their work, let alone reproduce it or improve on it.

A huge attention-based transformer with two billion parameters⁶² pretrained on the JFT-3B, an even larger version of JFT-300M⁶¹ dataset, achieved 70.53% on ObjectNet. Contrastive language-image pretraining,⁶³ which uses natural language supervision in the way of text-image paired training, manages to achieve 72.3% top-1 accuracy on ObjectNet. A similar method called locked-image text tuning,⁵⁷ where the image models are locked after pretraining and the text models are tuned for the task at hand, achieves 81.1%, albeit with the limiting requirement of JFT⁶¹ pretraining and millions of ground truth image-text pairs.

3.1.

Test Dataset

There are 113 classes that overlap between ObjectNet and ImageNet. We organized those classes into categories by their purposes, such as chair and bench (category: furniture); cell phone and laptop (electronics); and vase, lampshade (home décor). We then randomly sampled one class from each of the top 10 categories, making 10 class labels in total, which we refer to as ObjectNet_subset. They are: mug, drill, umbrella, TV, cell phone, chair, bicycle, tennis racket, stuffed animal, and vase.

3.2.

Toolchain

For each of our classes, we downloaded freely available 3D models from 3D marketplaces: TurboSquid, CGTrader, and free3D. Based on availability of models and the complexity of each object (some objects have more intraclass variability than others), a varied number of 3D models were downloaded for each object class. For instance, different umbrellas may vary in size, color, and slightly in design, and so do tennis rackets; but other classes such chairs vary far more widely in their types, from swivel chairs to basic plastic ones. Table 1 shows the number of 3D models downloaded and used for each object class. Geometry for the 3D models is meshes, a form of boundary representation.

Table 1

Count of 3D models used for each class.

We then imported those models into the Unreal Engine (v4.24),⁶⁷ a 3D game engine that allows 3D modeling, animation, and game development, along with support for plugins and Python scripting. We used the in-built Unreal Python plugin for scripting the spawning, rotation, and scaling of 3D models; modifying the lighting; and changing object backgrounds. In Unreal, we used the AirSim plugin⁶⁸ to capture images from predefined angles and distances from each object. Airsim enables API controls of virtual cars and RAVs inside Unreal Engine and has been specifically designed to enable research in autonomous vehicles, computer vision, and reinforcement learning.

3.3.

Experimental Setup

3.3.1.

Background and object layout

Multiple copies of each object are laid out in a rectangular grid on the Unreal environment floor. Each copy of the object has variations in rotation, scale, and background surfaces. Each object is also surrounded by a floor material in the shape of a big square, which would serve as the object’s background when photographed from above. There is also point light above each object for illumination. A screenshot of some of the objects with simple backgrounds (such as wooden surfaces) and point lights above them are shown in Fig. 2.

Fig. 2

Objects and 2D background tiles laid out in an Unreal engine, in a rectangular grid-like fashion with the point lights above them, shown as light-bulb icons. The objects in the centre of the square tiles are so small compared to the tiles, they are not visible. The reason for the size difference is that the tiles have to cover a large area when the pictures are taken from the top-down but offset from the centre.

Decals. We also create more complex backgrounds using decals as mentioned in Sec. 2.1. Decals in the real world are special papers with design that are pasted onto surfaces, such as glass and metal. Decals inside an Unreal engine are analogous to real-world decals in that they are materials that can be wrapped around 3D polygons. We create decals out of photographs sourced from the Internet and wrap them around huge square tiles on top of which objects are placed. A screenshot of a diverse array of complex decal-wrapped background tiles is shown in Fig. 3.

Fig. 3

A top-down screenshot of objects placed (close together for illustrative purposes) on decal tiles. When pictures of those objects are captured, they have a complex background with a lot of clutter, as shown in Fig. 5. A progressively zoomed in version of the actual tiles and objects are shown in Fig. 4.

3.3.2.

Lighting

Every point light is applied from a random position in a $2\mathrm{m}\times 2{\mathrm{m}}^2$, 2.5 m above ground level. That point light has a random color in the red, green, and blue (RGB) color channels between the ranges 100 to 255 (e.g., white light has RGB channels 255, 255, 255). The light bulb icons represent the point lights above each object. In order to avoid point light colors mixing with each other, the objects and their respective point lights are placed far apart in the virtual environment as shown in Fig. 4.

Fig. 4

The picture of a mug on a complex tile taken from the (a) actual height, (b) moving clockwise, and (c) progressively zoomed out, until the objects are no longer seen and then finally the neighboring tiles are visible, marked with red arrows for clarity.

Apart from point lights, which correspond to object illumination, there is also a default “skylight” in an Unreal Engine that corresponds to scene illumination. The intensity of the scene illumination is set to a minimal level so that even when the random point light intensities become too low, the objects are visible enough in low-light conditions. There are also other scene variations that are being applied, namely saturation and contrast. Floor decals are applied evenly to all objects multiplicatively, i.e., each object has 10 different copies with 10 different decals—whereas saturation and contrast settings are only applied on subsets, i.e., the entire dataset is divided into subsets and a combination of the following saturation and contrast settings are applied on each subset:

• • color saturation: three settings of 50%, 100%, and 150%

• • contrast: two settings of 100% and 150%.

This is because saturation and contrast result in minor visual changes, and applying such settings multiplicatively would make the resulting dataset exponentially larger without adding as much visual variation within the dataset.

3.4.

Capture of Synthetic Images

The second phase is image capture, done using AirSim after placing all the objects. At each object location, the images are captured at four different viewpoints, as shown in Fig. 5 where a stuffed animal is photographed from the top at height $h$, then staying at $h$, making the following displacements along the $x$ and $y$ axes: $(0,d)$, $(d,0)$, $(d,d)$. Based on the displacements, the camera angles are also adjusted so that they face the centre of the object at all times. The synthetic images captured contain only two-dimensional information of the decal backgrounds. When the camera is rotated and photographs are not taken directly from top-down, the three-dimensional information of the 2D backgrounds is not accurate. Yet, our results in Sec. 4.2 show that the model performs well despite this loss of 3D information.

Fig. 5

A stuffed animal 3D model pictured in four different angles.

To view the objects in the virtual environment on top of the backgrounds on their actual scale and spacing, a mug is photographed starting from the default height and then progressively zoomed out until some neighboring background tiles are visible, as shown in Fig. 4. The tiles and objects are far apart to prevent the light rays of nearby point light sources from mixing with each other. During the batch capture of images, parameters, such as image resolution, gamma, and field of view of cameras can be easily modified through a json file as mentioned in our Github page ( https://github.com/saiabinesh/hybrid-synth).

3.5.

Collection of Real Data

To collect real data, a Google Chrome extension called “Download All Images⁶⁹” to batch-download images was used. For every search query, which is the name of the object class, e.g., “drill,” all the search results from Google Images are downloaded after scrolling down to the end of the page, until the “end of the results” message is displayed. The extension helps download all the images displayed in the current Google images page. Finally, after scrolling down to the very end and downloading the images, they are manually inspected to see if there are some incorrect images, such as thumbnails and wrong results.

3.6.

Testing on ObjectNet

We choose ResNet152⁷⁰ as our backbone architecture, as it is a well-researched and popular architecture in the computer vision community, with good baseline performance figures for many of the common computer vision tasks. It has also been one of the primary networks benchmarked by the authors and creators of ObjectNet.² After the collection of both synthetic and real data, a ResNet152⁷⁰ backbone CNN (pretrained on ImageNet) is fine-tuned on these respective images and the top-1 accuracy for both the synthetic and real data are calculated.

3.7.

Learning Rate Tuning

Learning Rate (LR) is one of the most important hyper-parameters to tune, and it can be inferred from Fig. 8 that LR significantly affects the performance of Resnet152 on the ObjectNet dataset (up to 40% points difference in validation accuracy between LRs). In this section, we explain in detail our LR tuning strategy, which involves taking an upper bound LR value from an LR range test and using that in an exponential StepLR decay scheduler. We elucidate our heuristics to calculate the optimal parameters for the decay schedule.

4.1.

Efficiency of Synthetic Data Generation

The time taken to collect real data using Google Images includes the time taken to perform the following steps:

From our experiments, this process took an average of 37.5 minutes per class, which equates to 62.5 min per 1000 images. This is the best case scenario where one just has to query and download the images that are already indexed on the web by Google Images. A lot of times in real life, the effort and time involved are higher, as it involves having to acquire or find the actual objects belonging to that label, then photograph those objects.

Collection of synthetic data involves the following steps:

Placing objects took 40 min for a dataset of 31,200 images (76.92 s per 1000 images). The overall time taken for placing objects and capturing images was 2.34 min per 1000 images. Synthetic data are not only more than $10\times$ faster to collect as shown in Table 2 but also easier to prototype and create new versions of data.

Table 2

Time taken. Synthetic data takes 11× less time.

4.2.

Top-1 Accuracy

4.2.2.

Against real data scraped from Internet

We were also able to beat real data collected for our specific subset of classes as outlined in Sec. 3.5. We fine-tune the final layer of a pretrained Resnet152 backbone on both Real10 and Synthetic10_v4. Using our LR strategy, the upper bound LR for Real10 was fixed at $4.018\times {10}^{-4}$, and the upper bound for Synthetic10_v4 was fixed at $2.6778\times {10}^{-5}$. Early stopping was implemented with a patience value of 5, where no improvement in validation accuracy for five epochs will terminate the training. Some other common training parameters that we used for both synthetic and real images are as follows:

• • Input data transforms:

  • – Random crop: $224\times 224$

  • – Normalize: based on ImageNet mean and standard deviation

  • – Horizontal flip

• • Batch size: 4

• • Momentum: 0.9

When we tested both models on ObjectNet_subset, the Synthetic10_v4 model outperformed the top-1 accuracy of Real10 by more than 3% points, as shown in Fig. 6.

Fig. 6

Top-1 accuracy of models trained on real versus synthetic data, when tested on ObjectNet_subset.

Night-time images. To test night-time and low-light performance of our model trained on synthetic data, we created the Objectnet_night subset, some samples of which are shown in Fig. 7.

Fig. 7

Samples of a (a) bicycle, (b) drill, and (c) mug from the Objectnet_night subset we created out of the Objectnet_10 test set by filtering only night-time and other similar low-light images.

We then tested the top-1 accuracy on the Objectnet_night subset for both the Synthetic10_v4 and the Real10 models. These results tabulated in Table 4 show that our synthetic model loses only seven tenths of a percentage point when going from the full dataset to night subset, whereas the Real10 model loses 18.9% points. This is because in our data generation pipeline we included broad range of lighting variations as mentioned in Sec. 3. This includes minimum ambient scene lighting and random point light intensities that can go very low.

Table 4

Top-1 accuracy of models trained on real and synthetic data on Objectnet10 dataset and the Objectnet_night subset with only night-time and very low-light test images.

	Synthetic10_v4 (%)	Real10 (%)
Objectnet_10	72.0	68.2
Objectnet_night	71.3	49.3

4.3.

Learning Rate Scheduling and Training Strategy

We use the validation set of the Real10 Google Images dataset to tune LR. The graph in Fig. 8 shows validation accuracy at the end of each epoch up to 50 epochs, for six different LRs, on the same version of synthetic data. The LRs chosen for this graph are $1\times {10}^{-2}$, $1\times {10}^{-3}$, $1\times {10}^{-4}$ up to $1\times {10}^{-7}$. We stopped after 50 epochs because none of the models were showing evidence of further improvement by then.

Fig. 8

Validation accuracy at the end of each epoch plotted against epochs.

The effect of LRs on convergence is even more pronounced on the extremes where very low LR of $1\times {10}^{-7}$ is very slow to learn (improving 0.44% per epoch), compared to $1\times {10}^{-6}$ (1.6% per epoch). Similarly, the highest LRs cause divergence in learning and a subsequent decrease in the validation accuracy. The curve of $1\times {10}^{-2}$ starts at 10.4% and gets back to 10.2% at the end of 30 epochs with very little learning in between, with $1\times {10}^{-3}$ even decreasing in accuracy compared to the first epoch. We postulate that the decrease in performance is because high LRs overshoot the minima and settle on solutions that are in less optimal regions of the weight space. Accordingly, this graph demonstrates the importance of selecting a suitable LR in affecting the maximum accuracy potential and the convergence speed of the network.

Also Table 5 shows the effectiveness of our strategy compared to cyclical learning rate (CLR)⁷¹ and manual tuning. Our top-1 accuracy shows a nearly 30% increase to manual tuning and also beats CLR by 16%, when using the same LR.

Table 5

The difference in parameters and accuracy when using manual tuning or CLR71 versus using our strategy.

A standard parameter search takes at least 30 epochs of training to monitor the learning curve for different LRs to fix one value. Case in point, a typical LR search with logarithmically increasing LRs of $1\times {10}^{-7}$, $1\times {10}^{-6}$, $1\times {10}^{-5}$, $1\times {10}^{-4}$, $1\times {10}^{-3}$, and $1\times {10}^{-2}$, each run for 30 epochs, would take a total of 180 epochs to tune and set the LR schedule as opposed to two or three epochs. This equates to time/resource savings between a factor of 180/3 and 180/2 in this case, i.e., an average saving of 75×, similar to that of CLR, while achieving better accuracy. It is worth mentioning that our method resulted in higher accuracy of 72% compared to a standard grid search followed by stochastic gradient descent.

4.4.

Synthetic Data Generation Parameters

Inspired by studies, such as Refs. 9, 56, and 76, many variations including point lights, image saturation, and contrast were included minimally from the beginning. But to what extent each parameter needs to be varied, was an open question demanding to be answered.

Consider the case of lighting. In a previous study⁹ that used an Unreal Engine, random point lights with random intensities were shown to be a major factor in contributing to the performance of their object detector. But the optimal range of intensity and color variations of the point lights are unknown.

To study the effect of each of those variations, we started with a minimal number of variations (two or three) in each factor to keep the final dataset size and the Unreal environment at a manageable level (five variations in ten factors for each object would result in a dataset size of $5^{10}$). Then we incrementally tuned a combination of factors to monitor the performance improvement, if any, and decided to keep or discard those factors and/or its additional variations. The variations that caused the most improvement in the top-1 accuracy are listed in Table 6.

Table 6

Model versions and the features added. v1: basic version with simple 90 deg rotations, and variations in point lights, contrast, and saturation; v2 and v1 with more number of 3D models; v3: random rotation in the range (−45 deg, 45 deg) on all objects, more point light variations and complex backgrounds from Google Images; v4 and v3 with multiple scale images.

As shown in Table 6, complex backgrounds combined with point light variations in intensity and color, improved the accuracy by 27% points. The complex backgrounds to the objects were downloaded from Google Images and scripted as decals to the floor of the Unreal engine. An example of a chair, laid out on the default grey floor of the Unreal engine, and on three other background images embedded on the floor, is shown in Fig. 9.

Fig. 9

Sample images of a chair in four different backgrounds including the Unreal default gray.

In addition to the image capture angles (which simulate rotations), random object (around all three axes) in the range of ($-45\mathrm{deg}$, 45 deg) also boosted the accuracy because image capture angles are fixed and so the random rotations help capture even more varied views of the objects. So did having images of objects in multiple scales; v4, which is v3, but added with the same pictures taken from multiple scales results in a 11.2% points as seen in Table 6. This was simulated by moving the virtual drone proportionally closer and away from the object when taking the pictures, i.e., to simulate zooming in and out, the camera on the virtual drone is moved closer and away from the object, respectively.

4.5.

Ablation Studies

Although the incremental approach detailed in the previous section helped broadly understand the effect of various parameters on the performance of each feature added to the synthetic data, an ablation study is necessary to more accurately quantify loss of performance if each of those individual features were removed.

The limitation of this ablation study is the resultant decrease in the dataset size if the dataset was generated without that particular feature. For example, the overall dataset size is 31,200 images. This included 3120 images from each of the 10 backgrounds, including the default grey background. When nine of the complex backgrounds were removed from the data generation process, the resulting dataset became $10\times$ smaller owing to the number of backgrounds reducing from 10 to 1. To combat this limitation, each of the smaller datasets resulting from the feature removals was duplicated to match the size of the full featured dataset size (31,200 images).

It can be seen from Table 7 that the most important feature that contributed to the performance was complex backgrounds, with a 35.5% points decrease when just the default backgrounds were used. The second most important feature is rotations, which when removed, achieved only a 44.2% accuracy, which is a 28%-point-decrease. The remaining two major features of point light colors and multiple scales caused 12.6% and 11.1% point reductions in accuracy, which are smaller but nonetheless substantial.

Table 7

Ablation studies with comparison against the full dataset with no features removed.