How scary is the AI ​​that can draw?

Have you ever imagined what kind of magical sub-subs would be the combination of ants and pigs, crabs and whales, or any two of 100 creatures?

Now, AI can turn all these wild imaginations into reality!

And we just need to click the mouse to doodle casually, like this:

The horns of a rhino, the wings of an eagle, and the tail of a dinosaur can be combined into this:

Properly a professional creative work, don't be too friendly to painting Xiaobai.

More importantly, it may also inspire your creativity, which is one of the goals of the Google research team to launch this tool.

This AI painting artifact is called Chimera Painter, which is a web tool whose function is to generate highly realistic "little monsters" based on animal sketches.

After completing the graffiti, just click the "Convert" button, and it will automatically generate a "monster" with a 3D effect.

Interestingly, the Googel research team also used the monster image created by Chimera Painter to build a digital card game.

The attack value of each card in the picture is determined by the monsters above, and the skills of these monsters are determined by the two species they combine.

It is understood that the research and development of this AI tool is inspired by the "monsters" we usually see in games. Googel researchers believe that the creation of these monsters often requires game artists to have a high degree of artistic creativity and technical knowledge, and AI can act as a paintbrush to help them save time in artistic creation, such as one-click 3D rendering, and even enhance them creativity.

If there are 100 animals in the game, each of which can be merged with each other, it will be a lot of work for any artist, but it is very easy for machine learning.

So how does it do it?

GAN-based generative models

Chimera Painter, is a machine learning (ML) model. In order to generate high-quality images of monsters in any combination, the research team provided the model with thousands of biological images and marked special parts such as claws, legs, legs, and eyes for the model to train.

The training process of this model still utilizes the generative adversarial network (GAN), which we are already very familiar with, which can generate high-definition and realistic new images based on two convolutional neural networks: a generator and a discriminator. The way it works is that the generator is used to create new images and the discriminator is used to determine if those images are from the training dataset.

However, here the researchers propose a variant known as conditional GANs, in which the generator takes a separate input to guide the image generation process. Interestingly, this approach is quite different from other GAN work, which typically focuses on photo-realism, whereas the purpose of this tool is to fuse different species to generate a chimera.

To train the GAN, the research team created a dataset of full-color images containing silhouettes of individual creatures adapted from 3D biological models. The creature's outline describes the shape and size of each creature and provides a segmentation map to identify body parts.

The trained model can generate the best performing multi-species chimera based on the contours provided by the artist and embed it in Chimera Painter.

Create structured biological datasets

A problem encountered when using GANs to generate new species is that when rendering image details or low-contrast parts, spatial coherence may be lost, including the distinction between eyes, fingers, and even overlapping body parts with similar textures.

Therefore, it imposes certain requirements on the training dataset. Existing illustration libraries are not suitable as datasets for training ML models because they may have conflicting styles, or lack diversity. The dataset from which chimeras are generated needs to be unique, such as dramatic perspective, composition, and lighting.

To address this problem, the researchers developed a user-led semi-automatic approach to creating ML training datasets from 3D biological models. During this process, users will create and obtain a set of 3D creature models.

Specifically, they will create two sets of textures in Unreal Engine and overlay on the 3D model - one with full color textures (left) and one showing each part of the body (like head, ears, neck, etc.) , called the segmentation map (right).

Among them, the body subdivision part of Figure 2 is submitted to the model for training to ensure that the GAN understands the specific structure, shape, texture and proportion of various biological body parts.

The 3D creature models are all placed in a 3D scene and also use Unreal Engine. A set of automated scripts will take this 3D scene and interpolate between different poses, viewpoints, and zoom levels for each 3D biological model, creating full-color images and segmentation maps, forming the training dataset for the GAN.

Using this method, the researchers generated more than 10,000 image+segmentation map pairs for each 3D biological model, saving users about 20 minutes per image compared to manually creating this data.

Generate high-fidelity images

The size of the GAN's hyperparameters affects the quality of the model's output image. To verify which version of the model performed best, the research team collected and analyzed samples of different types of creatures generated by the model, and extracted some salient features from them, such as depth perception, patterns about creature textures, and realism of faces and eyes Wait.

This information will not only be used to train a new version of the model, but will be able to select the best image from each biological class (e.g. gazelle, lynx, gorilla, etc.) after the model has generated thousands of biological images.

Specifically, the research team optimized GAN through Perceptual Loss. This loss function component computes the difference between two images using features extracted from a separate convolutional neural network (CNN) that has been previously trained on millions of photos in the ImageNet dataset.

Features are extracted from different layers of the CNN and weights are applied to each feature, which affects the feature's contribution to the final loss value, and these weights are critical in determining the appearance of the final generated image.

Below are the results from training GANs with different perceptual loss weights.

The color change in the picture is mainly caused by the dataset, the reason is that a creature in the dataset often contains multiple textures (such as the red version and the gray version of the bat), however, ignoring the color change, many differences are also directly related to the perceptual loss value. changes related to.

The researchers found that certain values ​​produced sharper facial features, making the resulting creatures more realistic.

Below are some GAN-generated creatures trained with different perceptual loss weights, showing that the model can handle a small set of outputs and poses.

  online experience

In short, for art creators or painting lovers, Chimera Painter can easily create a large number of images by simply adjusting the partial shape, type or position of the creature, instead of drawing dozens of similar creatures from scratch. At the same time, the model also allows the use of creature silhouettes created with external programs such as Photoshop.