CONTENT
The what stays put
Deep inside an object-recognition network, Gatys and colleagues found that exact pixels fade but the layout survives: houses, river, sky. That becomes the content to keep.
Inside the network
FUN AI PHOTOS, EXPLAINED
Same photo,
new brushwork.
Your holiday snap, painted by a network.
The filters that turn a photo into a swirling oil painting grew out of a 2015 research paper. It showed that a network trained to recognise objects keeps two things apart: what is in a picture and how it is painted.
2015 the paper that started it
Compare the methods- Keep the content
- Borrow the style
- Blend the two
- Two pictures in, one out
THE SHORT ANSWER
Content from one picture,
style from another.
Style transfer starts with two images.
The network keeps the layout of one and the texture of the other.
STYLE
The how gets copied
Style is measured as which visual features tend to appear together, a table called a Gram matrix. It captures colour, texture and brushwork while ignoring where things sit.
How it is made
SPEED
From a minute to a blink
The first method repainted each picture slowly, step by step. By 2016 to 2017, trained networks did it about 1,000 times faster, fast enough for video.
Compare the speeds
FOUR STEPS INSIDE THE FILTER
How a photo
becomes a painting.
This is the recipe from the 2015 paper. Faster filters train a network to jump straight to the answer, but they chase the same two targets.
Look at both pictures
Run the photo and the painting through a network trained to recognise objects, and record how its layers respond.
Note the layout
From a deep layer, keep the photo's arrangement of shapes, which the paper calls its content.
Measure the brushwork
From several layers, record which features fire together in the painting. These Gram matrices are its style.
Nudge until both match
Adjust a new image, step by step, until it matches the photo's content and the painting's style as closely as possible.
TWO NUMBERS WORTH KNOWING
Slow to start,
then real time.
One figure on how much faster it got,
one on how much art and photography it learned from.
≈ 1,000× faster
Trained networks vs the original method
Johnson and colleagues reported similar-looking results three orders of magnitude faster than the 2015 optimisation. The 2017 AdaIN method claimed nearly the same gain while accepting any style.
SPEED-UP · 2016-2017 PAPERS
≈ 80,000 each
Photos and paintings used to train it
The AdaIN network learned from everyday photos in the MS-COCO collection and paintings mostly from WikiArt, roughly 80,000 of each.
TRAINING SETS · AdaIN 2017
The 2015 method used the 16 convolutional and 5 pooling layers of VGG-19, a 19-layer network built for the 2014 ImageNet image recognition challenge.
A QUICK SPEED CHECK
How long does
one painting take?
Pick a method to see how long it took to restyle one 512 by 512 pixel image on the same graphics card in a 2017 comparison. The bar shows how slow it is.
Times exclude preparing the style image. Phones and newer hardware differ, but the gaps between methods are the point.
WHY IT LOOKS ODD SOMETIMES
Texture is easy,
shape is hard.
The 2015 authors wrote that content and style cannot be completely separated: there is usually no image that perfectly matches both. Lean on style and the scene dissolves into brushwork; lean on content and the painting fades.
Later image-to-image models such as CycleGAN, which turned photos into Monet or Ukiyo-e styles, found the same limit. Colour and texture changes often worked; changes of shape, such as turning a dog into a cat, mostly failed.
Questions
Questions people ask
Who invented AI style transfer?
The method most filters build on was published in August 2015 by Leon Gatys, Alexander Ecker and Matthias Bethge in a paper called A Neural Algorithm of Artistic Style.
Which paintings did the first paper use?
It repainted a photo of the Neckarfront in Tübingen in the styles of Turner's The Shipwreck of the Minotaur, Van Gogh's The Starry Night, Munch's The Scream, a 1910 Picasso nude and Kandinsky's Composition VII.
What is a Gram matrix, in plain English?
It is a table of how often each pair of visual features shows up together across the whole picture. Because it ignores where things are, it captures texture and colour rather than layout.
Why are some style filters limited to a few looks?
Fast filters from 2016 trained one network per painting, so each new style needed a new network. The 2017 AdaIN method removed that limit by matching simple feature statistics instead.
Why do faces and buildings sometimes warp?
The style target only cares about texture, not shape, and the authors say content and style cannot be fully separated. When style is weighted heavily, edges and fine details give way to brushstrokes.
Sources
- Gatys, Ecker and Bethge (2015): A Neural Algorithm of Artistic Style, arXiv
- Johnson, Alahi and Fei-Fei (2016): Perceptual Losses for Real-Time Style Transfer and Super-Resolution, arXiv
- Huang and Belongie (2017): Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, arXiv
- Zhu, Park, Isola and Efros: CycleGAN project page, including failure cases
- Simonyan and Zisserman (2014): Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG), arXiv
Updated