Neural Style Transfer for Matte Painting and Fractal Art
The rapid advent of neural networks in the field of computer graphics have opened some interesting possibilities. Most examples of Deep Neural Style transfer I have seen focus on applying painterly styles to pictures or videos. I’ve seen plenty interesting results, but I cannot stomach to see yet another ‘Starry Night’ styled picture. Therefor I wanted to focus on a few other uses beyond the glorified Instagram filters.
The rapid advent of neural networks in the field of computer graphics have opened some interesting possibilities. Most examples of Deep Neural Style transfer I have seen focus on applying painterly styles to pictures or videos. I’ve seen plenty interesting results, but I cannot stomach to see yet another ‘Starry Night’ styled picture. Therefor I wanted to focus on a few other uses beyond the glorified Instagram filters.
Matte Painting
Using a real photograph as a style picture could be a very handy tool for the digital matte painter. The detail-hungry matte painter frequently clone stamps or cuts/pastes from photographic reference to create the realistic detail their client expects.
Neural Style transfer seems to offer an amazing opportunity here. You can mock up the basic geometry with any tool, either 2d or 3d and apply the photorealistic style to the image. While the result is far from perfect, it offers a great foundation to improve with other photoshop techniques. The examples below are examples of what you could do with Mandelbulb3D as a base, with photo-styles applied.
click on the images for full screen, and use the arrows to switch from source to result.
You might wonder what I used as style images in above examples. For some, I used multiple and blended them, but basically I don't have the copyright, so can't publish. Just do a Google image search for 'Nature' or 'City' - that kind of stuff. I also used the technique that I talked about in my last blog post - a Google Similar Image search on the fractal source image.
How does Neural Style Transfer work?
If you do not understand how neural style transfer works, I’ll try to offer some insight. I will give you the usual disclaimer that I don’t really understand how it works under the hood. Instead, I will offer a more intuitive understanding of the process.
Deep learning is the active operator. Deep because the algorithm has many layers, and Learning because it improves over time.
Think of the many layers as a hierarchy in a large company. Every single person can only do so much, and at the bottom of the ladder a person has little insight and little power. One layer up the people manage the people at the bottom, and they have some insight and more interesting strategies apply. This continues to be true all the way to the top, where the individual person is not per se a smarter person, but their decisions set off a much more complex cascade of actions They have many strategies to choose from as their operations are much more delicate.
In the style transfer program, the bottom layers only operate on the pixel level, which won’t create anything interesting at all. But keep adding layers (my understanding is that the algorithm I use contains 19) and at the top we get a larger-scale artificial ‘understanding’ of the image.
So what’s the image that the algorithm ‘understands’? What happens next is still magical to me, but both the source image and the style image are interpreted in this way, and a random algorithm is tried to match one to the other. The first random try won’t result in anything useful, so the computer measures how big this ‘error’ is and tries another one that will decrease this error. This continuously decreasing of the error is the ‘learning’ part. After a few hundred or thousands of iterations, this error is sufficiently low, and behold – style transfer has taken place.
Fractal Art with Neural Style
Apart from the (somewhat trippy) matte painting examples above, there are of course many more options. One of them is to apply a style to an infinite fractal zoom, as I did in my latest Fractal short, Recurrence. A clever Optical Flow system ensures that there’s stability over time, but at the same time the scale of the style will keep looking similar, just as with a fractal. Unfortunately, it did take an extremely long time to process this video, and I hope that in the future faster techniques will be made available.
Where's the software?
Finally you may wonder how you can do this Style Transfer yourself. There are a couple of websites, such as deepart.io that offer the service. If you want to do it yourself, be prepared for a lot of work - you need to install a lot of code before it will work.
I used this implementation by Cameron Smith, which gave me the best result and flexibility. It also supports video.
I believe that these techniques have the future, so artist-friendly implementations for these things will hopefully come soon. I for one would definitely be willing to pay for such visual wonder.
Julius Horsthuis
Neural Networks interpreting Fractals
There's something intensely satisfying about peering into a machine's mind.
For me, one of the most amazing things about fractals is that at first glance they seem to resemble something from real life, but at closer inspection this appears an illusion. With the advent of Google's Deep Dream and many other Neural Networks I thought it was an interesting experiment to see how AI would interpret a fractal picture. I've done two simple tests to see where this could go.
First, I gave a couple of fractal images to Google's similar image search algorithm, to see what Google would think is on the picture.
So I uploaded this image of a pure fractal:
What would Google 'think' is on this picture? Some similar images came up, not surprisingly most where buildings, churches and such:
With DeepArt you can transform any image into a 'style' using a base picture and a style picture. I used the fractal image as the base image, and the church as a style image to render this:
Here are some other examples, all made from a base image and a style image based on its 'similar images' result:
Another algorithm that uses deep learning colors in black and white pictures. Its algorithm has learned the best way to do this based on thousands of real images, so it can determine the difference between a sky which is supposed to be blue and a tree that is green.
Feeding it fractals in order to see what it would 'think' it needed to color it gave some interesting results. I made sure each image was black and white when I gave it to the algorithm. Most just got back completely sepia, which gave it still a beautiful, old look. Here's some of the most interesting results:
All processed using demos.algorithmia.com/colorize-photos/
Although all this technology is still very young, it's absolutely apparent to me that these techniques will start to play an extremely large role in creating and interpreting art, and this is very exciting to me.
There's something intensely satisfying about peering into a machine's mind.
Live Visuals
I am mixing clips from a vast repertoire of all my own fractal work -plus a lot of unreleased material – long colorful fly-throughs that work great on large screens, morphing alien worlds that make those on the dance-floor feel like they are on a journey – keeping in tune with the ebb and flow of the music.
In my early twenties, I used to call myself VJ for a while. Armed with two DVD players and a Panasonic MX-50 I used to frequent semi-large venues in Amsterdam, mixing together some funky visuals for the clubbers of those days. Mandelbulb3D hadn’t been invented yet, so I relied on 3D Studio MAX to create tunnels, kaleidoscopic colors, and jumping mushrooms. At age 23, I won the ‘best VJ’ award from the Twix Talent Room. But somewhere around the year 2005 my interests shifted to other endeavors and I let the VJ-ing slide.
Until now. Just last weekend I had my renaissance as a live visual artist – by being invited for the Shattered Barriers event in London. The psychedelic trance music went very well with my fractal artwork. It was incredibly fun and this is definitely something that I will want to be doing more of.
Which is why I’m writing this blog post. If you’re an event organizer, and want to book me as a live visual artist, now you can. Don’t worry – I’m not using DVD players and an MX-50 anymore, but obviously have got the latest hardware and software.
I am mixing clips from a vast repertoire of all my own fractal work -plus a lot of unreleased material – long colorful fly-throughs that work great on large screens, morphing alien worlds that make those on the dance-floor feel like they are on a journey – keeping in tune with the ebb and flow of the music.
Mandelbulb Maniacs and the Future of Fractal Art
Slowly but surely it seems that my work, and the work of other great fractal artists around the world, is being noticed by a broader audience and also appreciated as a more mature art form.
Not just online, but also offline these works are being recognized and praised. Keep an eye on my Events page that I'll try to update frequently with events and festivals my work will show in one way or another.
Slowly but surely it seems that my work, and the work of other great fractal artists around the world, is being noticed by a broader audience and also appreciated as a more mature art form. I won't go so far as to say it's popular - as I believe this abstract art will still only appeal to a certain audience, but there's traction nevertheless.
The Facebook group Mandelbulb Maniacs almost has 6000 members. These are not just people creating fractals, but also a lot of people who have seen the magnificent shapes these algorithms can produce, and they want to see regular updates in their timeline. In this group some truly amazing artists reside, and I'm often very intrigued by what Dortus Gunst, Philippe Jacqueroux, Gary Webb, Barbara Russell and many others I can't possibly all mention produce on a daily basis. There are also some interesting 'trends' that seem to emerge, especially because everybody shares their parameters, which means you can learn from each other's masterpieces. It would be an understatement to say that I've learned a lot from this group.
Not just online, but also offline these works are being recognized and praised. People use it for obvious reasons as VJ material in clubs and venues, but there's also an increasing number of digital art festivals that love to showcase fractalicious art. Last year I traveled to Geneva and Montreal to create a show for the full-dome format. This immersive format really provides fractals with an extra dimension. This year it seems that there will be more opportunities to showcase to the world what's possible in the infinite world of fractals.
I'm currently working on various projects that contain fractals in one way or another, and I hope to present you soon a list of places around the world where my work can be seen. For the moment there are some events lined up In the Netherlands, but hopefully I will be able to add more worldwide events soon. Right now, I'm getting ready to go to Thailand, to meet a great artist who's working on an immersive film. I hope that we'll be able to create something inspiring together.
Keep an eye on my Events page that I'll try to update frequently with events and festivals my work will show in one way or another.
Symmetry
Symmetry is actually really quite interesting: It’s everywhere in nature; in the shape of a leaf to virtually every living animal - they all have a symmetry axis through the middle. Therefore, symmetry is actually a pleasing sight. Any random shape with a vertical symmetry axis has the potential to look familiar: Rorschach blobs are probably the best example of this.
This is my first blog post – and it’s about symmetry, which is something I struggle with and think about a lot when I animate. There are plenty of these types of subjects that I’d like to discuss, like four-dimensional rotations or the use of music. You will find them updated regularly in this blog and you can participate in a discussion about these matters- since there must be so many ways of looking at and dealing with these issues.
I've been thinking about discussing Symmetry in one of my tutorials before, but I've decided it was too distracting from the subject.
Symmetry is actually really quite interesting: It’s everywhere in nature; in the shape of a leaf to virtually every living animal - they all have a symmetry axis through the middle. Therefore, symmetry is actually a pleasing sight. Any random shape with a vertical symmetry axis has the potential to look familiar: Rorschach blobs are probably the best example of this. It’s also the easiest way to create ‘a face’ – like in ‘the Resolution of the Thirteen Fractal Skulls’.
If we add even more symmetry axes, we start to get a kaleidoscope, turning virtually every random pattern into a pleasing flowery shape. Fractals carry a lot of symmetry and, pleasing as this may seem, I try to avoid it at all cost. Whenever I see a vertically symmetrical picture, I can’t help but feel I’m only looking at half a picture; the other half obscured by a mirror. With kaleidoscopic images the percentage goes down.
Since almost all fractals have symmetry axes scattered around like madness, I try to find camera angles that hide this. I almost never succeed at this for 100%, but the less symmetry I see, the more confident I am having something worthwhile. It helps sell the idea that every shape is unique- even though this isn't always the case. Even when the symmetry is undeniably beautiful, like in architecture, I offset the camera to film it from an angle, like in ‘Geologic Time’.
In the immersive short, ‘the Cryogenian’ the entire journey takes place through one long symmetrical corridor. Here I used lighting to create uniqueness, but I do believe it would have been much more interesting if there was a better way to break up the symmetry.
When morphing (animating) fractals, symmetry poses yet another problem. Matter will appear from nowhere or disappear into a mirror plane. I find this a deeply disturbing feature and it distracts from the beautiful nature of the morphs. Almost every fractal short I’ve made containing morphs, seems to have this problem- most notably ‘Looking Glass’ – but there are plenty others. Sometimes I use ‘modifiers’ in Mandelbulb3D to throw off the geometry a little, mostly _helispiral or _gnarl3d, but they are slow to render and add a lot of render time.
What are your ways to fight symmetry, or how do you embrace it? Let me know your thoughts,
Julius Horsthuis