When fxguide recently covered the visual effects of The Wolverine, we marveled at the transition shots of Yashida from an old man to his younger self in the film’s final confrontation. These were completed by Weta Digital under the supervision of Martin Hill. Popular prof. Tahl Niran, who has taught Nuke and VFX Fundamentals courses at fxphd, played a key role in compositing those transitions. We asked him about the work.
What brief did you have for the aging and de-aging of Yashida?
Niran: Martin Hill, who was VFX supervisor for Weta, held a kick off briefing for the sequence where he took the team through the assembled edit for the sequence. He explained that we had an initial run of 5 shots where we had to transition from an actor who appears to be about 90 years of age through to around mid 20s in close up with dialogue, and then the reverse effect which would happen later in the sequence as part of the other action. He was very clear at that stage that he was aiming for a very natural and subtle approach that was not an obvious ‘effect’, but rather a very believable progression that happened seamlessly over the shots. To paraphrase, he didn’t want the audience to be able to pinpoint where the transition was taking place but still wanted a clear progression over the shots.
What was the challenge going in that you knew you had to solve?
Niran: I think there were a couple of things that we knew were going to be a challenge going in. All the shots where very close up, so there would need to be a lot of attention to detail. There was dialogue being delivered and a very strong facial performance, which we didn’t want to lose or dilute in any way. Finally, that people are very aware of the appearance of human faces and they create an immediate emotional response in any viewer, anytime you try and manipulate them it can lead to a very creepy or odd result if not done well.
How was the live action filmed for the two actors in the suit – and what additional photography and other information (HDRIs, FACS sessions etc) did you refer to during the process?
Niran: There was a lot of material that was shot for this sequence. Martin was also present for all the main photography. But basically two actors were filmed in the suit on the location performing all the action and dialogue. The actor who plays Yashida when he is an old man was filmed with the prosthetic ‘ageing’ makeup that you see a number times throughout the film, as well as without this make-up (as he is significantly younger in real life). Then the process was repeated with the actor who plays Yashida as a young man. So you had three ‘passes’ for every shot. Depending on where in the sequence the shot would occur, one of those three plates would be our master and we would work on that. This was all also filmed with additional ‘witness’ cameras which meant that we had a very good reference for facial shapes and performance and most importantly for matchmove.
Additionally there was huge amount of on-set data collected. People at Weta have a very well established practice for data capture and I am sure I am only scratching the surface with what I know, HDR’s for IBLs, reference photographs of both actors in a variety of angles. There was a FACS session done which ended up being essential later on. As well as many texture reference images.
Tahl’s popular VFX Foundations courses can be taken as fxphd Vault Course – find out more details on our fxphd store page
What was the overall approach – in terms of the use of 2D and 3D?
Niran: I think everyone wants things to be as simple as possible but still get the best looking result. In this case we ended up with quite a complex process, just because we started with a simple solution and built in additional detail and subtlety to really make the whole effect as refined as possible. So by the end there was a lot going on, and quite a few people contributing. Ultimately it was a very hybrid 2d/3d approach. Every frame you see has a combination of live action plate which has been manipulated and also contains 3d rendered portions. The challenge was that we had two different actors, as well as the actors in different stages of make-up (both in their ‘normal’ state and with an ageing treatment applied).
Ultimately we would create a hybrid digital asset which was the older actor’s skin detail on the younger actor’s facial features which in turn was digitally transitioned. So we basically had four different ‘stages’ of ageing which we gave unique names to identify them, so you have 4 different CG assets and tying them together is a lot of comp work and under, and sometimes on top of, that is a lot of warping and manipulation of both the CG and the plate in Nuke.
What approach did you specifically take in Nuke – can you break it down step by step?
Niran: We started by projecting the plate onto matchmoved geometry in Nuke then baked that out to UV space. This gave us two things, the texture, but also the plate lighting. Simultaneously we created a full digi double of our two actors, which is easy for me to say but is a massive amount of work. Luckily this is a company with a well known and highly successful history in highly detailed characters. I can’t stress enough how much I was amazed by the work done by the 3d team on these shots, myself and the other compositors would sit there in awe when renders turned up!
The baked UV map was used as a template to paint a series of animated transition maps in Nuke. These were basically black and white maps which controlled how we would blend from asset to asset. I built a basic map with timings and some movement just to see how this would look in one shot and then we rolled it out to all the others. Later these maps would be re-created uniquely for every shot. Miguel Diaz Cachero, the other compositor doing the de-ageing, and I spent a lot of time crafting these transitions for each shot based on Martin’s notes, with James Russell, Kirsty Lawlor and Jan Dubbeke creating the re-aging shots. We started with one overall map per shot, but later broke this down into 4 different maps with slightly different qualities and timing.
– 1 which would be used for the big mesh deformation between character models. This map drove mesh deformation through and in-house tool.
– 1 for controlling things like wrinkles and displacements which varied a lot between different ageing states.
– 1 for the diffuse skin textures and things like blotches and age spots.
– 1 for the specularity and sheen of the characters skin which was very different between older and younger.
Then these maps went to the shader team to drive a very complex skin shader network. Andrew Taylor did the shader setup and actually was really heavily involved in the process. He and I sat next to each other throughout the show, so Miguel and I were able to rapidly update our maps and have them feed back into the shader and in turn into lighting. Maps were generally 4k, but for some shots I had to go to 8k as we were doing transitions along directions of some wrinkles which were smaller than a pixel wide.
Renders were lit using IBLs with a lot of refining from the TD’s. Daisuke Maki and Chris Husson were responsible for lighting the 5 de-ageing shots. They did an amazing job and we could place the CG assets side by side with the live action and they would hold up perfectly. But for certain areas of the actors’ performances we wanted to keep the original details from the plate. So we often needed to blend portions of the live action exactly with patches of CG. In order to make them blend seamlessly across shadows and light variations in the plate we did a little trick to isolate lighting from the plate, unwrapped in UV space, and could then dial in plate lighting across soft edges and seams which worked really well.
So I had two processes which contributed to this – the first was the initial lighting which accounted for any subtle differences between the plate and the CG on the first frame of the sequence, then a secondary process which accounted for any difference in lighting between the first frame and any subsequent frames in the shot. There were some flickering and flashes which were practical effects on set so it allowed us to dial all that in using controls in Nuke.
One other thing you may miss without repeat viewings is that Yashida has a scraggly grey beard when he is older, but the younger Yashida is clean shaven. This meant that we had to do a lot of little manipulations for things like facial hair and eyebrows which indicate a person’s age. Black hairs grow back in between grey ones and other hairs actually change colour along their length. The TD’s were able to create a variety of dynamic hair transitions using Weta’s hair tools and we then used deep compositing to help integrate them in a way that would not have been possible using holdouts.
Finally, for the areas of the shots where we used just the live action plate, we applied subtle treatments in a number of areas which also transition along with a rendered de-ageing. Things like adding cataracts to the eyes which clear and evaporate, and ageing treatments to the lips, teeth and gums. To do this we planar tracked in a cataract effect built on procedural noise. We also did a lot of little morphing corrections to the plate for things like moles and liver spots. The shots really ended up being a kitchen sink approach.
…and then there is the suit itself, which is also a combination of live action practical and CG render!
What do you think made the de/ageing shots so successful?
Niran: Martin Hill was very clear that he didn’t want to do an obvious morph or wipe effect. Mostly when you see a transition of a person to another character there is a clear ‘edge’ of the effect which you can read. One of the things that I think works is that there isn’t one for Yashida’s de-ageing. All throughout the show people would come over to my monitor and comment on the ‘seams’ between the live action portions and the CG rendered bits or wrinkles or spots that didn’t line up. There was a point a few weeks into it when, even on still frames zoomed in, they all started commenting on the wrong bits and guessing wrong and that was when I knew that we were getting it right.
I can’t stress enough how much of the success of this shot relies on the amazing character skills of the team at Weta. I was really nervous at the start of these shots but in the back of mind I knew that I was working at the company that have made some of the most memorable digital character performances of all time. A lot of people did really amazing work to bring this to life. The whole sequence relied on a really detailed matchmove, not just of overall position but on really precise subtle shapes of eyes and lips, which they did just brilliantly. Animation was just perfect, you have someone talking and performing in tight close-up and it is totally believable. The team behind the textures did an incredible job making a Yashida that was accurate down to the pores in his skin. All the TD’s who worked on the shot too. Just a great team.
Tell us about your role as a prof. with fxphd – what are the courses you’ve taught and how has that experience been?
Niran: To be honest I couldn’t name all of the classes I have been involved in off the top of my head, without digging back through the archives. I know that I have done things on Shake, Maya, Mocha, Silhouette, Nuke and VFX foundations but to me it is all just vfx knowledge and all of equal value.
I started working with fxphd at the very first round of courses and have been involved ever since. Back then there was really very little out there in terms of compositing training. I had written some free tutorials on multipass compositing and dealing with new tools and techniques like ambient occlusion and a lot of people were really interested in the material and contacted me about doing more. I was just a freelance compositor working around the place. One day I was reading some threads on fxguide about how to learn more about compositing and got in talking to Mike about maybe being involved in this new training concept that he and some of the other fxguide guys were putting together, that turned out to be fxphd.
After that went so well I went back to doing shots and teaching classes in my non-existent down time. That led to the chance to go to Mumbai and train some customers for the Foundry. While there Ben Minall and I had the idea to do an event like the old fxguide live events but with a focus on Nuke and the first Nuke Masterclass was born, which we ended up doing as a collaboration between fxphd and The Foundry.
All in all it has been a fantastic experience. I have met lots and lots of great people. Had the chance to travel all over the world and just learned a tremendous amount. My favourite thing would have to be the community aspect. There are people who I have met through this experience that I consider to be really close friends and I value that tremendously. I cannot tell you the number of times over the past 6 or so years where I have been working at a facility when another artist will come up to me and say, “Hey, I recognised you name / voice from your classes at fxphd!” It’s a really funny and great experience.
What benefits do you think fxphd offers students and artists in the industry?
No one knows everything about visual effects and everyone can always get better and learn more. You should never stop learning.
I have always tried to make the material I teach at fxphd the sort of thing that I wanted to learn myself. If I discovered something new or learnt a new technique, I would always try and put that in a class somewhere. Starting out in compositing it was really hard to get knowledge and information about how to do things or even how tools were supposed to work. fxphd is more than just the classes – it is a environment where people can learn, try things and then get help with problems along the way.
From the outset John, Mike and I always said that the courses shouldn’t just be ‘demo’ material where everything works like in a pre-packaged shot, but rather it should be about how I as a real working artist would deal with the challenges that come up when trying to make visual effects. I think fxphd provides a unique opportunity for people who are learning to challenge themselves and try things is a stress free environment. I still work in production doing shots and understand that sometimes you have to fall back on what will get the shot done and out the door. But fxphd can be a place to experiment and try new things, to really expand your repertoire and develop solutions and workflows.
Find out more about Tahl at his website www.tahlniran.com.
Special thanks to Weta Digital. All images TM and © 2012 Twentieth Century Fox Film Corporation. All rights reserved.