“Obviously, we are well into post on the second film,” said Jon Labrie, chief technology officer, Weta Ltd. “The second film is much more complex than the first one. First and foremost, we’ve got Gollum to realize in a lot of shots and he’s looking really good. There’s also Treebeard. Obviously, we not only need to have a creature that looks good, but we’ve got lip sync issues and facial animation issues to be tackling. We were able to crack those things, kind of, on the first film with the Cave Troll, but we’re getting into the meat of the problem right now.”
The facility will also have to create a huge battle scene for the Helm’s Deep battle, which will put its proprietary crowd creation system, Massive, to the test.
“Weta is growing to accommodate the second film. We peaked at about 165 digital artists on film one. We’re going to get up to around 220-230 digital artists for the second film,” said Labrie.
Labrie outlined some of the infrastructure upgrades that Weta was undertaking to accommodate the next film as well as some of the lessons learned from The Fellowship of the Ring.
“We’re making pretty significant steps into Linux-based workstations. They now appear to be becoming stable enough to be a viable alternative in both the 2D and 3D space. The problem with Linux is that it’s an open source system, so if you are having issues or difficulties with its stability, it’s like pushing on a rope; there’s no single vendor to deal with. You have to be self-deterministic in terms of how things work. You have to make your own choices and do your own tests on motherboards, graphics cards, applications, operating system releases, all those kinds of things.”
Weta had just taken delivery of 25 Linux workstations from IBM and Labrie reported that IBM and Hewlett Packard were the frontrunners for additional Linux workstation upgrades.
As a sign of the facility’s rapid growth, it was replacing its 64-port switch from Foundry networks with a 256-port core switch.
“We are also growing the central storage in the facility. We’re at about 9 TB right now, and we’re going to be growing that to about 20 TB,” he said.
In terms of offline storage, the facility was upgrading its StorageTek Tape Robot. “We’ve got about 45 TB right now and it’s growing significantly. Our lag time right now is about a day to get information out of there, there’s just so many requests stacked up. And that’s not really suitable so we’re going to double the number of slots from 560 to about 1200 slots, and we’re going to go away from DLT technology and go to either SDLT or LTO. “We’re also very interested in CXFS from SGI — a distributed file system. Weta has this centralized storage model, so not only the 3D artists, but also the 2D artists — the compositors, rotoscopers, and painters — they are not typically working against frames or plates that are local on their systems,” he explained. “They are working against data that’s remote. You pay a little in terms of interactivity in making that choice, but on the back end, you make up for it when you send your work off to render, because if you’re fortunate, you manage to score 30 or 40 procs off the render wall.”
Hence, the decision to go with a centralized storage system ultimately proved quite popular with the compositing team.
“I can’t tell you how many of the compositors have come to me and said ‘it’s sometimes frustrating because the thing’s not local, but it’s totally worth it when we send away a render, because the turnaround for a 2k render is extraordinary.’ Our entire compositing department — 35 seats — they all work at 2k. They don’t work on proxies ever. They can get through many iterations in a day and you know the old saying, ‘visual effects work is never done, it’s only abandoned.’ Well, in corollary, the more times you can iterate it, the more times you can look at and get other people to comment on it, the better the work is going to be.”
Labrie reported that the facility will also need to expand its render farm from 400 processors to 700.
“And since we are replacing a lot of workstations or adding second workstations to a lot of artists’ desks, I think we’re going to go from about 800 processors to about 1400 –1500 processors overall at Weta, if you include desktop and central resources.”
Post Mortem “We had a bit of a post mortem after the first film, to understand what we really needed to tackle for the second film. We had to do a lot of infrastructure clean up... lots of things that got messy in film one,” said Labrie.
He reported that Weta plans to use subdivison surfaces more aggressively for computer generated creatures in the second film than in the first.
“That’s a decision that was taken by Joe Letteri, ex-ILM visual effects supervisor, who joined us for the second film. He was interested in making the break and going from NURB surfaces to subdivision. And we have been working pretty aggressively on the shaders that are being used specifically for Gollum and Treebeard. We feel that we’ve made pretty significant advances in the look of those creatures.”
In terms of animating Gollum, Labrie reported that it was still unclear how Gollum would be generated — motion capture or keyframe animation.
“We have done a lot of reference motion capture with the actor who was cast as the voice talent — Andy Serkis. The reference animation that was captured in the motion capture studio was very good. I don’t know if it’s going to make it in the final film, but I suspect that you’re going end up with a fair bit of keyframe animation, and probably see a lot of motion capture as well. I think it’s going to be all over the map depending on the shot’s requirements.”
Another major lesson learned, according to Labrie, was in asset management. Weta had originally signed an agreement with Informix to purchase the Media 360 asset management package for the facility.
“We pushed it reasonably successfully into two departments [creatures and models] but at a very minimal level of functionality — register, retrieve, search and browse. That was it. “Basically the paradigm kind of broke down. Media 360 never understood the idea of a sequence of images being treated as a single asset,” explained Labrie. “But it’s pretty fundamental to what we do. They wanted us to register each individual frame as an asset. You have to understand that I’ve got scanners here that are reading frames every 3.5 seconds, and there were no real robust tools to automate the process of registering the media.”
But he stressed that, “It’s not all their fault. It was also Weta discovering what it means to actually try to implement a digital asset management system. There were a couple of lessons that we took away from that — first and foremost, you can’t ask your users to do a lot of extra steps to register assets. If it’s not integrated into their workflow, if it’s not kind of happening ‘automagically,’ then it’s probably not going to happen at all. They’re going to skip the step. They want the back end stuff. They want to be able to scan, search and browse. But what do you mean I have to register? What do you mean I have to say that this asset depends on that one?”
Labrie stressed that the process of registering assets has to be trivial to the artist to ensure that it actually happens.
“One of the more interesting approaches that I’ve seen is that you can define directories that are constantly being scanned for new assets and when new assets appear, they automatically get registered into the system, as long as they meet certain criteria in terms of file-naming conventions,” he explained. “As we figure out what we’re going to implement in the future, we’re looking toward that type of implementation where the asset ingestion process is more or less automated.”
He explained that Weta had attempted a publish-or-perish model, where artists were required to register their assets in order to get their work approved or share it with other users. “We thought that was a very clever way to force our hand, but it didn’t really work because there was still the opportunity to use the back door — call your friend up and say ‘it’s in this directory.’ And so they didn’t publish it.
“The other thing that we learned is that that it is not useful to go to all of the different departments and ask, ‘how do you want your asset management system to work?’ because you’re going to get 12 different answers. You have to accommodate all of the different workflows, but it can’t be all things to all people,” Labrie said. “And when you start getting into too many customizations... you can end up spending more time on the specification than the actual solution will save you on the back end. It’s a classic chaos problem in IT. How much is good enough? Do you actually want to use extreme programming to attack something like asset management?
“And so it has got to be unobtrusive, and ultimately, the benefits that you get from it have to be so clear and compelling that people will naturally gravitate to it,” Labrie summed up. “That’s one of the holy grails now and nobody seems to have cracked it.”
Editors’ Note: Since speaking with Film & Video, Jon Labrie has left Weta to pursue other opportunities.