The stream of our SIGGRAPH Realtime Live demo is up on teh internets. If you haven’t seen the actual live demo, check it out!
I feels amazing to win the award for best Realtime Graphics amongst such industry giants. There are so many companies from so many industries participating now, and the event has grown such much. Feels really humbling to be honored with this for a third year; no pressure!
I have been working on Ryse for almost two years now, it’s one of the most amazing projects I have had the chance to work on. The team we have assembled is just amazing, and it’s great to be in the position to show people what games can look like on next-gen hardware.. Autodesk asked us to come out to Anaheim and talk about some of the pipeline work we have been doing, and it’s great to finally be able to share some of the this stuff.
A lot of people have been asking about the fidelity, like ‘where are all those polygons?’, if you look at the video, you will see that the regular Romans, they actually have leather ties modeled that deform with the movement of the plates, and something that might never be noticed: deforming leather straps underneath the plates modeled/rigged holding together every piece of Lorica Segmata armor, and underneath that: a red tunic! Ryse is a labor of love!
We’re all working pretty hard right now, but it’s the kind of ‘pixel fucking’ that makes great art -we’re really polishing, and having a blast. We hope the characters and world we have created knock your socks off in November.
No longer working for Crytek, maybe I can comment on some industry related things without worrying that my opinions could be misconstrewn as those of my former employer.
EuroGamer visited Quantic Dream this week, the studio working on the game ‘Heavy Rain’, who’s founder, de Fondaumière, arrogantly proclaimed that there was ‘no longer an uncanny valley‘, and that there are ‘very, very few‘ real artists in the video game industry. (A real class act, no?)
So their article starts with “We can’t tell you how Heavy Rain looks, sounds or plays…”, which I find kind of ridiculous seeing as how the studio’s only real claim to fame right now is the hype of it’s co-founder who casually claims they have accomplished one of the most amazing visual feats in the history of computer graphics (in real-time no less!).
Across the world there are thousands of outstanding artists chasing this same dream, from Final Fantasy, to Polar Express and Beowulf; people have tried to cross the ‘uncanny valley’ for years, and are getting closer every day. At Christmas you will be treated to what is probably one of the closest attempts yet. (Digital Domain’s work in Benjamin Button)
I have yet to see anything stunning to back up the talk, 8 months after making his statement about crossing the uncanny valley, they released this video, which was just not even close, to be frank.
It looks like they aren’t using performance capture. Without markers on the face this means they have to solve the facial animation from elsewhere, usually a seated actress who pretends to be saying lines that were said in the other full body capture session. There’s a reason why studios like Imageworks don’t do this, it’s hard to sync the two performances together. If they have accomplished what other’s have not, with much less hardware/technology, it means they have some of the best artists/animators out there, and I say hats off to them.
But with every image they do release, and every arrogant statement, it is digging the hole deeper. The sad thing is they could release of of the greatest interactive experiences yet, but their main claim is the most realistic cg humans yet to be seen, and if they fail at this, it will overshadow everything.
At least they know how their fellow ps3 devs over at Guerilla must have been feeling for a few years now.
Facial motion capture stabilization is basically where you isolate the movement of the face from the movement of the head. This sounds pretty simple, but it is actually a really difficult problem. In this post I will talk about the general process and give you an example facial stabilization python script.
Disclaimer: The script I have written here is loosely adapted from a MEL script in the book Mocap for Artists, and not something proprietary to Crytek. This is a great book for people of all experience levels, and has a chapter dedicated to facial mocap. Lastly, this script is not padded out or optimized.
To follow this you will need some facial mocap data, there is some freely downloadable here at www.mocap.lt. Grab the FBX file.
Get at least 3 markers on the actor that do not move when they move their face. These are called ’stabilization markers’ (STAB markers). You will use these markers to create a coordinate space for the head, so it is important that they not move. STAB markers are commonly found on the left and right temple, and nose bridge. Using a headband and creating virtual markers from multiple solid left/right markers works even better. Headbands move, it’s good to keep this in mind, above you see a special headrig used on Kong to create stable markers.
It is a good idea to write some tools to help you out here. At work I have written tools to parse a performance and tell me the most stable markers at any given time, if you have this data, you can also blend between them.
Load up the facial mocap file you have downloaded, it should look something like this:
In the data we have, you can delete the root, the headband markers, as well as 1-RTMPL, 1-LTMPL, and 1-MNOSE could all be considered STAB markers.
As you can see, mocap data is just a bunch of translating points. So what we want to do is create a new coordinate system that has the motion of the head, and then use this to isolate the facial movement.
You create a library ‘myLib’ that you load into motionbuilder’s python environment. This is what does the heavy lifting, I say this because you don’t want to do things like send the position of every marker, every frame to your external app via telnet. I also load pyEuclid, a great vector library, because I didn’t feel like writing my own vector class. (MBuilder has no vector class)
So we will now create our own library that sits inside MBuilder, this will essentially be a ‘toolkit’ that we communicate with from the outside. Your ‘myLib’ can be called anything, but this should be the place you store functions that do the real processing jobs, you will feed into to them from the outside UI later. The first thing you will need inside the MB python environment is something to cast FBVector3D types into pyEuclid. This is fairly simple:
#casts point3 strings to pyEuclid vectorsdef vec3(point3):
return Vector3(point3, point3, point3)#casts a pyEuclid vector to FBVector3ddef fbv(point3):
return FBVector3d(point3.x, point3.y, point3.z)
Next is something that will return an FBModelList of models from an array of names, this is important later when we want to feed in model lists from our external app:
#returns an array of models when given an array of model names#useful with external apps/telnetlib uidef modelsFromStrings(modelNames):
output = for name in modelNames:
Now, if you were to take these snippets and save them as a file called myLib.py in your MBuilder directory tree (MotionBuilder75 Ext2\bin\x64\python\lib), you can load them into the MBuilder environment. (You should have also placed pyEuclid here)
It’s always good to mock-up code in telnet because, unlike the python console in MBuilder, it supports copy/paste etc..
In the image above, I get the position of a model in MBuilder, it returns as a FBVector3D, I then import myLib and pyEuclid and use our function above to ‘cast’ the FBVector3d to a pyEuclid vector. It can now be added, subtracted, multiplied, and more; all things that are not possible with the default MBuilder python tools. Our other function ‘fbv()‘ casts pyEuclid vectors back to FBVector3d, so that MBuilder can read them.
So we can now do vector math in motionbuilder! Next we will add some code to our ‘myLib’ that stabilizes the face.
Adding Stabilization-Specific Code to ‘myLib’
One thing we will need to do a lot is generate ‘virtual markers’ from the existing markers. To do this, we need a function that returns the average position of however many vectors (marker positions) it is fed.
#returns average position of an FBModelList as FBVector3ddef avgPos(models):
mLen = len(models)if mLen == 1:
total = vec3(models.Translation)for i inrange(1, mLen):
total += vec3(models[i].Translation)
avgTranslation = total/mLen
Here is an example of avgPos() in use:
Now onto the stabilization code:
#stabilizes face markers, input 4 FBModelList arrays, leaveOrig for leaving original markersdef stab(right,left,center,markers,leaveOrig):
pMatrix = FBMatrix()
lScene = lSystem.Scene
newMarkers = def faceOrient():
Rpos = vec3(avgPos(right))
Lpos = vec3(avgPos(left))
Cpos = vec3(avgPos(center))#build the coordinate system of the head
xVec = (Cpos - Rpos)
xVec = xVec.normalize()
zVec = ((Cpos - vec3(faceAttach.Translation)).normalize()).cross(xVec)
zVec = zVec.normalize()
yVec = xVec.cross(zVec)
yVec = yVec.normalize()
facePos = (Rpos + Lpos)/2
pMatrix = xVec.x
pMatrix = xVec.y
pMatrix = xVec.z
pMatrix = yVec.x
pMatrix = yVec.y
pMatrix = yVec.z
pMatrix = zVec.x
pMatrix = zVec.y
pMatrix = zVec.z
pMatrix = facePos.x
pMatrix = facePos.y
pMatrix = facePos.z
lScene.Evaluate()#keys the translation and rotation of an animNodeListdef keyTransRot(animNodeList):
for lNode in animNodeList:
if(lNode.Name == 'Lcl Translation'):
lNode.KeyCandidate()if(lNode.Name == 'Lcl Rotation'):
Rpos = vec3(avgPos(right))
Lpos = vec3(avgPos(left))
Cpos = vec3(avgPos(center))#create a null that will visualize the head coordsys, then position and orient it
faceAttach = FBModelNull("faceAttach")
faceAttach.Show = True
faceAttach.Translation = fbv((Rpos + Lpos)/2)
faceOrient()#create new set of stabilized nulls, non-destructive, this should be tied to 'leaveOrig' laterfor obj in markers:
new = FBModelNull(obj.Name + '_stab')
newTran = vec3(obj.Translation)new.Translation = fbv(newTran)new.Show = Truenew.Size = 20new.Parent = faceAttach
lPlayerControl = FBPlayerControl()
FStart = int(lPlayerControl.ZoomWindowStart.GetFrame(True))
FStop = int(lPlayerControl.ZoomWindowStop.GetFrame(True))
animNodes = faceAttach.AnimationNode.Nodesfor frame inrange(FStart,FStop):
#build proper head coordsys
faceOrient()#update stabilized markers and key themfor m inrange(0,len(newMarkers)):
markerAnimNodes = newMarkers[m].AnimationNode.Nodes
We feed our ‘stab‘function FBModelLists of right, left, and center stabilization markers, it creates virtual markers from these groups. Then ‘markers’ is all the markers to be stabilized. ‘leavrOrig’ is an option I usually add, this allows for non-destructive use, I have just made the fn leave original in this example, as I favor this, so this option does nothing, but you could add it. With the original markers left, you can immediately see if there was an error in your script. (new motion should match orig)
Creating an External UI that Uses ‘myLib’
Earlier I mentioned Creating Interactive MotionBuilder User Interface Tools, where I explain how to screenscrape/use the telnet Python Remote Server to create an interactive external UI that floats as a window in MotionBuilder itself. I also use the libraries mentioned in the above article.
The code for the facial stabilization UI I have created is here: [stab_ui.py]
I will now step through code snippets pertaining to our facial STAB tool:
This returns a list of strings that are the currently selected models in MBuilder. This is the main thing that our external UI does. The person needs to interactively choose the right, left, and center markers, then all the markers that will be stabilized.
At the left here you see what the UI looks like. To add some feedback to the buttons, you can make them change to reflect that the user has selected markers. We do so by changing the button text.
This also stores all the markers the user has chosen into the variable ‘rStabMarkers‘. Once we have all the markers the user has chosen, we need to send them to ‘myLib‘ in MBuilder so that it can run our ‘stab‘ function on them. This will happen when they click ‘Stabilize Markerset‘.
Above we now use ‘modelsFromStrings‘ to feed ‘myLib’ the names of selected models. When you run this on thousands of frames, it will actually hang for up to a minute or two while it does all the processing. I discuss optimizations below. Here is a video of what you should have when stabilization is complete:
Kill the keyframes on the root (faceAttach) to remove head motion
Remember: Your stabilization will only be as good as your STAB markers. It really pays off to create tools to check marker stability.
Sometimes the terminal/screen scraping runs into issues. The mbPipe function can be padded out a lot and made more robust, this here was just an example. If you look at the external python console, you can see exactly what mbPipe is sending to MBuilder, and what it is receiving back through the terminal:
Sending>>> selectedModels = FBModelList()
Sending>>> for item in selectedModels: print item.Name
['Subject 1-RH1', 'Subject 1-RTMPL']
All of the above can be padded out and optimized. For instance, you could try to do everything without a single lPlayerControl.StepForward() or lScene.Evaluate(), but this takes a lot of MotionBuilder/programming knowhow; it involves only using the keyframe data to generate your matrices, positions etc, and never querying a model.
I was talking to my friend Marco the other day. As he is a real programmer, he is somewhat equipped with the needed skills required to decode MotionBuilder’s procedurally-generated Python documentation. We were both frustrated, fighting with the ‘Python Console Tool’, when I showed him the telnet interface he was like “why don’t you just use that?”
And this is what I started doing. I now do much of my tests and work in the telnet console, because, unlike the built in console that Motion Builder offers, the telnet window at least offers copy/paste, and you can press the up arrow to cycle through previous arguments that you have entered. I would suggest using this until Autodesk adds usable features to their ‘Python Console Tool’.
Vicon Datastations allow you to string off a remote trigger which can allow you to start and stop of a motion capture take with a physical button. This could allow you to start/stop motion capture with sensors or anything else. In our case, we wanted to start/stop another device at the same exact time and have it sync’d with the mocap data, also, allow one person to run the device and the mocap session.
Disclaimer:I am aware that the remote interface is the same for the V8i/612/624/460/V6 Datastations, but I built this for the V8i, which looks like this:
This is what the ‘J1 REMOTE‘ port looks like on the back of your Datastation:
Here is the description of the J1 Remote in the Vicon hardware manual:
Located directly below the camera inter face connectors, the J1 connector function is to allow the remote control of data capture from external switches or photoelectric sensors. Connecting Start (pin 3) or Stop (pin 5) to Ground (pin 7) will initiate the selected function. Pin 1 generates a negative going TTL gated reference signal, which is aligned, to the camera Horizontal Synchronisation (HD) signal and present when data capture is being per formed.
The hardware manual will tell you that the J1 Remote Interface Connector is a Lemo Part (FGG.1B.307.CLAD52). So you will have to order this (follow the link). Below is the pin out from the manual, it’s pretty simple stuff:
Building the Trigger
Working with Relays
So, what we want to do is make a start and a stop button, or you could make an on/off switch. I made a button. The button flips a relay, which is like a switch. Below you see 5 pins, labeled ‘start‘, ‘stop‘, ‘grnd‘ and ‘coil‘. When you apply power to the coil, it will connect the grnd from stop to start and vice versa. Because it’s a magnet that flips the switch, nothing from the inner circuitry of the trigger can send any interference to the Vicon Datastation.
Below you see two relays, one triggers start/stop, the other triggers an LED. You can get relays that flip multiple poles at once. If you wanted to start/stop other devices with the same buttons you would add more relays, or use a multi-pole. In my example below I was sure to get relays and LEDs that work with a 9v battery, this way you do not need resistors or anything to alter the voltage.
This is what a final remote trigger can look like, green starts, red stops. The green LED can be on while capturing. The above relay will flip the light on/off based on button contact, even if red is pressed first, so you may want to go a different route if someone has butter fingers. The cord is durable microphone cord, as we only need 3 wires (start/stop/grnd, mic cable = left/right/grnd).
Note: The J1 Remote Trigger works in Vicon Workstation, however, when Vicon updated it’s software to IQ, they did not want to spend the time to continue support of the remote trigger. IQ supports newer technology like the ‘MX Remote’ made by Vicon, which they would rather have you purchase. So yes, if you update your Vicon Software, certain features of your Vicon hardware will become useless.