FVEA 419/619: Motion Capture for Artists 2015

marey2red

“Progress is not possible without deviation” –Frank Zappa

Evolving Class Schedule for 2015:

January 12
First class meeting.  We opened the class with a general discussion of various motion capture technologies and the nature of the system we have at CalArts.  During the discussion we explored ways in which the course might work for the diverse group of students making up the class.  The specifics of the week-to-week course structure will be determined in large part by the individual interests of the group.  Following the discussions we took a brief break and then began the second part of the afternoon viewing several approaches to motion capture –starting with a brilliantly cynical warning about the type of mocap abuse we hope to avoid.  Most of the following videos include scenes created via the PhaseSpace mocap system at CalArts. The final video is a look at the innovative early approach to motion capture for computer graphics used in the pioneering television commercial Brilliance created by Robert Abel and Associates.  All of these projects serve as useful examples for contemplating the possibilities of what we might do in this semesters course.  In response to a question about the mocap done for Shimbe’s thesis film, The Wonder Hospital, we looked at shots of his Bunraku mocap experiments from my photo album of the initial PhaseSpace tests at CalArts.

.

The California Institute of Motion Capture, CalArts Producers Show Intro 2007
We can do this the easy way or the hard way –or not do it at all.
Let’s push the boundaries of irony and do the dance of the mushroom cloud anyway!

.


Ke Jiang made Taxi while he was a student at CalArts.  He used The PhaseSpace mocap system to create a quirky performance by taking advantage of the artifacts that occur at the edge of the capture volume:

.


Visiting Artist Max Hattler conducted a workshop during the Program in Experimental Animation interim sessions in 2011.  The goal was to produce one or more short works using abstracted motion capture.  Forms I (Taekwondo) is one of those works:

.


Shimbe used the PhaseSpace motion capture system in a unique way for the making of this film. He rigged a Bunraku puppet with active markers and directed Danielle Ash as the puppeteer. The natural floppiness of the puppet provided an extraordinary quality to the targeted motion. You can see some still photos of the process in my photo album of the initial PhaseSpace tests at CalArts.

“The Wonder Hospital, a 3D & puppet animated film, is a surreal journey of oddity and empty illusion. In a mysterious hospital, modification of physical beauty is not what you would expect. A girl’s desire for superficial beauty leads her to chase after the luring ‘After’ images on a path of advertisements throughout the hospital. But in the end she finds something unimaginable and irreversible.”

.


A Maya playblast from 18 March 2010 of Sara Pocock‘s little girl character animated via a simple mocap T-pose test. The T-pose test was performed in class by Justin Leon in order to double check that we had setup the MotionBuilder marker mapping process correctly before moving on to a directed capture session. We came close do doing a brief capture session but ran out of time and had to postpone the session until the upcoming class. The realtime data from the T-pose test is all that we used in this test. No clean-up, filtering, retargeting, or other adjustments were done. Justin’s simple casual movements gave the character an unintended sense of presence. In subsequent class meetings Justin and Sara worked on directed performance tests in order to gain more experience with that form of mocap –even though Sara’s goal was to keyframe all of the animation in the final film.

.


For her MFA thesis project, A Scenic View of the End of the WorldIvy Flores chose to collaborate with choreographer Daniel Charon and composer Alex Wand in an iterative process wherein each participant would base what they were doing on the work the others had done in an earlier version.  This enfolding process modified the work with each iteration until Ivy reached the form she found most interesting.  The motion capture of the dancers was recorded with the PhaseSpace Impulse system mounted in the computer animation lab located in room F105 and processed to extract essential movement.  The final installation was presented in the Black and White Studio Gallery A404 from April 4-5, 2013.

.


Ivy Flores’ documentation of the process of creating  “A Scenic View of the End of the World”.

.


Prior to her work with the PhaseSpace Impulse motion capture system, Ivy created this performance animation piece using two orthogonally placed video cameras and Adobe After Effects motion tracking.

.


Rachel Ho collaborated with Julian Petschek, and Rob Gordon in the creation of the live motion capture performance, Mo Cap Mo Problems, staged in the Black and White Studio Gallery A404 as part of the Motion Capture for Artists course exhibition in the Spring of 2013.

Mo Cap Mo Problems is a 15 minute performance and video installation that employs live motion-capture in the engagement of virtual characters and spaces. The performance deals with issues of identity and technology in the service of pop culture, as explored through role-playing and the form of music gigs/concerts.”

.

Following upon the critical success of Mo Cap Mo Problems, Rachel Ho developed SLEIGHTING. This video features footage from 5 shows performed on April 3rd, 2014 in the CalArts B&W Studio A404 and was constructed to promote performances as part of the LAX 2014 Festival on September 20th, 2014, at the Bootleg Theater, Los Angeles.

SLEIGHTING is an unprecedented approach to multimedia performance using real-time motion capture and pre-visualization tools to enable a new breed of performer. It is about showmanship, hype and the future of entertainment.

Through the ability to pilot avatars before a live audience, SLEIGHTING creates a new type of superstar who is no longer confined to being one personality, but is able to be anyone and everyone. Like the superstar DJ or sportsman who commands arenas full of fans, SLEIGHTING presents itself as a future art and sport, and an event that people descend upon to witness and partake in. In this case, the arena is now the site of reinvention for the event film, and the director is now conductor and performer, empowered through technological extensions of the self.
The show has a running time of around 20 minutes and mainly consists of three sketches in which the spectacle of interfacing with virtual realities drives both narrative and design. Real-time motion capture and pre-visualization tools, typically used in the film and gaming industries, are now used to DJ virtual camera moves and special effects for this live event.”


This promo documents several aspects of an early marker based motion capture system. Markers were hand tracked from a 2D screen and then plotted into the 3D character’s motion channels. The software used in this project grew out of the code used to drive motion control cameras. That code was soon incorporated into the first off-the-shelf 3D CG package created and marketed by Wavefront, Inc. Robert Able and Associates created the ground breaking Brilliance commercial for a single airing during Super Bowl XIX. Notice the subtle and not so subtle interplay of hype and confidence building between studio, agency, and client.

.

January 19
Martin Luther King Holiday

.

January 26
Second class meeting.   View and discuss video documentation of the 2013 Mocap for Artists Course Exhibition.  View and discuss a series of video’s covering the use of mocap by actors driving photo realistic characters in feature film making (see the class page on Theatrical Cinema for more examples). First and last assigned reading of the semester:  Maureen Furness’s comprehensive overview paper, Motion Capture.

.

February 2
Third class meeting.  Suit up class member Megan Lewicki and learn about the placement of markers on the suit relative to the body, examine the other components of the PhaseSpace Impulse system, and  explore several ways the live data may be viewed in the PhaseSpace Impulse software.  Start up Autodesk MotionBuilder and go over loading the PhaseSpace OWL plug-in into a scene, loading in an “actor” element, assigning markers to the “actor”, dropping a “character” onto the “actor” and recording some performances.  Add additional “characters” to demonstrate multiples driven by a single “actor”.

.

February 9
Fourth class meeting. Guest artist Rachel Ho presented her work and a comprehensive overview of new directions in mocap.  Megan Lewicki suited up and performed as Rachel’s characters in the world of “Sleighting”.
.

February 16
President’s Day Holiday.

.

February 23
Fifth class meeting.  Set up a trial run to ensure that all students were effectively able to run Parallels in non-coherence mode, start Windows 7,  launch Autodesk MotionBuilder 2014, and test the use of streaming marker data from the PhaseSpace Motion Master program.  Once everyone had MotionBuilder working properly with Master we suited up Isaiah Baez and recorded him in the classic T-pose and some impromptu moves.  This was followed by an exploration of the correct procedures necessary to assign marker sets to the appropriate segments of the “actor” module and  configure the recommended rigid body clusters.  We then added the “character” Aragor into the scene and selected the “actor” as the input source for driving Aragor.

.

March 2
Sixth class meeting.  The plan was to leave CalArts no later than 1:15pm in several car pool groups for the drive to Darnell Williams’ mocap studio in Venice.  Our ETA was 2:15pm (or earlier) and our expected time of departure from the studio was 3:15pm with an ETA of 4:00pm (or earlier) at CalArts. The drive to Venice went as planned however due to unexpected heavier than usual traffic for that time of day we all arrived at CalArts late which unfortunately caused a number of students to be late to their 4:00pm classes.

Darnell gave a brief tour of the studio and spoke a bit about his personal history as an artist and of the new directions he is taking his studio. Darnell demonstrated a new markerless camera based mocap system that he is experimenting with integrating into his workflow. Following a Q&A with the class, Darnell led them into the capture space one-by-one to record Tee, Crane, and Atlas poses to serve as mapping reference for a capture session. He then directed a capture test of the largest group he had attempted to date (with the caveat that it might be more than the system was capable of realizing at that point in time –and thus a risky yet worthy challenge).

vlcsnap-2015-03-12-20h42m50s111
Click on the photo above for a link to more images from the class visit to Darnell’s studio

 .

March 9
Seventh class meeting. Gordon Straub led the class in working with the Autodesk Character Generator online application.

autodeskCharacterGenerator
Character Generator gives an  artist the ability to create a customized model with characteristics selected from a set of visual templates. The geometry of the generated character is textured mapped and rigged for animation, and may be saved in an FBX file format suitable for use in a number of animation programs including Autodesk MotionBuilder.  Autodesk provides a set of Essential Skills Videos for Character Generator to guide artists in using the program.

.

March 16
Eighth class meeting.

.

March 23
Spring Break.

.

March 30
Ninth class meeting. A comprehensive look at the development and application of Facial Performance Capture.

.

April 6
Tenth class meeting. Review and discuss examples of various approaches to facial animation ending with a viewing of “Sweetie”.  Hands-on work with Motion Builder running under Parallels on the Mac Pro workstations in the F105 computer lab.  We encountered several difficulties with that process including out of control camera move tools in MotionBuilder.

Kevin Spacey discusses his experience with performance capture for the game Call of Duty: Advanced Warfare

 

Jasper Brekelmans’ work developing motion capture software such as Brekel Pro Face 2 designed for use with Microsoft’s latest Kinect sensor has been of great importance to the community of artists working with commodity tools. In addition to this work Jasper is employed in the animation industry, working at companies such as Motek Entertainment.  Jasper recently lent his skills to the Sweetie campaign for Terre des Hommes.

 

.

April 13
Eleventh class meeting.  Explain the issues with the MotionBuilder camera move tools under Parallels and describe the click-drag-release work around.  Discuss project plans.  Check calibration and re-calibrate if necessary. We decided the old calibration was acceptable for use with Kyle Chang’s project since he did not want completely clean motion, but instead wanted to exploit the popping and twisting created by the missing markers of out of range data. Kyle suited up and class members worked to populate the suit with markers. We  examined the need to re-encoded the driver with a fresh profile. Several students elected to do a follow along in MoBu on their workstations while we adjusted the “actor model” to match the markers, created marker sets, followed by rigid body assignments.  I loaded the “Humanoid” character  for the demo while students loaded characters of their own choosing and we selected the “actor” as the input source for the “character”. We recorded several takes of Kyle’s performance.  After class I experimented with some test renders from a section of one take in order to demonstrate two possibilities for Kyle to consider when creating the rendered output for compositing into his film project.


Kyle requested that we record using a green screen for compositing the motion capture character with the other animation in his film project.  Here the character has been rendered as three dimensional shaded CG mesh.

 


This version is rendered with a flat white character.  There are many other rendering options available and Kyle will be working on exploring those options (as well as selected which motion sequences to use) next week.

 

.

April 20
Twelfth class meeting.  John Brennan’s scheduled visit is postponed to April 27 due to the demands of a project he is working on.  This weeks class meeting centers around a visit from Solène Morvan who serves as Lead Developer /  Support Manager at Dynamixyz.  Solène will provide an overview of their technology and demonstrate the facial performance capture process with one or more student volunteers.


This GDC 2014 trailer was made by Bretagne Production International using Dynamixyz’ Performer Suite and MOCAP Design’s headcam. Dynamixyz technology uses an innovative development called “Expressive Space” to produce high quality face analysis/synthesis in real-time. Expressive Spaces are complex spaces composed of many different dimensions. The system generates these spaces statistically by automatically analyzing video sequences. An Expressive Space can be custom-built to model a person’s expressiveness; their expressions can then be applied to a virtual character and commanded in real-time.

.

April 27
Thirteenth class meeting. Visiting artist and mocap expert John Brennan will discuss his work and cover best practices with the PhaseSpace system and other motion capture tools.

.

May 4
Fourteenth class meeting. Final class session and party!

.

May 11
The School of Film/Video Bijou Festival.

.

.