Creating Custom Lipsync Code in Flash

FLEISCH

sKye: As an animator who's working on a point-and-click adventure game, one thing that was incredibly important to me from the start was accurate, customizable lipsync.

However, even though I'm currently making my game in a format that really is MADE for animation (Flash, or you can call it Animate if you're a yucky person), I felt like, in most cases, actually having a document that was CHOCKFUL of frames of lipsync was cumbersome. Keep in mind that adventure games, even mine as a relatively small one, are made up of hundreds of lines to coincide with every possible combination of choices made in an interactive story experience, and most of that "animation" is honestly fairly rudimentary. You might have, like, five sets of poses, AT THE MOST, in any given situation, with the mouth & jaw being the only thing that changes in that pose.

What I needed was to have complete control over that timing-- over what pose or set animation plays at what given moment during that line of dialogue... but with CODE, rather than storing thousands and thousands of identical mouth frames in my game.

FOIST

For my lipsync process, I've personally settled on using eight basic mouth shapes:

• static closed mouth
• mm
• oo
• ee
• eh
• nn
• ah

&

• oh.

On "mm", I squish the head down a little, & on "oo", I squish the head down a little more. "Static closed mouth", "ee", "eh" & "nn" get no squash or stretch. Then, on "ah", I stretch the head up a little, & on "oh", I stretch the head the most. I think it makes for a really fun cartoon-y Humongous Entertainment-esque result of the character being more alive, in spite of the many limitations in play.

I followed an example that I considered to be most convenient at the time, which I can't find the origin of anymore. It was an AS2 example which I then revamped to work in AS3. It looks like this:

How it works is you load in a sound file, hit Play, then press a button as you listen to the sound file, and every time you press the button, it will output the time you've pressed the button, along with a command afterwards to change the mouth graphic to a different frame. So you get a script like this:

The original script had the script respond to mouse clicks, but I think that one can get rather tired of clicking faster than tapping the keyboard, so I changed my version of the lipsync tool to being activated with the space bar.

What I ended up with goes a lil' something like this: you make a button (mine is an ugly radial black-green gradient circle, as it was when I found the example) and call its instance name "play_btn". Then you make another button (ugly radial black-red gradient circle) and call its instance name "stop_btn". Then, you make three little dynamic textboxes for the timecode information you'll be needing to know and manipulate-- call the instance names for these three textboxes "startatBox", "currentBox", & "durationBox".

Then, right upon the frame these buttons and boxes reside, you put this ActionScript:

stop();

// REFRESH LIPSYNC VARIABLES

for(var a:int=0; a<1000; a++) {
	this["doOnce" + a] = 0;
}

// BUTTONS

play_btn.addEventListener(MouseEvent.CLICK, startSound);
stop_btn.addEventListener(MouseEvent.CLICK, stopSound);

// IMPORTING

var SyncSound:Sound = new joe014();
var SyncChannel:SoundChannel = new SoundChannel();

// DURATION

var duration:Number = Math.round(SyncSound.length)/1000;
durationBox.text = String(duration);

// START BUTTON

function startSound(e:MouseEvent):void {
	SyncChannel.stop();
	for(var a:int=0; a<1000; a++)
    {
        this["doOnce" + a] = 0;
    }
	var startat = Number(startatBox.text) * 1000;
	SyncChannel = SyncSound.play(startat);
}

// STOP BUTTON

function stopSound(e:MouseEvent):void {
	SyncChannel.stop();
}

// CREATING NEW LIPSYNC SCRIPT

var n:Number = 0;
var currentTime:Number;
stage.addEventListener(KeyboardEvent.KEY_DOWN, myKeyDown);
function myKeyDown (e:KeyboardEvent):void {
if (e.keyCode == Keyboard.SPACE) {
	if (currentTime>0)
	{
		n = n+2;
		var stopCheck = "this.doOnce"+n;
		trace ("if (currentTime > "+currentTime+" && "+stopCheck+"!=1){"+stopCheck+" = 1;MovieClip(root).joeava.joehead.gotoAndPlay(2);}");
	}
}
}

// CONSTANTLY UPDATING SCRIPTS

stage.addEventListener(Event.ENTER_FRAME, EnterFrameLoop);
stage.addEventListener(Event.ENTER_FRAME, DialogueScript);

function EnterFrameLoop(e:Event):void {
	currentTime = Math.round(SyncChannel.position)/1000;
	currentBox.text = String(currentTime);
}

function DialogueScript(e:Event):void {
	currentTime = Math.round(SyncChannel.position)/1000;
}

One of the important things I've placed down twice in the code is to completely refresh those doOnce variables that are being generated that ensure that no mouth shape plays more than once. Obviously, in theory, the mouth commands for a single line could go HIGHER than 1,000, but... what kind of good game would ever have a single line that needed more than 1,000 mouth shapes? I think it's a good limitation for game-feel.

Another important note-- obviously, your own command that gets output would be different, depending on the instance names you chose and your symbol hierarchy in place.

For me, as is seen in my code example, my character is in a main symbol with an instance name of "joeava". Inside it, the head is in its own symbol with an instance name of "joehead". Then, in there, the first frame of the head has the ActionScript code, stop(); to keep all the mouth frames from being on a constant loop. After the stop(); frame are groups of three frames for each phoneme. The frame labels are set to each of the phonemes mentioned above, then on the third frame of each of those mouth shapes, I turn that frame into a keyframe and enter the code, gotoAndPlay(1); to bring it back to the static closed mouth. And, of course, your dialogue sound clip will, in all likehood, not be called "joe014", so just replace the common-sense parts with the stuff that would relate to your situation.

... To reemphasize, MY commands are targetting "joehead" in "joeava", and asking "joehead" in "joeava" to play a selected phoneme at a selected time. You can cue ANY symbol of ANY instance name any WHERE to play or do just any THING! It's all up to You.

Then, after I'm finished hitting Space bar to the beat of the full piece of dialogue (or starting that dialogue at a specific part by entering in a timecode in my "startatBox"), I would look through my outputted code, and then fill in the appropriate actual mouth shapes to correspond with each line of code-- so instead of a frame reading...

if (currentTime > 0.887 && this.doOnce6!=1) {
	this.doOnce6 = 1;
	MovieClip(root).joeava.joehead.gotoAndStop(2);
}

... it'll read...

 if (currentTime > 0.887 && this.doOnce6!=1) {
	this.doOnce6 = 1;
	MovieClip(root).joeava.joehead.gotoAndStop("oo");
}

Then I paste the output into the open DialogueScript in my code, then hit Play and see how the line is looking. If I timed anything wrong, I'd modify accordingly. This might seem a little taxing, but especially on longer dialogue, this line-by-line approach actually tended to be WAY faster than doing it all frame-by-frame, oddly enough.

After all that finagling, to implement the lipsync in my game, I have a little Movie Clip hidden off on the side of the screen called "dialogue". Inside of it, the first frame is called "none", where it obviously stays any time no dialogue is played. Each frame after that represents one of the dialogue lines waiting to be activated by an in-game interaction. So, when something on screen is pressed, and the code says something like...

dialogue.gotoAndStop("joe014");

Then the code on the "joe14" frame is activated, which I shall show the most relevant bits of (some of the sound-related stuff DOES depend on what I put before it using GreenSock audio stuff, but I think it's pretty readable):

stop();

// SKIP

stage.addEventListener(KeyboardEvent.KEY_DOWN, skipdialogue_joe014);

function skipdialogue_joe014 (e:KeyboardEvent):void {
if (e.keyCode == Keyboard.SPACE || e.keyCode == Keyboard.ESCAPE) {
	SyncChannel.stop();
	stage.removeEventListener(KeyboardEvent.KEY_DOWN, skipdialogue_joe014);
	stage.removeEventListener(Event.ENTER_FRAME, dialoguescript_joe014);
	gotoAndStop("none");
}
}

// REFRESH

for(a=0; a<1000; a++) {
	this["doOnce" + a] = 0;
}

// SOUND SET-UP

SyncTransform.volume = SyncControl;
var Sync_joe014:Sound = new joe014();
SyncChannel.stop();
SyncChannel = Sync_joe014.play(0, 1, SyncTransform);
SyncChannel.addEventListener(Event.SOUND_COMPLETE, soundFinish_joe014);
function soundFinish_joe014($evt:Event):void {
    SyncChannel.removeEventListener(Event.SOUND_COMPLETE, soundFinish_joe014);
	stage.removeEventListener(KeyboardEvent.KEY_DOWN, skipdialogue_joe014);
	stage.removeEventListener(Event.ENTER_FRAME, dialoguescript_joe014);
	gotoAndStop("none");
}

// LIPSYNC
	
stage.addEventListener(Event.ENTER_FRAME, dialoguescript_joe014);
function dialoguescript_joe014(e:Event):void {
	currentTime = SyncChannel.position/1000;
	if (currentTime>0.046 && this.doOnce46 != 1)
	{
		this.doOnce46 = 1;
		MovieClip(root).joeava.joehead.gotoAndPlay("oo");
	}
	if (currentTime>0.139 && this.doOnce48 != 1)
	{
		this.doOnce48 = 1;
		MovieClip(root).joeava.joehead.gotoAndPlay("ee");
	}
	if (currentTime>0.464 && this.doOnce50 != 1)
	{
		this.doOnce50 = 1;
		MovieClip(root).joeava.joehead.gotoAndPlay("ah");
	}
// etc., etc., etc.
}

Haha, you'll see this particular code starts later than doOnce1 -- that happened a lot at the time, due to me having to try multiple times from multiple points in the line of dialogue to get the timing right with my spacebar, leading to me modifying the code the second time around to start after the initial doOnce series had ended. doOnce? More, like... doManyTimes... ?

SEGUND

It took years of intense struggle, but I had made that big jump from AS2 in MX to AS3 in CS6. That's not where my learning curve hits a decline, though-- recently, having to switch systems forced my upgrading hand AGAIN.

Around this Time, I was faced with a new challenge in my pipeline, since I had perfected my ability to make this space-bar puppetry technique work, but ONLY for games. So.

• How do I successfully export out my coded lipsync as a VIDEO, if I wanted to use it in an actual animated project?

I had to adjust the code for the tool just a scootchamagootch to function with being exported out. Not EVERY AS3 code is perfectly compatible with being exported out, but thankfully for me (see what I had to do to export vector art from Flash as pixel art from MX/CS6 to modern-day), enough of the stupid-simple stuff works, and it's mainly stupid-simple stuff I gravitate to, even when I have to chain a whole bunch of that stupid-simple stuff together pretty inelegantly. (How's that for an oxymoron?)

Here's the modified code, which instead of being tethered to counting the playhead position of a sound file (which exported video doesn't seem to understand), it just counts up a generic Timer element, instead.

// REFRESH LIPSYNC VARIABLES

for(var a:int=0; a<1000; a++) {
	this["doOnce" + a] = 0;
}

// BUTTONS

play_btn.addEventListener(MouseEvent.CLICK, startSound);
stop_btn.addEventListener(MouseEvent.CLICK, stopSound);

// IMPORTING

var SyncSound:Sound = new joe014();
var SyncChannel:SoundChannel = new SoundChannel();

// DURATION

var duration:Number = Math.round(SyncSound.length)/1000;
durationBox.text = String(duration);

// START BUTTON

function startSound(e:MouseEvent):void {
	SyncChannel.stop();
	for(var a:int=0; a<1000; a++)
    {
        this["doOnce" + a] = 0;
    }
	var startat = Number(startatBox.text) * 1000;
	SyncChannel = SyncSound.play(startat);
}

play_btn.dispatchEvent(new MouseEvent(MouseEvent.CLICK));

// STOP BUTTON

function stopSound(e:MouseEvent):void {
	SyncChannel.stop();
}

// CREATING NEW LIPSYNC SCRIPT

var n:Number = 0;
var currentTime:Number;
stage.addEventListener(KeyboardEvent.KEY_DOWN, myKeyDown);
function myKeyDown (e:KeyboardEvent):void {
if (e.keyCode == Keyboard.SPACE) {
	if (n==0) {
		n=196;
	}
	if (currentTime>0)
	{
		n = n+3;
		currentTime = currentTime-0.0;
		var stopCheck = "this.doOnce"+n;
		trace ("if (currentTime > "+currentTime+" && "+stopCheck+"!=1){"+stopCheck+" = 1;MovieClip(root).joeava.joehead.gotoAndPlay(2);}");
	}
}
}

// CONSTANTLY UPDATING SCRIPTS

stage.addEventListener(Event.ENTER_FRAME, EnterFrameLoop);
stage.addEventListener(Event.ENTER_FRAME, DialogueScript);

function EnterFrameLoop(e:Event):void {
	currentTime = Math.round(SyncChannel.position)/1000;
	currentBox.text = String(currentTime);
}

var startTime = getTimer();

function DialogueScript(e:Event):void {
	var att = (getTimer() - startTime)/1000;
	if (att > 1.744 && this.doOnce2!=1){this.doOnce2 = 1;MovieClip(root).joeava.joehead.gotoAndPlay("nn");}
	if (att > 1.788 && this.doOnce3!=1){this.doOnce3 = 1;MovieClip(root).joeava.joehead.gotoAndStop("ah");}
	if (att > 1.9 && this.doOnce5!=1){this.doOnce5 = 1;MovieClip(root).joeava.joehead.gotoAndPlay("oo");}
// EC TETERA, EC TETERA, EC TETERA sounds like some kind of unearthly beast.
}

Then you just Export Video/Media, and the mouth timing will perfectly export out intact!

TRACE

Honestly, when it comes to updating applications like Flash, there are many negatives. The layout is atrocious-- it doesn't respect the economy of my screen at all, and it always seems to be getting in the way of itself. BUT, there IS one blamstarp of a new feature that updating has granted me, a TRUE CARTOON BOON: **__AUTOMATED LIPSYNC__**!

So. Again.

• Since automated lipsync will create a series of timed mouth frames in your Timeline with pretty accurate results, how do I then convert those frames into the more compact AS3 code I've grown accustomed to?

Moving forward, in many situations, it now makes a lot more sense to start off with the Automated Lipsync feature, modify that in any place where it's wrong, then somehow convert the frames into code. My silly but serviceable technique goes thusly.

I thought of putting a little command to trace (If you're wondering, trace means to output some kind of alert text when testing a Flash movie, much like all the output code we've been receiving this whole time) out a code at the start of each mouth frame I chose, but since for this Lipsync feature, I have to turn the mouth into a Graphic instead of a Movie Clip (in the MAIN scene, not embedded inside "joeava" like earlier-- for this part in the process, I have to take it out of that hierarchy), it doesn't respect ActionScript commands anymore. So, therefore, INSIDE the mouth Graphic, I made a SEPARATE Movie Clip that would run the trace command-- and as bad as that sounds, it gets me What I Want.

Here's the code placed in the Movie Clip on my "mm" frame:

if (MovieClip(root).currentTime>0) {
		MovieClip(root).n = MovieClip(root).n+1;
		var stopCheck = "this.doOnce"+MovieClip(root).n;
		trace ("if (currentTime > "+MovieClip(root).currentTime+" && "+stopCheck+"!=1){"+stopCheck+" = 1;MovieClip(root).joeava.joehead.gotoAndStop(\"mm\");}");
}

And the other mouth shapes would have the same script, save it be the phoneme name it calls its own.

In this situation, there IS no actual buttons or textboxes for the tool-- now, thanks to the last quandary, I've decided on just using the Timer element with a Small Circle of Friends:

// REFRESH LIPSYNC VARIABLES

for(var a:int=0; a<1000; a++) {
	this["doOnce" + a] = 0;
}

// CREATING NEW LIPSYNC SCRIPT

var n:Number = 0;
var currentTime:Number;

// CONSTANTLY UPDATING SCRIPTS

stage.addEventListener(Event.ENTER_FRAME, EnterFrameLoop);
stage.addEventListener(Event.ENTER_FRAME, DialogueScript);

function EnterFrameLoop(e:Event):void {
	currentTime = Math.round(getTimer() - startTime)/1000;
}

var startTime = getTimer();

function DialogueScript(e:Event):void {
	var att = (getTimer() - startTime)/1000;
}

So now, when I export out a new Test Movie from Flash, when the line plays in the Timeline, it spits out the corresponding timing code, like so--

One last hiccup-- the first time through, every line is a little over 100 milliseconds off. Just wait for the line to play through again, and the second time through it'll be correctly timed, if you'll just direct your attention over here--

It might sound less than optimal to have to actually watch the dialogue animation twice, let alone ONCE, but... as an animator, this is as fast a process as I've ever gotten to for getting out proper lipsync, so I'M not complaining. S'faster than any rendering process I've ever heard wind of. This right here is the cool, cool breeze of mouth-shape undulations bobblin' oh-so smooth, like Spooky from Kid's Typing. Like carykh, nothing refreshes me more than that smooth, smooth flap'o'th'lip.