Transparent Video On The iPhone, It Can Be Done… With Flash!

The creative for a client project I worked on recently involved several video loops of actors shot on green screen. The videos needed to overlap, so the actors didn’t look like they were in discreet spaces, but in one seamless environment. The problem with this creative was that the iPhone SDK doesn’t include a video codec with alpha channel (ie animation codec with millions of colors +). I’m not aware of all the CODECs that are allowed but I have used .H264 and also .mov with PhotoJPEG.
I played around with using PNG image sequences in a UIView animation sequence, but the file size was an unacceptable 400MGs. I also had a look into some of the more complex aspects of AVFoundation using AVVideoCompositions with two layers, one for the video, and another for a video with a solid color representing the transparency. The idea was to then extract the color out, turning it into a transparent layer and combinig the layers back into a composition. However I wasn’t even sure if this would leave me with video with a transparent layer outside of the AVComposition, that could be placed on a PNG background, so I’d still be left with the same problem. It then occurred to me to test out Adobe’s AIR 2.7 for the iPhone to export using an .flv. It sounded crazy and I didn’t think it was going to work, but to my surprise it did.

Now I have absolutley no idea how Adobe package up a Flash app for iPhone, but I think it’s amazing that a video format that not only isn’t compatible with, but also has technical functionality that can’t be replicated on the native iOS SDK can work through a third party’s system. So just to share with you some of the results, I was attempting to play four transparent videos, all slightly overlapping with a png image in the background. The playback was unacceptably slow with this many streams. I then tried, two transparent videos with a PNG background and the playback was fine, I didn’t notice any frame grabbing or glitches, and one full screen iPhone video with transparency and a PNG image in the back ground ran fine!

Again, I’d like to express my amazement at being able to play an .flv with transparency on the iPhone. I have no idea how Adobe are able to do it, but I really hope Apple step up their game and include a video codec with alpha channel sometime soon. The only down side to using ARI 2.7 is the lack of support for native UIKit components, meaning you have to build everything in Flash/Flex.

If you want to play around with transparent video on the iPhone with Flash you’ll need to download the trial version of Flash Builder 4.5 from here, and check out this post on setting up your system to export from Flash Builder to your iPhone. You do need to be registered as an Apple dev, as to my knowledge there is no simulator in the Adobe set up, but it was all pretty straight forward (no less painful than having to do it through Apple and XCode).

Here’s the AS3 code, I just created a new Mobile ActionScript Project, as I’ve heard Flex Mobile Projects add unwanted overhead to your app (but don’t quote me on that). You’ll also need to make sure that the info in the project’s XML file (application descriptor file) matches all of your code signing identity info ie com.mycompany.myAppID
One thing to note is that I didn’t use the StageWebView object to place my video. According to Adobe’s docs StageWebView’s shouldn’t overlap with each other, so this wasn’t suitable for this project. The only piece of AIR specific code is the

new File(new File("app:/transparentVids/BB_bg.png").nativePath).url;

which is how we access files from the application bundle, the rest is just standard AS3. Again if you don’t have a need for transparency, I’d suggest you use a StageWebView with H264 CODEC the performance will probably be significantly better particularly on older iOS devices.

package
{
	import flash.display.StageAlign;
	import flash.display.StageScaleMode;
	import flash.display.MovieClip;
	import flash.events.*;
	import flash.media.Video;
	import flash.net.NetStream;
	import flash.net.NetConnection;
	import flash.filesystem.File;
	import flash.net.URLRequest;
	import flash.display.Sprite;
	import flash.display.Loader;
	
	public class Main extends Sprite
	{
		private var ns:NetStream;
		private var ns2:NetStream;
		private var ns3:NetStream;
		
		public function Main()
		{
			super();
			stage.align = StageAlign.TOP_LEFT;
			stage.scaleMode = StageScaleMode.NO_SCALE;
			this.init();
		}
		
		private function init():void
		{
			var bgFilePath:String = new File(new File("app:/transparentVids/BB_bg.png").nativePath).url;
			var fPath:String = new File(new File("app:/transparentVids/myTransparentVid1.flv").nativePath).url;
			var fPath2:String = new File(new File("app:/transparentVids/myTransparentVid2.flv").nativePath).url;
			
			var nc:NetConnection = new NetConnection();
			nc.connect(null);
			this.ns=new NetStream(nc);
			ns.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns.play(fPath);
			
			var nc2:NetConnection = new NetConnection();
			nc2.connect(null);
			this.ns2=new NetStream(nc2);
			ns2.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns2.play(fPath2);
			
			var nc3:NetConnection = new NetConnection();
			nc3.connect(null);
			this.ns3=new NetStream(nc3);
			ns3.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns3.play(fPath);
			
			var nc3:NetConnection = new NetConnection();
			nc3.connect(null);
			this.ns3=new NetStream(nc3);
			ns3.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns3.play(fPath2);
			
			var loader:Loader = new Loader();
			addChild(loader);
			loader.load( new URLRequest(bgFilePath));
			
			var vid:Video = new Video();
			vid.attachNetStream(ns);
			vid.width = 320;
			vid.height = 120;
			addChild(vid);
			var vid2:Video = new Video();
			vid2.attachNetStream(ns2);
			vid2.y = 120;
			vid2.width = 320;
			vid2.height = 120;
			addChild(vid2);
			var vid3:Video = new Video();
			vid3.attachNetStream(ns3);
			vid3.y = 240;
			vid3.width = 320;
			vid3.height = 120;
			addChild(vid3);
			
		}
		
		private function asyncErrorHandler(event:AsyncErrorEvent):void {
			//handle error
		}
	}
}