Friday, May 6, 2016

Advice sought on 360 video processing SDKs

For a demo I would like to take come 360 video (panoramic, basically a moving environment map) such as that in this image:

An image such as you might get as a frame in a 360 video (
And I want to select a particular convex quad region (a rectangle will do in a pinch):

And map that to my full screen.

A canned or live source will do, but if live the camera needs to be cheap.   MacOS friendly preferred.

I'm guessing there is some terrific infrastructure/SDK that will make this easy, but my google-fu is so far inadequate.


Nick said...

Hi Pete, I'm wondering if this will do the job?

It's really light weight, just uses a shader to unwarp an image. Code on github.

- Nick

Peter Shirley said...

Thanks Nick! That may indeed be useful. Looks like it does not support video but the reprojection is there. And source is always a good thing :)

Nick said...

I think Bhautik will add it if we ask him :)

Peter Shirley said...

I'm trying to track some history now just for fun. I think this is the cubemap:

Bhautik Joshi said...

Hi there :] I'm going to get a start on the video support today. I don't think it'll take to long but in theory the central function you need is something that can ingest an equirectangular frame and spit out the region you are interested in. The main shader here does exactly that, but with lots of bells and whistles that you don't need for your particular use-case:

I can help you trim it down to just the bits you need if you like :]

Peter Shirley said...

Awesome! TYVM. Is equirectangular the defacto standard now?

Bhautik Joshi said...

De-facto? At least for now it is. It's an ok-enough-to-use, easy-to-represent format that can both be used to encode a 360x180 scene and also have it so that it's easy to 'preview' what is going on by just looking at the rectangular, unmapped texture.

I'm sure you know that there are all sorts of great reasons why a cubemap or some other type of projection is much better, but these don't have as much standardization and none of the consumer 360-cam manufacturers are supporting these other formats - yet.

One of the things that I'd like to do with vrEmbed is make it a passthrough for converting between types. It's possible to, for example, use it to take cubemap faces and spit out an equirectangular texture. Not easy or intuitive, but definitely doable :]

Peter Shirley said...

I much prefer polar maps of any kind to cube maps in a ray tracer-- for just the reason you say-- how to do easily represent that in a single array?

For my case, I wont be looking straight up or done, so equirectanglar is excellent.

Peter Shirley said...

I started looking at the code. All javascript? I don't know js but it looks clean and it's on my to-do list to learn it. Good excuse.

Bhautik Joshi said...

The scaffolding is javascript, but the bit that does all the real work - the shader - is GLSL and that'll run anywhere :] If you wanted to you could run that same piece of GLSL in whatever graphics framework you have wherever - be it c++, python, maya, XYZ random production tool or yes, javascript :]

Bhautik Joshi said...

Also: representing polar maps as a single array. I think I'd need to know a bit more about what application you had in mind and the kind of dimensionality you needed (eg. are you optimizing for space? speed of access? ability to preview?).

Just to throw it out there, if you wanted a space-efficient, 1D array that didn't penalize angular resolution you could simply represent it as a series of concentric circles - or better yet, a spiral starting at the center of the polar (I'm assuming fisheye) projection.

Peter Shirley said...

Oh I just meant cubemaps are simple but storing them as a single image sounds problematic.

GLSL is a rare thing I have used a lot, so that's lucky :)

All I want to do is map the "right" portion of the panorama onto a rectangle in 3D. By right I mean what the viewer would see through that window. So pretty straightforward. It's just for a proof-of-concept demo so it just working is all that matters.

Master Zap said...

I'm a little confused at what you want to do, that you can't do by uploading your video as a 360 to youtube and playing it fullscreen from it? :)

E.g. like this:


Peter Shirley said...

Playing it on youtube is SUPER close to what I want. I just need to be able to do an arbitrary quad viewport rather than a rectangle. With youtube's viewer source I bet it would be just a change in a few lines of code. Of course that might be a hard change to do or it might introduce terrible inefficiency :)

Peter Shirley said...

I just got this work in Quartz Composer with a custom core image kernel. vrembed looks like it is the best portable thing out there for this sort of thing and I will probably use it later because I need things to work on android.