Spectaculum

Spectaculum (lat. spectacle, show, sight) is a hardware accelerated view for visual content on Android, supporting everything from single pictures to 360° VR video. It works with all sources that can write to a Surface or SurfaceTexture, e.g. MediaPlayer, MediaPlayer-Extended, ExoPlayer, Camera, and Camera2 API. A demo app is available on Google Play.

Features

• shader effects
• zooming/panning
• frame extraction
• lightweight & no dependencies
• Immersive 360° picture & video including stereo VR

Use cases

• Photo gallery / picture viewer (zooming & panning of image)
• Camera apps (live preview, live shader effects, digital zoom, capturing of zoomed/effect-modified result)
• Video players (image adjustments through shader effects, e.g. brightness/contrast)
• 360°/immersive/VR content display/playback with sensor/gyroscope viewport (tutorial, sample app)

Requirements

• Android API 15+ (Android 4.0.3 Ice Cream Sandwich)

Changelog

• 1.3.0: add possibility to subscribe to Effect and Parameter changes
• BaseEffect abstract class made public (makes it easier to write custom effects)
• getRotationMatrix method added to the ImmersiveEffect (allows usage of the rotation for other purposes, e.g. immersive audio)
• addListener/removeListener methods added to all effects
• addListener/removeListener methods to all parameters
• v1.2.1: privacy policy added to app (required for the Play Store)
• v1.2.0: add watermark and stack effects
• WatermarkEffect added to overlay a watermark (e.g. a station logo)
• StackEffect added to combine multiple effects
• ColorEffect moved from demo to core module
• InterlaceEffect added to demo
• bugfixes in demo app
• v1.1.0: Immersive effect classes streamlined, touch/scroll navigation for immersive render effect
• EquirectangularSphereEffect renamed to ImmersiveEffect
• SensorRotationNavigation renamed to ImmersiveSensorNavigation
• ImmersiveTouchNavigation added
• v1.0.0: initial release

Support

For questions and issues, please open an issue on the issue tracker. Commercial support, development and consultation is available through Protyposis Multimedia Solutions.

Usage

Include net.protyposis.android.spectaculum.SpectaculumView into your layout, get a reference, and attach a listener to the InputSurfaceHolder to get a reference to the Surface and SurfaceTexture. They can then be passed to any component that draws onto them. Call onPause and onResume from the equivalent activity lifecycle methods. That's all.

// Get reference to the Spectaculum view
SpectaculumView spectaculum = (SpectaculumView) findViewById(R.id.spectaculum);
// Attach input surface listener
spectaculum.getInputHolder().addCallback(new InputSurfaceHolder.Callback() {
    @Override
    public void surfaceCreated(InputSurfaceHolder holder) {
        // Use one of these to draw to the view
        Surface inputSurface = holder.getSurface();
        SurfaceTexture inputSurfaceTexture = holder.getSurfaceTexture();
    }
    @Override
    public void surfaceDestroyed(InputSurfaceHolder holder) {}
});

For self-contained subclasses that handle the drawing on their own, see the modules below.

Important: SpectaculumView needs to receive the onPause/onResume lifecycle events from the activity it is part of:

public class SpectaculumActivity extends Activity {
    private SpectaculumView mSpectaculum;
    public void onCreate(Bundle savedInstanceState) {
        mSpectaculum = (SpectaculumView) findViewById(R.id.spectaculum);
    }
    @Override
    protected void onPause() {
        super.onPause();
        mSpectaculum.onPause();
    }
    @Override
    protected void onResume() {
        super.onResume();
        mSpectaculum.onResume();
    }
}

API

This is a quick overview of the most important methods. Please check the JavaDoc for further explanation and the demo app for practical usage.

Effects

Shader effects implement the Effect interface or one of the supplied abstract base classes (BaseEffect, ShaderEffect) and apply their effect to a source texture that is written to a target framebuffer (but all of this is handled by the effects framework).

ShaderEffect can be extended to create simple shader effects with a single shader program. Such an effect consists of 3 components: a fragment shader (and optionally a vertex shader), a ShaderProgram or almost always a TextureShaderProgram, and the ShaderEffect. The ShaderEffect implements the initShaderProgram() method and initializes the ShaderProgram with a shader file (or string) and sets up optional parameters. Simple example effects are the SimpleToonEffect (most basic effect without parameters) and ContrastBrightnessAdjustmentEffect (basic effect with parameters). The demo code also contains an example called ColorFilterEffect.

Parameters can be added in the constructor or initShaderProgram(), depending on the use case. A user of the effect can query the parameters at runtime and change them dynamically. For example, the demo app dynamically constructs a panel overlay with all parameters that the user can directly adjust. Parameters basically implement the Parameter interface, but usually extend the BaseParameter class. There are four parameter implementations for basic data types available: BooleanParameter, EnumParameter, FloatParameter and IntegerParameter.

Advanced effects can be implemented by directly extending BaseEffect. These effects are not constrained to a single shader and can therefore contain multiple shader programs, textures, framebuffers and subeffects and combine them to create complex effects. See FlowAbsEffect and QrMarkerEffect for examples.

Effect examples

From left to right: No effect, Sobel edge detect, color filter, contrast/brightness adjustment, watermark, flowabs, stacked simple toon + watermark

Immersive effects from left to right: No effect, immersive 360°, stacked immersive 360° + watermark, immersive 360° stereo 3D

List of included effects

• Spectaculum (Core)
  • ColorFilterEffect (RGB color filter)
  • ContrastBrightnessAdjustmentEffect
  • FlipEffect (none, vertical, horizontal, both)
  • KernelBlurEffect (3x3 blur kernel)
  • KernelEdgeDetectEffect (3x3 edge detect kernel)
  • KernelEmbossEffect (3x3 emboss kernel)
  • KernelGaussBlurEffect (3x3 gaussian blur kernel)
  • KernelSharpenEffect (3x3 sharpen kernel)
  • SimpleToonEffect (simple cartoon effect with sobel edge detection and color quantization)
  • SobelEffect (Sobel edge detection)
  • StackEffect (combines multiple effects together)
  • WatermarkEffect (adds a watermark, e.g. a station logo)
• Spectaculum-Effect-FlowAbs
  • FlowAbsEffect (flowabs) including FlowAbsBilateralFilterEffect, FlowAbsDOGEffect (difference of gaussians), FlowAbsGaussEffect, FlowAbsTangentFlowMapEffect
• Spectaculum-Effect-Immersive
  • ImmersiveEffect (360° monoscopic and stereoscopic equirectangular sphere projection rendering)
• Spectaculum-Effect-QrMarker
  • QrMarkerEffect (QR code detector)
  • QrMarkerEffect.CannyEdgeEffect (Canny edge detection)
• Spectaculum-Demo
  • InterlaceEffect (scanline effect)

Most of these effects come with adjustable parameters. Check the source code to see which parameters are available, or the demo app to play with them.

Immersive / 360° / VR

Rendering 360° content is just another shader effect. To render equirectangular 360° content, include the Spectaculum-Effect-Immersive module and apply the ImmersiveEffect. The effect is parameterized, and the first three float parameters control the rotation around the X/Y/Z axes. The fourth parameter is an enum parameter of type ImmersiveEffect.Mode, which can be MONO for monocular content, or STEREO_SBS/STEREO_TAB for stereoscopic 3D content in side by side or top and bottom layout. Unfortunately I cannot test the 3D modes due to lack of hardware.

Of course, a 3D VR view needs to change its viewport depending on where the consumer or device is looking. The effect provides a setRotationMatrix method which accepts a 4x4 rotation matrix, which is recommended for frequent updates over the XYZ parameters.

This module also includes additional classes for viewport navigation that can be attached to the effect. The ImmersiveTouchNavigation class handles navigation in the viewport through touch scroll gestures. It adds a boolean parameter to switch the touch rotation on/off (default is off), and an enum parameter to set the render mode (mono, stereo). The ImmersiveSensorNavigation class handles the viewport rotation through the device's rotation sensor. It adds a boolean parameter to switch the sensor rotation on/off ()default is off). ImmersiveSensorNavigation is experimental and does not work correctly yet, because I could not figure out how to map the sensor coordinate system. Any help is appreciated!

Maven

Release builds of the library are available in the JCenter Maven repository. Available versions are listed in the changelog or the Bintray repository.

repositories {
    ...
    jcenter()
}

dependencies {
    ...
    compile 'net.protyposis.android.spectaculum:spectaculum:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-camera:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-image:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-mediaplayer:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-mediaplayerextended:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-effect-flowabs:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-effect-immersive:1.2.0'
    compile 'net.protyposis.android.spectaculum:spectaculum-effect-qrmarker:1.2.0'
}

Modules

Spectaculum-Core

Just the main package. Contains SpectaculumView and a few shader effects.

Spectaculum-Camera

Contains CameraView which provides a live preview of the camera. Adds methods supportsCameraSwitch() to check if the device has multiple cameras, and switchCamera() to switch to the next camera (usually front to back / back to front).

Spectaculum-Image

Contains ImageView to display a bitmap through setImageBitmap(Bitmap).

Spectaculum-MediaPlayer

Contains VideoView, a video playback view compatible with Android's VideoView. It implements only a subset of Android's methods, mostly playback control methods (e.g. subtitles are not supported).

Spectaculum-MediaPlayerExtended

Contains MediaPlayerExtendedView, a spectaculumized version of MediaPlayer-Extended's VideoView, which is also an API-compatible alternative to Android's VideoView with enhanced functionality (playback speed adjustment, frame-exact seek, DASH playback).

Spectaculum-Effect-Immersive

Adds the ImmersiveEffect and viewport navigation classes for 360°/VR/3D content as described above.

Spectaculum-Effect-FlowAbs

This module adds the FlowAbs shader effect and demonstrates the possibility to construct and use very elaborate shader effects. It also offers various sub-effects that the flowabs-effect is composed of, including (flow-based) difference of Gaussians, color quantization and a tangent flow map. This is mostly a technical demo, not for productive use.

Spectaculum-Effect-QrMarker

This module is another example of an effect composed of multiple shaders. It is taken from QrMarker and provides a rather pointless and very slow QR marker identification effect, and a nice Canny edge detection effect. Again a technical demo, not for productive use.

Spectaculum-Demo

An app that demonstrates all the main functionality of the modules (different views and effects) and serves as an example on how they can be used. Also contains example code on how to use MediaPlayer and ExoPlayer directly with SpectaculumView.

Download the Spectaculum Demo from Google Play.

Known Issues

• Effect-FlowAbs: The OrientationAlignedBilateralFilterShaderProgram / FlowAbsBilateralFilterEffect does not work correctly for some unknown reason and is deactivated in the FlowAbs effect, making it slightly less fancy

Device specific:

• Camera: preview aspect ratio is slightly off on the Nexus 7 2013 back camera (seems to be a system bug)
• Effect-FlowAbs: Not working on Tegra devices because shaders contain dynamic loops

License

Copyright (C) 2014, 2015, 2016 Mario Guggenberger [email protected]. Released under the Apache 2.0 license, except for Spectaculum-Effect-FlowAbs which is licensed under the GPLv3. Documentation and artwork licensed under CC BY-SA 4.0.