MetalPetal

An image processing framework based on Metal.

• Design Overview
  • Goals
  • Core Components
    • MTIContext
    • MTIImage
    • MTIFilter
    • MTIKernel
  • Alpha Type Handling
  • Optimizations
  • Concurrency Considerations
  • Advantages over Core Image
  • Extensions
    • Working with SceneKit
    • Working with SpriteKit
    • Working with Core Image
    • Working with JavaScript
    • Texture Loader
• Builtin Filters
• Example Code
  • Create a MTIImage
  • Apply a Filter
  • Render a MTIImage
  • Display a MTIImage
  • Connect Filters (Swift)
  • Process Video Files (with VideoIO)
  • Process Live Video (with VideoIO)
• Quick Look Debug Support
• Best Practices
• Build Custom Filter
  • Shader Function Arguments Encoding
  • Simple Single Input / Output Filters
  • Fully Custom Filters
  • Multiple Draw Calls in One Render Pass
  • Custom Vertex Data
  • Custom Processing Module
• Install
  • CocoaPods
    • Sub-pod Swift
    • Sub-pod AppleSilicon
  • Swift Package Manager
• iOS Simulator Support
• Trivia
• Contribute
• License

Design Overview

MetalPetal is an image processing framework based on Metal designed to provide real-time processing for still image and video with easy to use programming interfaces.

This chapter covers the key concepts of MetalPetal, and will help you to get a better understanding of its design, implementation, performance implications and best practices.

Goals

MetalPetal is designed with the following goals in mind.

• Easy to use API

  Provides convenience APIs and avoids common pitfalls.

• Performance

  Use CPU, GPU and memory efficiently.

• Extensibility

  Easy to create custom filters as well as plugin your custom image processing unit.

• Swifty

  Provides a fluid experience for Swift programmers.

Core Components

Some of the core concepts of MetalPetal are very similar to those in Apple's Core Image framework.

MTIContext

Provides an evaluation context for rendering MTIImages. It also stores a lot of caches and state information, so it's more efficient to reuse a context whenever possible.

MTIImage

A MTIImage object is a representation of an image to be processed or produced. It does directly represent image bitmap data instead it has all the information necessary to produce an image or more precisely a MTLTexture. It consists of two parts, a recipe of how to produce the texture (MTIImagePromise) and other information such as how a context caches the image (cachePolicy), and how the texture should be sampled (samplerDescriptor).

MTIFilter

A MTIFilter represents an image processing effect and any parameters that control that effect. It produces a MTIImage object as output. To use a filter, you create a filter object, set its input images and parameters, and then access its output image. Typically, a filter class owns a static kernel (MTIKernel), when you access its outputImage property, it asks the kernel with the input images and parameters to produce an output MTIImage.

MTIKernel

A MTIKernel represents an image processing routine. MTIKernel is responsible for creating the cooresponding render or compute pipeline state for the filter, as well as building the MTIImagePromise for a MTIImage.

Alpha Type Handling

If an alpha channel is used in an image, there are two common representations that are available: unpremultiplied (straight/unassociated) alpha, and premultiplied (associated) alpha.

With unpremultiplied alpha, the RGB components represent the color of the pixel, disregarding its opacity.

With premultiplied alpha, the RGB components represent the color of the pixel, adjusted for its opacity by multiplication.

Most of the filters in MetalPetal accept unpremultiplied alpha and opaque images and output unpremultiplied alpha images. Some filters, such as MTIMultilayerCompositingFilter accepts both unpremultiplied/premultiplied alpha images.

MetalPetal handles alpha type explicitly. You are responsible for providing the correct alpha type during image creation.

There are three alpha types in MetalPetal.

MTIAlphaType.nonPremultiplied: the alpha value in the image is not premultiplied.

MTIAlphaType.premultiplied: the alpha value in the image is premultiplied.

MTIAlphaType.alphaIsOne: there's no alpha channel in the image or the image is opaque.

Typically, CGImage, CVPixelBuffer, CIImage objects have premultiplied alpha channel. MTIAlphaType.alphaIsOne is strongly recommanded if the image is opaque, e.g. a CVPixelBuffer from camera feed, or a CGImage loaded from a jpg file.

You can call unpremultiplyingAlpha() or premultiplyingAlpha() on a MTIImage to convert the alpha type of the image.

For performance reasons, alpha type validation only happens in debug build.

For more about alpha types and alpha compositing, please refer to this amazing interactive article by Bartosz Ciechanowski.

Optimizations

MetalPetal does a lot of optimizations for you under the hood.

It automatically caches functions, kernel states, samplers, etc.

Before rendering, MetalPetal can look into your image render graph and figure out the minimal number of intermedinate textures needed to do the rendering, saving memory, energy and time.

It can also re-organize the image render graph if multiple “recipes” can be concatenated to eliminate redundant render passes. (MTIContext.isRenderGraphOptimizationEnabled)

Concurrency Considerations

MTIImage objects are immutable, which means they can be shared safely among threads.

However, MTIFilter objects are mutable and thus cannot be shared safely among threads.

A MTIContext contains a lot of states and caches. There's a thread-safe mechanism for MTIContext objects, making it safe to share a MTIContext object among threads.

Advantages over Core Image

• Fully customizable vertex and fragment functions.

• MRT (Multiple Render Targets) support.

• Generally better performance. (Detailed benchmark data needed)

Extensions

Working with SceneKit

You can use MTISCNSceneRenderer to generate MTIImages from a SCNScene. You may want to handle the SceneKit renderer's linear RGB color space, see issue #76 The image from SceneKit is darker than normal.

Working with SpriteKit

You can use MTISKSceneRenderer to generate MTIImages from a SKScene.

Working with Core Image

You can create MTIImages from CIImages.

You can render a MTIImage to a CIImage using a MTIContext.

You can use a CIFilter directly with MTICoreImageKernel or the MTICoreImageUnaryFilter class. (Swift Only)

Working with JavaScript

See MetalPetalJS

With MetalPetalJS you can create render pipelines and filters using JavaScript, making it possible to download your filters/renderers from "the cloud".

Texture Loader

It is recommanded that you use APIs that accept MTICGImageLoadingOptions to load CGImages and images from URL, instead of using APIs that accept MTKTextureLoaderOption.

When you use APIs that accpet MTKTextureLoaderOption, MetalPetal, by default, uses MTIDefaultTextureLoader to load CGImages, images from URL, and named images. MTIDefaultTextureLoader uses MTKTextureLoader internally and has some workarounds for MTKTextureLoader's inconsistencies and bugs at a small performance cost. You can also create your own texture loader by implementing the MTITextureLoader protocol. Then assign your texture loader class to MTIContextOptions.textureLoaderClass when creating a MTIContext.

Builtin Filters

• Color Matrix

• Color Lookup

  Uses an color lookup table to remap the colors in an image.

• Opacity

• Exposure

• Saturation

• Brightness

• Contrast

• Color Invert

• Vibrance

  Adjusts the saturation of an image while keeping pleasing skin tones.

• RGB Tone Curve

• Blend Modes

  • Normal
  • Multiply
  • Overlay
  • Screen
  • Hard Light
  • Soft Light
  • Darken
  • Lighten
  • Color Dodge
  • Add (Linear Dodge)
  • Color Burn
  • Linear Burn
  • Lighter Color
  • Darker Color
  • Vivid Light
  • Linear Light
  • Pin Light
  • Hard Mix
  • Difference
  • Exclusion
  • Subtract
  • Divide
  • Hue
  • Saturation
  • Color
  • Luminosity
  • ColorLookup512x512
  • Custom

• Blend with Mask

• Transform

• Crop

• Pixellate

• Multilayer Composite

• MPS Convolution

• MPS Gaussian Blur

• MPS Definition

• MPS Sobel

• MPS Unsharp Mask

• MPS Box Blur

• High Pass Skin Smoothing

• CLAHE (Contrast-Limited Adaptive Histogram Equalization)

• Lens Blur (Hexagonal Bokeh Blur)

• Surface Blur

• Bulge Distortion

• Chroma Key Blend

• Color Halftone

• Dot Screen

• All Core Image Filters

Example Code

Create a MTIImage

You can create a MTIImage object from nearly any source of image data, including:

• URLs referencing image files to be loaded
• Metal textures
• CoreVideo image or pixel buffers (CVImageBufferRef or CVPixelBufferRef)
• Image bitmap data in memory
• Texture data from a given texture or image asset name
• Core Image CIImage objects
• MDLTexture objects
• SceneKit and SpriteKit scenes

let imageFromCGImage = MTIImage(cgImage: cgImage, isOpaque: true)

let imageFromCIImage = MTIImage(ciImage: ciImage)

let imageFromCoreVideoPixelBuffer = MTIImage(cvPixelBuffer: pixelBuffer, alphaType: .alphaIsOne)

let imageFromContentsOfURL = MTIImage(contentsOf: url)

// unpremultiply alpha if needed
let unpremultipliedAlphaImage = image.unpremultiplyingAlpha()

Apply a Filter

let inputImage = ...

let filter = MTISaturationFilter()
filter.saturation = 0
filter.inputImage = inputImage

let outputImage = filter.outputImage

Render a MTIImage

let options = MTIContextOptions()

guard let device = MTLCreateSystemDefaultDevice(), let context = try? MTIContext(device: device, options: options) else {
    return
}

let image: MTIImage = ...

do {
    try context.render(image, to: pixelBuffer) 
    
    //context.makeCIImage(from: image)
    
    //context.makeCGImage(from: image)
} catch {
    print(error)
}

Display a MTIImage

let imageView = MTIImageView(frame: self.view.bounds)

// You can optionally assign a `MTIContext` to the image view. If no context is assigned and `automaticallyCreatesContext` is set to `true` (the default value), a `MTIContext` is created automatically when the image view renders its content.
imageView.context = ...

imageView.image = image

If you'd like to move the GPU command encoding process out of the main thread, you can use a MTIThreadSafeImageView. You may assign a MTIImage to a MTIThreadSafeImageView in any thread.

Connect Filters (Swift)

MetalPetal has a type-safe Swift API for connecting filters. You can use => operator in FilterGraph.makeImage function to connect filters and get the output image.

Here are some examples:

let image = try? FilterGraph.makeImage { output in
    inputImage => saturationFilter => exposureFilter => output
}

let image = try? FilterGraph.makeImage { output in
    inputImage => saturationFilter => exposureFilter => contrastFilter => blendFilter.inputPorts.inputImage
    exposureFilter => blendFilter.inputPorts.inputBackgroundImage
    blendFilter => output
}

• You can connect unary filters (MTIUnaryFilter) directly using =>.

• For a filter with multiple inputs, you need to connect to one of its inputPorts.

• => operator only works in FilterGraph.makeImage method.

• One and only one filter's output can be connected to output.

Process Video Files (with VideoIO)

VideoIO is required for the following examples.

Working with AVPlayer:

import VideoIO

let context = try MTIContext(device: device)
let asset = AVAsset(url: videoURL)
let handler = MTIAsyncVideoCompositionRequestHandler(context: context, tracks: asset.tracks(withMediaType: .video)) {   request in
    return FilterGraph.makeImage { output in
        request.anySourceImage => filterA => filterB => output
    }!
}
let composition = VideoComposition(propertiesOf: asset, compositionRequestHandler: handler.handle(request:))
let playerItem = AVPlayerItem(asset: asset)
playerItem.videoComposition = composition.makeAVVideoComposition()
player.replaceCurrentItem(with: playerItem)
player.play()

Export a video:

import VideoIO

let configuration = AssetExportSession.Configuration(fileType: .mp4, videoSettings: .h264(videoSize: composition.renderSize), audioSettings: .aac(channels: 2, sampleRate: 44100, bitRate: 128 * 1000))
configuration.videoComposition = composition.makeAVVideoComposition()
self.exporter = try! AssetExportSession(asset: asset, outputURL: outputURL, configuration: configuration)
exporter.export(progress: { progress in
    
}, completion: { error in
    
})

Process Live Video (with VideoIO)

VideoIO is required for this example.

import VideoIO

// Setup Image View
let imageView = MTIImageView(frame: self.view.bounds)
...

// Setup Camera
let camera = Camera(captureSessionPreset: .hd1920x1080, configurator: .portraitFrontMirroredVideoOutput)
try camera.enableVideoDataOutput(on: DispatchQueue.main, delegate: self)
camera.videoDataOutput?.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]

...

// AVCaptureVideoDataOutputSampleBufferDelegate

let filter = MTIColorInvertFilter()

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
    }
    let inputImage = MTIImage(cvPixelBuffer: pixelBuffer, alphaType: .alphaIsOne)
    filter.inputImage = inputImage
    self.imageView.image = filter.outputImage
}

Quick Look Debug Support

If you do a Quick Look on a MTIImage, it'll show you the image graph that you constructed to produce that image.

Best Practices

• Reuse a MTIContext whenever possible.

  Contexts are heavyweight objects, so if you do create one, do so as early as possible, and reuse it each time you need to render an image.

• Use MTIImage.cachePolicy wisely.

  Use MTIImageCachePolicyTransient when you do not want to preserve the render result of an image, for example when the image is just an intermediate result in a filter chain, so the underlying texture of the render result can be reused. It is the most memory efficient option. However, when you ask the context to render a previously rendered image, it may re-render that image since its underlying texture has been reused.

  By default, a filter's output image has the transient policy.

  Use MTIImageCachePolicyPersistent when you want to prevent the underlying texture from being reused.

  By default, images created from external sources have the persistent policy.

• Understand that MTIFilter.outputImage is a compute property.

  Each time you ask a filter for its output image, the filter may give you a new output image object even if the inputs are identical with the previous call. So reuse output images whenever possible.

  For example,

  //          ╭→ filterB
  // filterA ─┤
  //          ╰→ filterC
  // 
  // filterB and filterC use filterA's output as their input.
  

  In this situation, the following solution:

  let filterOutputImage = filterA.outputImage
  filterB.inputImage = filterOutputImage
  filterC.inputImage = filterOutputImage
  

  is better than:

  filterB.inputImage = filterA.outputImage
  filterC.inputImage = filterA.outputImage
  

Build Custom Filter

If you want to include the MTIShaderLib.h in your .metal file, you need to add the path of MTIShaderLib.h file to the Metal Compiler - Header Search Paths (MTL_HEADER_SEARCH_PATHS) setting.

For example, if you use CocoaPods you can set the MTL_HEADER_SEARCH_PATHS to ${PODS_CONFIGURATION_BUILD_DIR}/MetalPetal/MetalPetal.framework/Headers or ${PODS_ROOT}/MetalPetal/Frameworks/MetalPetal/Shaders. If you use Swift Package Manager, set the MTL_HEADER_SEARCH_PATHS to $(HEADER_SEARCH_PATHS)

Shader Function Arguments Encoding

MetalPetal has a built-in mechanism to encode shader function arguments for you. You can pass the shader function arguments as name: value dictionaries to the MTIRenderPipelineKernel.apply(toInputImages:parameters:outputDescriptors:), MTIRenderCommand(kernel:geometry:images:parameters:), etc.

For example, the parameter dictionary for the metal function vibranceAdjust can be:

// Swift
let amount: Float = 1.0
let vibranceVector = float4(1, 1, 1, 1)
let parameters = ["amount": amount,
                  "vibranceVector": MTIVector(value: vibranceVector),
                  "avoidsSaturatingSkinTones": true,
                  "grayColorTransform": MTIVector(value: float3(0,0,0))]

// vibranceAdjust metal function
fragment float4 vibranceAdjust(...,
                constant float & amount [[ buffer(0) ]],
                constant float4 & vibranceVector [[ buffer(1) ]],
                constant bool & avoidsSaturatingSkinTones [[ buffer(2) ]],
                constant float3 & grayColorTransform [[ buffer(3) ]])
{
    ...
}

The shader function argument types and the coorresponding types to use in a parameter dictionary is listed below.

Simple Single Input / Output Filters

To build a custom unary filter, you can subclass MTIUnaryImageRenderingFilter and override the methods in the SubclassingHooks category. Examples: MTIPixellateFilter, MTIVibranceFilter, MTIUnpremultiplyAlphaFilter, MTIPremultiplyAlphaFilter, etc.

//Objective-C

@interface MTIPixellateFilter : MTIUnaryImageRenderingFilter

@property (nonatomic) float fractionalWidthOfAPixel;

@end

@implementation MTIPixellateFilter

- (instancetype)init {
    if (self = [super init]) {
        _fractionalWidthOfAPixel = 0.05;
    }
    return self;
}

+ (MTIFunctionDescriptor *)fragmentFunctionDescriptor {
    return [[MTIFunctionDescriptor alloc] initWithName:@"pixellateEffect" libraryURL:[bundle URLForResource:@"default" withExtension:@"metallib"]];
}

- (NSDictionary<NSString *,id> *)parameters {
    return @{@"fractionalWidthOfAPixel": @(self.fractionalWidthOfAPixel)};
}

@end

//Swift

class MTIPixellateFilter: MTIUnaryImageRenderingFilter {
    
    var fractionalWidthOfAPixel: Float = 0.05

    override var parameters: [String : Any] {
        return ["fractionalWidthOfAPixel": fractionalWidthOfAPixel]
    }
    
    override class func fragmentFunctionDescriptor() -> MTIFunctionDescriptor {
        return MTIFunctionDescriptor(name: "pixellateEffect", libraryURL: MTIDefaultLibraryURLForBundle(Bundle.main))
    }
}

Fully Custom Filters

To build more complex filters, all you need to do is create a kernel (MTIRenderPipelineKernel/MTIComputePipelineKernel/MTIMPSKernel), then apply the kernel to the input image(s). Examples: MTIChromaKeyBlendFilter, MTIBlendWithMaskFilter, MTIColorLookupFilter, etc.

@interface MTIChromaKeyBlendFilter : NSObject <MTIFilter>

@property (nonatomic, strong, nullable) MTIImage *inputImage;

@property (nonatomic, strong, nullable) MTIImage *inputBackgroundImage;

@property (nonatomic) float thresholdSensitivity;

@property (nonatomic) float smoothing;

@property (nonatomic) MTIColor color;

@end

@implementation MTIChromaKeyBlendFilter

@synthesize outputPixelFormat = _outputPixelFormat;

+ (MTIRenderPipelineKernel *)kernel {
    static MTIRenderPipelineKernel *kernel;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        kernel = [[MTIRenderPipelineKernel alloc] initWithVertexFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:MTIFilterPassthroughVertexFunctionName] fragmentFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:@"chromaKeyBlend"]];
    });
    return kernel;
}

- (instancetype)init {
    if (self = [super init]) {
        _thresholdSensitivity = 0.4;
        _smoothing = 0.1;
        _color = MTIColorMake(0.0, 1.0, 0.0, 1.0);
    }
    return self;
}

- (MTIImage *)outputImage {
    if (!self.inputImage || !self.inputBackgroundImage) {
        return nil;
    }
    return [self.class.kernel applyToInputImages:@[self.inputImage, self.inputBackgroundImage]
                                      parameters:@{@"color": [MTIVector vectorWithFloat4:(simd_float4){self.color.red, self.color.green, self.color.blue,self.color.alpha}],
                                    @"thresholdSensitivity": @(self.thresholdSensitivity),
                                               @"smoothing": @(self.smoothing)}
                         outputTextureDimensions:MTITextureDimensionsMake2DFromCGSize(self.inputImage.size)
                               outputPixelFormat:self.outputPixelFormat];
}

@end

Multiple Draw Calls in One Render Pass

You can use MTIRenderCommand to issue multiple draw calls in one render pass.

// Create a draw call with kernelA, geometryA, and imageA.
let renderCommandA = MTIRenderCommand(kernel: self.kernelA, geometry: self.geometryA, images: [imageA], parameters: [:])

// Create a draw call with kernelB, geometryB, and imageB.
let renderCommandB = MTIRenderCommand(kernel: self.kernelB, geometry: self.geometryB, images: [imageB], parameters: [:])

// Create an output descriptor
let outputDescriptor = MTIRenderPassOutputDescriptor(dimensions: MTITextureDimensions(width: outputWidth, height: outputHeight, depth: 1), pixelFormat: .bgra8Unorm, loadAction: .clear, storeAction: .store)

// Get the output images, the output image count is equal to the output descriptor count.
let images = MTIRenderCommand.images(byPerforming: [renderCommandA, renderCommandB], outputDescriptors: [outputDescriptor])

You can also create multiple output descriptors to output multiple images in one render pass (MRT, See https://en.wikipedia.org/wiki/Multiple_Render_Targets).

Custom Vertex Data

When MTIVertex cannot fit your needs, you can implement the MTIGeometry protocol to provide your custom vertex data to the command encoder.

Use the MTIRenderCommand API to issue draw calls and pass your custom MTIGeometry.

Custom Processing Module

In rare scenarios, you may want to access the underlying texture directly, use multiple MPS kernels in one render pass, do 3D rendering, or encode the render commands yourself.

MTIImagePromise protocol provides direct access to the underlying texture and the render context for a step in MetalPetal.

You can create new input sources or fully custom processing unit by implementing MTIImagePromise protocol. You will need to import an additional module to do so.

Objective-C

@import MetalPetal.Extension;

Swift

// CocoaPods
import MetalPetal.Extension

// Swift Package Manager
import MetalPetalObjectiveC.Extension

See the implementation of MTIComputePipelineKernel, MTICLAHELUTRecipe or MTIImage for example.

Install

CocoaPods

You can use CocoaPods to install the lastest version.

use_frameworks!

pod 'MetalPetal'

# Required if you are using Swift.
pod 'MetalPetal/Swift'

# Recommended if you'd like to run MetalPetal on Apple silicon Macs.
pod 'MetalPetal/AppleSilicon'

Sub-pod Swift

Provides Swift-specific additions and modifications to the Objective-C APIs to improve their mapping into Swift. Highly recommended if you are using Swift.

Sub-pod AppleSilicon

Provides the default shader library compiled in Metal Shading Language v2.3 which is required for enabling programmable blending support on Apple silicon Macs.

Swift Package Manager

Adding Package Dependencies to Your App

iOS Simulator Support

MetalPetal can run on Simulator with Xcode 11+ and macOS 10.15+.

MetalPerformanceShaders.framework is not available on Simulator, so filters that rely on MetalPerformanceShaders, such as MTIMPSGaussianBlurFilter, MTICLAHEFilter, do not work.

Simulator supports fewer features or different implementation limits than an actual Apple GPU. See Developing Metal Apps that Run in Simulator for detail.

Trivia

Why Objective-C?

Contribute

Thank you for considering contributing to MetalPetal. Please read our Contributing Guidelines.

License

MetalPetal is MIT-licensed. LICENSE

The files in the /MetalPetalDemo directory are licensed under a separate license. LICENSE.md

Documentation is licensed CC-BY-4.0.