ISG Advanced Options (And Edge Transcoder)

The iconik Storage Gateway has a number of advanced options that you can tune by editing the configuration file config.ini. Most of these options also apply to the iconik Edge Transcoder, except the ones related to scanning

• Back up config.ini before you make these changes.
• Configuration options are key value pairs such as proxy-resolution=1920x1080
• Configuration options should be on individual lines.
• Restart the ISG after making the changes

config.ini location

Please make sure if you are editing on Windows the line endings are of a Unix type

Setting Proxy Format Specification

It's possible to change the proxy format that the ISG will generate for video media. Please note that setting the bitrate too high or low can have a negative impact on the perceived performance of the Web Interface.

Proxy resolution

By setting the key proxy-resolution you can define the max-dimensions of the video proxy created.

• Minimum value: 640x360
• Default value: 1280x720
• Maximum value: 1920x1080

For example:

proxy-resolution=1920x1080

Proxy bitrate

By setting the key proxy-bitrate you can define the max bitrate of the proxy created. Please be aware that setting this higher can cause large file sizes for your proxy.

• Minimum value: 200000
• Default value: 3000000
• Maximum value: 99000000

For example:

proxy-bitrate=4000000

Image proxy resolution

By setting the key image-proxy-resolution you can define the max-dimensions of the image proxy created.

• Minimum value: 180x180
• Default value: 2048x2048
• Maximum value: 16000x16000

For example:

image-proxy-resolution=2048x2048

For ISG proxy resolution can be also set on the image transcoding profile.

Disable HDR to SDR conversion for keyframes

ISG and Iconik Edge Transcoder will try to use HDR to SDR conversion options for keyframes with ffmpeg. You can disable this behaviour by setting the key enable-hdr-to-sdr-keyframe-conversion to false

enable-hdr-to-sdr-keyframe-conversion = false

Controlling file transfers and transcodes

Number of parallel uploads and downloads

By setting the key file-upload-parallel-uploads-num you can control how many simultaneous uploads the ISG will do. The default is 4 By setting the key file-upload-parallel-uploads-threads-num you can control how many parallel chunks for each upload is used. The default is 5 By setting the key upload-chunk-size you can control the size of parallel chunks for each upload in bytes. The default is 104857600 (100 Mb)

file-upload-parallel-uploads-num = 4

file-upload-parallel-uploads-threads-num = 5

upload-chunk-size = 104857600

By setting the key file-download-parallel-downloads-num you can control how many simultaneous downloads the ISG will do. The default is 4 By setting the key file-download-parallel-downloads-threads-num you can control how many parallel chunks for each download is used. The default is 5 By setting the key download-chunk-size you can control the size of parallel chunks for each download in bytes. The default is 104857600 (100 Mb)

file-download-parallel-downloads-num = 4

file-download-parallel-downloads-threads-num = 5

download-chunk-size = 104857600

Number of parallel transcodes

By setting the key max-transcoding-jobs you can control the number of simultaneous local transcode jobs for each attached transcoder.

max-transcoding-jobs = 5

This should be roughly the number of processing cores of your machine divided by 4. So if you 8 cores in your processor, the number should be 2.

A set of config properties was added to control the number of simultaneous local transcode jobs per transcoder type. Default value is set from max-transcoding-jobs.

imagemagick-max-transcoding-jobs = 5

ffmpeg-max-transcoding-jobs = 5

vantage-max-transcoding-jobs = 5

redline-max-transcoding-jobs = 5

watch-folder-max-transcoding-jobs = 5

editready-max-transcoding-jobs = 5

Metadata extraction is responsible for determining file's media type and what transcoder must process it.

mediainfo-max-extraction-jobs = 5

If there's a large amount of fast transcode jobs, like small images or short video clips, it might be beneficial to increase the number of jobs for mediainfo extraction.

Increasing number of parallel jobs will also increase CPU and I/O load.

Enabling debug

Debug is enabled by setting the option debug to true

debug = true

Mediainfo timeouts

If you get timeouts reading media info you can increase that, the default is 60s

extract-mediainfo-timeout = 180

• Minimum value: 1
• Default value: 60
• Maximum value: 3600

Extracting metadata

In addition to mediainfo, the ISG and Edge Transcoder applications will attempt to read additional metadata (XMP, EXIF, IPTC). They use the same timeout as for mediainfo. You can disable metadata extraction with

additional-metadata-extraction = false

You can control how much data is read from a file when the Edge Transcoder extracts metadata by setting the key extract-metadata-chunk-size to a number of bytes. For example

extract-metadata-chunk-size = 10485760

• Minimum value: 1048576
• Default value: 8388608
• Maximum value: 104857600

The default value is 8388608 (8 Mb).

Control the number of checksum workers

Checksumming is both CPU and I/O intense. You can control the number of parallel workers doing checksumming. The default is 5 or the number of CPUs on your server, whichever is lower

checksum-max-workers = 10

You can also enable and disable it (but it is required for "Consider identical files the same")

enable-checksum-calculation = true

Enable checksum calculation based on modification time

The ISG will consider a file changed if its size changed, updating its checksum if checksum calculation is enabled. If your content changes without changing file size you can trigger checksum calculation by enabling modification time comparison.

checksum-on-modification-time-change = true

Disable the UUID-suffix on uploaded files

By default the ISG will add a UUID to the end of the file name when uploading to a cloud storage to make sure it does not conflict with any existing files. This can be disabled if you know that only this ISG will write to that cloud storage.

auto-generate-filename-for-remote-upload = false

Overriding format detection

Some times you might have files that gets identified as the wrong type. You can override this by file extension by providing a list:

image-extensions-override=.cr3,.raf,.psd,.tiff

video-extensions-override=.mp4,.mxf

audio-extensions-override=.wav,.m4a

This means that regardless of what mediainfo returns they will be sent to the transcoder that handles the type of format you have specified.

Some formats require to flatten layers, this list can be modified.

imagemagick-layer-extensions=.ai,.tif,.tiff,.png,.psd,.pdf,.psb,.gif,.eps

For Edge transcoder some formats also require downloads, since they can't be read over https.

image-extensions-requiring-download=.rw2,.arw,.dng,.pdf,.raf,.cr3,.cr2,.heic

Overriding proxy and keyframe storage

You can override the proxy and keyframe storage used for an ISG by putting the follwing in the config.ini file:

proxy-storage-id=<storage_id>

keyframe-storage-id=<storage_id>

ImageMagick Custom Path

If you have installed ImageMagick outside of the system path it can be useful to specify a path to imageMagick configuration files:

image-magick-config = /custom/path

This path should be where policy.xml, delegates.xml etc. is located

Use embedded JPGs for RAW formats

Many RAW formats contain an embedded image that ISG will extract and use to create a proxy. Extracting the embedded image requires exiftool to be installed and is enabled by default. You can disable this behaviour by setting

use-embedded-raw-images = false

You can configure which file extensions the ISG considers RAW formats by this setting (default in the example)

raw-image-extensions = .rw2,.arw,.dng,.cr2,.cr3,.raf,.nef,.3fr,.x3f,.nrw

Apply custom RGB conversion profiles (ICC)

If you have for example TIFF files in CMYK that requires a specific ICC to convert to RGB, you can specify that. You can specify any number of icc files in order separated by semicolon. Note that the value is a comma separated list of extension=path1;path2

apply-image-convert-profile=.tif=/profiles/original1.icc;/profiles/export1.icc,.tiff=/profiles/original1.icc;/profiles/export1.icc

WAL mode

WAL can improve performance on large ISG installation. It is controlled by this setting.

sqlite-journal-mode = wal

To turn it off set

sqlite-journal-mode = delete

Video encoder

You can change the default h.264 encoder. For example on Apple silicon you can try this for better performance. Note that this also requires you to set the resolution and bitrate on the attached transcoder to get decent quality

video-encoder=h264_videotoolbox

To control video quality you can select the control rate mode, which is encoder-dependant. See the ffmpeg documentation. You can choose between: -q:v, -crf

video-quality-flag=-q:v

video-quality-flag-value=50

Video encoder for edit proxies

You can set custom video encoder for edit proxies

edit-proxy-video-encoder=libx264

To control video quality you can select the control rate mode, which is encoder-dependant. See the ffmpeg documentation. You can choose between: -q:v, -crf

edit-proxy-video-quality-flag=-crf

edit-proxy-video-quality-flag-value=20

Audio encoder for edit proxies

Default is AAC (aac) but for some formats it might be better to use PCM (pcm_s24le)

edit-proxy-audio-encoder=pcm_s24le

Caching proxy for Edge Transcoder

Edge Transcoder can make use of a local caching proxy to improve performance.

use-file-cache-proxy = true

Start caching proxy instructions for Amazon Linux 2 and Ubuntu.

White balance adjustment for Raw formats

You can try to add this if you have ImageMagick7 with libraw. Note that it ONLY works on IM7 compiled with libraw. This options tries to use the camera settings for White Balance

imagemagick-white-balance-extensions=.cr3,.nef

Force symlink use for all Watch Folder transcoders on ISG

If a watch folder transcode software supports working with symlinks enabling this will provide a benefit of a symlink.

force-soft-link-on-all-watch-folder-transcoders = true

By default, a hard link is used that fallbacks to copy a whole file to the watch folder if it's on another file system.

Additional Options when using LucidLink Filespaces on the ISG host

The LucidLink remote upload indicator API can be used to detect when a file is ready for ingest. In order to use it with ISG your LucidLink instance must meet the following requirements:

• LucidLink Client version 2.1.4246 or above required.
• LucidLink 2.0 Filespace or above required

To enable remote upload indicator API support in ISG set

enable-lucidlink-upload-api=true

When using multiple Filespaces the ISG needs to know the instance ids in use. Starting with build #5127 of the LucidLink Client the ids can be automatically detected, but you can also specify them as a comma-separated list.

lucidlink-instances=0,1,2

The ISG will try to detect what mount points are in use for each LucidLink instance. You can also hard-code the filespace mount points the ISG uses with the remote upload indicator API. The order of the list of mount points must match the lucidlink-instances list if using multiple filespaces.

lucidlink-mount-points=/Volumes/fs1,/Volumes/fs2,/Volumes/fs3

Custom edit proxy suffix

You can specify a suffix that will be on the filename of already generated proxy files which are stored next to the original file that will be pattern matched and stored as an Edit Proxy by iconik.

custom-proxy-suffixes=_customeditproxy

You can change the suffix that is added to the edit proxy created by local transcoders.

• Default value: _editproxy

edit-proxy-suffix=_editproxy

Update asset title when a file is renamed

You can control if an asset title should be updated if ISG discovered a renamed file.

• Default value: true

update-title-on-filename-change = true

Resume from the last scan position on application restart

You can set if scanner should resume from the last scan folders on application restart. Set to false to disable.

• Default value: true

resume-scan-on-restart = true

Troubleshooting performance

You can control max amount of parallel ingests for new files.

ingest-concurrency-value = 2

• Minimum value: 1
• Default value: 2
• Maximum value: 8

You can control max amount of parallel directory scans.

scanner-concurrency-value = 2

• Minimum value: 1
• Default value: 2
• Maximum value: 16

Increasing these values might provide a higher ingest and scan speed, but the speed can be limited by the available host resources, including network and storage performance.

You can enable verbose performance logging.

debug-performance = true

When an ISG detects a file is missing it will update the file status and remove it from collections. You can specify a minimum time required before missing file handling is executed.

missing-files-time-threshold = 60

• Minimum value: 0
• Default value: 60

You can change the size of missing files queue to affect response time of files being marked missing.

missing-files-queue-size = 500

• Minimum value: 100
• Default value: 500
• Maximum value: 1000000

You can set timeout in seconds for a single scan folder system call.

scan-folder-timeout = 600

• Minimum value: 30
• Default value: 600
• Maximum value: 3600

Learn more

• iconik Storage Gateway Overview