Adding a Source – Failover

You can combine two or more Sources to create a unified Failover Source. Whenever one of the Sources is disrupted (or interrupted completely), ZEN Master will seamlessly merge the streams, retrieving the missing packets from the alternate Source.

The streams can be merged using one of the following methods:

• Stream Switch - ZEN Master treats the streams as different and does not try to merge them. If packets are missing from one stream, ZEN Master will switch to the alternate stream.
• Content Hitless Merge - if the Sources do not have RTP headers, Zixi will merge the streams using the MPEG-TS payload content itself. This method uses Zixi's patented DNA like sequencing algorithm to fill in the missing packets.
• SMPTE 2022-7 Hitless - if the Sources use RTP, and the RTP headers were maintained, Zixi can use the SMPTE 2022-7 to compare and merge the two streams. This option uses the sequence and the timestamp in the packets to compare which packets are missing, providing optimal results.

Zero Bitrate and Offline events always trigger failover. For v15+ Broadcasters, you can configure additional rules to failover for other content quality errors (e.g. P1 Transport Events, Frozen Video etc.) that occur beyond a specified threshold.

The following procedure describes how to add a Failover Source in ZEN Master. In addition, an explanation of each of the configuration parameters is given in the table below.

To add a Failover Source:

1. In the main navigation, click Sources.
2. Click + Add.
   The Select Source Type window is displayed.
   
3. Click on Failover.
   The Create New Failover window opens.
   
4. In the Info section, in the Name field, enter a unique name to identify this Source in the system.
5. In the Tags field, select one or more of the predefined Tags from the drop-down list.
   Tags are used for access control. By selecting a Tag, you are relating this Source to that Tag. Tags are associated with certain roles (through the Users & Roles screen), and roles, in turn, are associated with users and user groups. If you would like to create a new Tag, see Creating a Tag.

6. In the Events Configuration Profile field, select from the dropdown list the profile that you would like to apply to this Source. By default, the Default profile is applied.

   Events Profiles consist of a series of rules that determine precisely how various types of Events are logged and what results they trigger.

7. The Source's location is used to represent it in the proper position in the Map display (see Maps). By default, the location is determined automatically based on the Source's IP. If you would like to manually set the location of the Source (Optional), use the following procedure:
   1. Click on the Edit icon next to the Location field.
      A search box is shown below the Location field.
   2. Enter identifying info for the location (e.g. street address, zip code, coordinates).
      Search results are displayed below the search box as you type.
   3. Click on the desired location.
      The selected location populates the Location field.

8. In the Input section, in the Sources field, click the + button next to two or more Sources that you would like to include in the failover group.

   If the selected Sources are not binary identical then there will be a one second delay in the failover to the backup source.

9. For each source, configure the following:
   1. Under Priority - to designate a Source as a preferred Source, select the Preferred checkbox or to to designate a Source as a Slate, select the Slate checkbox. Preferred source take precedence over non-preferred source (i.e., sources that were not marked as preferred). Slate sources are sources that stream slate content, which is content that can be served when there are gaps in a live stream ad break that cannot be filled with a dynamically-served ad.   

   2. Under Min Bitrate (Kbps) - specify the bitrate threshold level in Kbps to perform the failover (i.e., if the bitrate drops under the threshold the failover will take place).

      If you would like to remove a selected Source, click on the  button next to the Source.

10. For Failover Mode, select the desired mode for failing over/merging the streams. Options are:
    • Stream Switch – ZEN Master treats the streams as different and does not try to merge them. If packets are missing from one stream, ZEN Master will switch to the alternate stream. If the payload is not binary identical it will fall back to Stream Switch.
    • Content Hitless - if the Sources do not have RTP headers, Zixi will merge the streams using the MPEG-TS payload content itself. This method uses Zixi's patented DNA like sequencing algorithm to fill in the missing packets.
    • SMPTE 2022-7 Hitless - if the Sources use RTP, and the RTP headers were maintained, Zixi can use the SMPTE 2022-7 to compare and merge the two streams. This option uses the sequence and the timestamp in the packets to compare which packets are missing, providing optimal results.

11. If you have selected the Content Hitless or the SMPTE-2022-7 Hitless option, in the Merge Search Window [ms] field, enter the time frame for which the system will search for the relevant packets in a matching stream. (Default: 1000)

    If there is a latency difference between the streams, the Search Window value should be higher than the latency difference between the streams.

    If Merge Mode is Disabled, the time period specified in the Search Window  field defines how long the stream will wait for packet before attempting to switch between streams.

12. If you have selected the Stream Switch option, in the Zero Bitrate/Offline Detection Window [ms] field, enter the time frame with zero bitrate or offline state after which the system will perform the failover
13. Failover Rules allow you to set rules for what type of error and how often they occur to change to the failover stream. If you would like to change the rules, use the following procedure (supported for Broadcasters v15+):
    1. Select the Checkbox next to the error for which you would like to change the error rules. Options are: P1 Transport Errors, Frozen Video, Blank Picture, Silent Audio, and Low Bitrate. (Zero Bitrate/Offline is set as selected on a fixed basis.)
    2. Set the parameters for:
       • Sample Duration - the duration (in seconds) of the window of each sample.
       • Samples - the number of samples that you check in each sampling.

       • Samples Threshold - the number of error results (within a sampling) that trigger the failover. 

       • PID - scopes the rule to a specific PID in the transport stream (e.g., if you have multiple audio programs it will only apply the rule to the specified one)

         For example, if you set Sample Duration=10, Samples=6, Samples Threshold=2: then if during a 60 second time period an error occurs during 2 out of the 6 ten-second windows, failover will be triggered.

14. In the Configuration section, in the Ingest Cluster field, select from the drop-down list the Ingest Cluster on which the Sources that you would like to group together are located.

15. In the Target Broadcaster/s field, select your broadcaster preferences. Either select a specific broadcaster (for Manual clusters) OR specify your preference for using primary vs. backup broadcasters, options are: Prefer primary Broadcasters, Primary Broadcasters only, Backup Broadcasters only or Prefer Backup Broadcasters.

    For managed clusters, it is not recommended to select a specific broadcaster, as the broadcaster’s availability may not remain stable.

16. If you would like to generate email notifications whenever a PID change is detected, select the Alert on PID changes checkbox.

17. Select the Log SCTE-35 checkbox if you would like to log SCTE-35 markers for this stream.

    To log SCTE-35 markers, SCTE-35 needs to be enabled both on the Ingest Cluster and on the Source configuration.

    SCTE-35 logging is only available for v13.1.40262+ Broadcasters.

18. If Log SCTE-35 was selected, you can choose to receive alerts for SCTE-35 markers by selecting the Enable SCTE-35 alerts checkbox.
19. In the Public Output field, if you would like to allow access to the stream directly through a supported player, select the Allow checkbox.
20. If you selected the Allow checkbox, then enter a password in the Password field to authenticate access to the stream.

21. If you would like to enable Content Analysis (CQA), verify that the Enable Content Analysis checkbox is selected (default).

    It is generally recommended to enable Content Analysis unless it is causing technical problems. If Content Analysis is disabled, the system won't detect errors in the video and audio content of the stream and won't generate warnings for such events. Even when Content Analysis is disabled the system will still detect TR101 errors in the stream.

22. If you would like to monitor CEA-608/CEA-708 closed captions, select the Monitor CEA-608/CEA-708 checkbox.

    Monitoring CEA-608/CEA-708 Closed Captions requires using a version 14.4 or newer Broadcaster.

23. If you would like to disable auto-pull, select the Disable Auto-Pull checkbox.

    Disabling Auto-Pull will lock Channels using this Source to the Source's Ingest Cluster and Targets to the Source's current active Broadcaster. This setting only applies to Pass-Through Channels.

24. If Auto-Pull is enabled, in the Auto-Pull Latency [ms] field, specify the desired latency for auto-pull actions. (Default: 1000 ms)

25. In the WebRTC Playback section, select the desired playback method.

    WebRTC enables you to monitor the stream via web browsers and mobile applications with real-time communication (RTC). WebRTC needs to be activated independently for each Source in ZEN Master (by default it is disabled).

    
    Select one of the following options:

    • Disabled - users will not be able to view this Source using WebRTC (default)
    • Pass-Through - passes the stream through to WebRTC without transcoding. This option can only be used if the input stream is already in a WebRTC compliant format (video - H.264 codec with Baseline or Main profile and no B frames, Audio - Opus codec).

    • Auto - ZEN Master automatically transcodes the stream using the optimal profile for each specific user. In this option Zixi Broadcaster will decode the video, transforming it into RAW frames, which will be passed to the WebRTC library, where it will be re-encoded for each individual user.

      Best quality. However, if monitored by multiple users this can consume extensive resources.

    • 1080p/720p/480p/360p - ZEN Master transcodes the stream to a WebRTC compliant  profile with the specified resolution. The transcoding is done by Zixi Broadcaster's transcoder.

      This method requires a Zixi Broadcaster with X264 or Nvidia for transcoding. This method allows many users to monitor the stream without encoding separately for each individual user, which saves CPU capacity. Since there is only one stream version, it is recommended to encode to a low resolution and low bitrate so that the stream will be viewable by all users monitoring the stream.

26. In the Initial State section, select the Disabled checkbox if you would like to disable this Source upon initial configuration.

27. Select the Muted checkbox if you would like to disable error/recovery notifications for this Source upon initial configuration.

    You can Enable/Unmute the Source at any time.

28. If you would like to adjust the advanced settings, click Advanced.
    The Advanced settings fields are displayed:
    
29. If you would like to track usage for this Source, enter your Billing Code details. See Billing Codes.
30. In the Auto Pull MTU field, specify the maximum transmission unit (largest data packet that a network-connected device will accept) in bytes for the Auto Pull.
31. To set the detection of frozen video, in the Frozen Video Minimal Duration field, specify the default minimal duration in seconds of frozen video after which the Broadcaster will trigger a Frozen video alert. Default: 10 seconds. This setting is supported for Broadcaster version 16.3 and higher. 
32. To set the detection of blank video, in the Blank Video Minimal Duration field, specify the default minimal duration in seconds of blank video after which the Broadcaster will trigger a Blank video alert. Default: 10 seconds. This setting is supported for Broadcaster version 16.3 and higher. 
33. Click Save.

Hitless Failover Source Configuration Parameters Table