shaka-player

# Shaka Player Gap Jumping ## Overview Gap jumping is a feature to automatically jump over gaps (i.e. missing content) in a stream. There are several cases where content may not be present. This missing content can cause gaps in the video which will cause the player to stop, waiting for content that may never exist. Gap jumping is one of the most requested features by developers. There are two kinds of gaps we need to consider: gaps in the manifest and gaps in the media itself. Gaps in the manifest can be fixed by simply adjusting the start and end times of the segments. We already adjust the segment index to make the segments continuous. Gaps in the media cannot be detected early (since the gaps may not appear in the manifest); they can only be detected once the media has been appended to the MediaSource and the browser gives us a gap in the buffered ranges. ## Background There have been a number of bugs filed asking for this feature. #### [#180](https://github.com/shaka-project/shaka-player/issues/180) One of the first bugs asking for gap jumping. The author talks about how live streams can commonly lose source content (e.g. from network outages or weather). This can cause the packagers to introduce gaps in either the media segments or the manifest. #### [#555](https://github.com/shaka-project/shaka-player/issues/555) This one is also asking about live streams and outages. The author discusses several options for dealing with interruptions in the live streams. The suggestions were: - Use 404s to indicate a segment does not exist on the server. - Use “short Periods” (a Period with no content, but @duration) to indicate a gap. - Fill with dummy content. It is brought up that using 404 for missing segments can cause problems since we can’t tell if a segment will never be available or is just not created yet. Also that dummy content is usually generated by the server; however, the author notes that this may not be feasible. #### [#661](https://github.com/shaka-project/shaka-player/issues/661) This content with small gaps in the media that result in playback stalling. Chrome and Firefox will actually play through the gap, but we don’t allow it since it wouldn’t work on Safari. ## Design If there are gaps in the media (e.g. missing segments), they should not appear in the manifest. If there are missing Periods, then the @start attribute should appear in the later Period to indicate a gap. If there are gaps in the manifest, they will be logged and removed by the manifest parser. Internally, the segment index will be continuous. Even if there is a whole Period missing that is minutes long, the segment index will be continuous. This means that the previous Period will be extended to cover the gap and the last segment of that Period will also be extended. Keeping the segment index continuous allows most gap jumping logic to be in Playhead. StreamingEngine will not know about gaps and will simply buffer segments like before. It will buffer segments independently of whether there are gaps in the media. Then if Playhead detects a gap in the media, it will jump it there. There are two kinds of gaps: small gaps that may not be noticeable (and may be jumped automatically), and large gaps caused by missing segments in live streams. Small gaps (less than 0.5 seconds) should be jumped automatically on all browsers to provide seamless playback. Larger gaps should be identified and sent to the app to make a decision. If the app wants to (through a configuration parameter), we will jump those gaps also. Additionally, we should gracefully handle attempted seeks into gaps (for both buffered and unbuffered seeks). ### Configuration - `.streaming.smallGapLimit` (default 0.5) the threshold (in seconds) for a gap to be considered small and jumped automatically. - `.streaming.jumpLargeGaps` (default false) if true, automatically jump large gaps. (see event below) ### Small vs Large Gaps To support gap jumping across browsers, we will jump small gaps on all browsers without user interaction. These gaps should be small so they do not interrupt playback visually. Larger gaps can be optionally jumped, but we will tell the app about it first. There will be a new event `largegap`. When the playhead enters a large gap, the video will be paused, but we won’t enter a buffering state. We will then fire the event to the app, giving it the current time and the start and end time of the gap. If `jumpLargeGaps` is set, we will use the default action of the event to get feedback from the event listeners. The “default” action here is to jump the gap. If the event handlers call `preventDefault()`, we will not jump this gap. If `jumpLargeGaps` is not set, we will still fire the event, but never jump the gap ourselves. ### Manifest Parser The manifest parser should remove all gaps from the index. The segment index should be continuous, even for large gaps. We are not going to support “empty periods”; all Period elements should have content. However, there can be gaps between segments or between Period elements. If the gap is larger than a threshold, a warning is printed. ### Streaming Engine StreamingEngine will not require any changes. Since the segment index is continuous, it will not see any gaps and just buffer like normal. We will need to modify the buffering logic a little. We will calculate the amount buffered based on the content buffered (i.e. we will ignore the time in the gap). This ensures smooth playback with gap jumping. Like now, it will continue buffering a sequence of segments from the index. It will only buffer segments in media order, waiting for any segments that appear in the index but are not ready on the server (404). This ensures that any gap in the buffered ranges is caused by gaps in the content, not by content that will be available later. We already fetch the segment before the target time when seeking. We also only adjust the end time of a segment, so the start time is “close” to the real time. This means that when we seek, there will always be content buffered before the playhead. Once we have buffered past the playhead, we will know if we have seeked into a gap. ### Playhead The Playhead will hold the logic for gap jumping. When the video plays into a gap, the browser will do two things: (1) fire a `waiting` event and (2) change the readyState to 2 (`HAVE_CURRENT_FRAME`) or 1 (`HAVE_METADATA`) if this was a seek. To handle this, the Playhead will listen to the `waiting` event and update the current time to just past the gap. When we seek, StreamingEngine will clear the buffer. This will cause a `waiting` event to fire, which we ignore since nothing will be buffered. Once StreamingEngine starts buffering new content, we need to determine if we are still in a gap. If we are inside a gap, the ready state will be 1 or 2 (it will be 3 or 4 while playing). We will poll the ready state and if it is 1 or 2, we are inside (or near) a gap and will need to jump it. If the ready state is 3 or 4, we can play and we don’t need to do anything more now. Playhead will only change the current time if we are in a gap, or close to it. If the first buffered range is slightly ahead of the playhead, we will jump it; it will also jump if there are multiple buffered ranges and the playhead is close to a gap. Playhead will also handle firing the event for large gaps. Event firing is synchronous, so Playhead will fire the event and then jump the gap if needed.