[Red5devs] [PATCH] Make red5 output correct timestamps for live streams

Art Clarke aclarke at xuggle.com
Mon Oct 19 21:55:39 PDT 2009 

Hi Dave,

Actually there's a separate branch (see xuggle_timestamp_fixes) in the Red5
tree that has the final version of this patch; you should test that
instead.  We've tested it extensively here at Xuggle and the good news is it
fixes the problem pretty conclusively (we've been able to run 1-2 hour
conferences using it, but prior red5 trees the conferences would be out of
sync within 15 minutes).  The bad news is that branch has the Red5
frame-dropping support (for low b/w connections) removed and we're working
on a new version.

- Art

On Mon, Oct 19, 2009 at 9:51 PM, David Hogan <mr.david.hogan at gmail.com>wrote:

> Did anyone do any further testing of this patch? I'll be doing my own tests
> of r3735 clean vs patched (with a latency of around 250ms, to make things
> interesting). I'll report back my results ..
>
> 2009/8/15 Art Clarke <aclarke at xuggle.com>
>
>> Another version of the patch (v3) for application to r3735.  This is
>> mostly the same as the last patch (v2) with the following additions:
>>
>> 1) We now correctly detect late arriving video packets and correctly
>> re-order them so as to now lose late key frames (this case appears to
>> happen when Flash decides to send audio during the middle of a large
>> keyframe).
>> 2) We now detect cases where Flash (for whatever reason) sends a
>> repeated audio packet when it shouldn't.  In that case we drop the
>> duplicate audio.
>>
>> This has now been running for over an hour in our time stamp
>> validation system and we're getting the right packets out, with
>> constant sync between audio and video, even from multiple different
>> streams.
>>
>> It doesn't fix Dave's issue with syncing, but I'm not sure his issue
>> is related; he is using a separate Java RTMP library acting as a proxy
>> for rebroadcasting, and that library is doing its own time stamp mojo
>> (which doesn't appear to account for issue #2 above).  That works with
>> the old way of timestamping in Red5, but only because the old version
>> always moved the next packet to be after the last packet, regardless
>> of stream, and then Flash works around that by syncing only on the
>> audio.
>>
>> - Art
>>
>>
>> On Thu, Aug 13, 2009 at 11:16 AM, Art Clarke<aclarke at xuggle.com> wrote:
>> > And here's attempted patch #2.  Again, I'm not checking this in
>> > because it's a pretty big change.  But I wanted to get a preliminary
>> > version of the patch out to those (Tiago, Dave & Paul) who might be
>> > able to review and suggest changes (and see if their problems with
>> > timestamps magically go away).
>> >
>> > I took the alternate approach for this patch -- instead of trying to
>> > patch it with the minimal changes to Red5, instead I fix the root
>> > problem with timestamps by fixing how RTMPProtocolDecoder and
>> > RTMPProtocolEncoder deal with them.  Here's a summary of the changes:
>> >
>> > 1) Timestamps are now always ABSOLUTE in Red5 with this patch.  The
>> > protocol codecs ensure this.
>> > 2) Any code that was dealing with "isTimerRelative" now goes down the
>> > "false" path.
>> > 3) Red5 will change RTMP relative timestamps into absolute timestamps
>> > before the RTMPEvent leaves the decoder.  Red5 will change absolute
>> > timestamps into relative timestamps when encoding RTMP packets IFF the
>> > delta between frames < 250 milliseconds and all other RTMP relative
>> > timestamps constraints are met.  The 250 milliseconds is a somewhat
>> > arbitrary value but will only be inefficient if you're broadcasting
>> > video at 4 FPS or less, and who the hell does that.  (By the way,
>> > somewhat arbitrary means based on testing at a prior job we found that
>> > delays of > 250 milliseconds in video were perceivable by over 90% of
>> > people.)  This fixes some problems with audio and video getting out of
>> > sync when microphone silence detection was turned on and off.
>> > 4) Streams over 4.5 hours should now work (running test overnight to
>> confirm)
>> > 5) Abort messages should be handled from the client if/when they show
>> > up (very rare, but they do happen).
>> > 6) getExtendedTimestamp() now goes away since it's really not used
>> > given the changes above.
>> >
>> > Anyway, still testing so more changes may come, but if folks want to
>> > give it a go-around I'd appreciate it.  Dave, if you're doing the same
>> > test as before, make sure (a) your mic is enabled from the start of
>> > the test and (b) your video player (whichever you're using) does not
>> > ignore gaps in audio at the start of FLV files (for example, mplayer
>> > will bring all audio forward if there are gaps).  If you still see a
>> > problem, if you could post the recording you made to a website and
>> > send me the link I'd love to look at it.
>> >
>> > - Art
>> >
>> > On Mon, Aug 10, 2009 at 11:48 AM, Art Clarke<aclarke at xuggle.com> wrote:
>> >> Hi folks,
>> >>
>> >> First the bad news: I know this will be a surprise to some people, but
>> Red5
>> >> outputs completely incorrect timestamps when re-publishing live
>> streams!
>> >>
>> >> Now the good news:  Attached is a patch that makes Red5 output correct
>> >> timestamps for live streams.
>> >>
>> >> The impact of this change: When your microphone gets muted (or silence
>> >> detection kicks in), Red5 should now keep audio and video synced, even
>> once
>> >> the microphone comes back on, and for long running broadcasts.  Also it
>> is
>> >> now technically possible to sync audio and video across broacasters,
>> but
>> >> that is left as an exercise for the reader (and it's not simple -- just
>> >> possible now).
>> >>
>> >> The problem:  For live streams, Red5 sets the timestamps on RTMP
>> packets on
>> >> subscriber streams using the relative timestamps of the last packet
>> sent to
>> >> the client -- not based on the last timestamp of the last packet of the
>> same
>> >> type.  That means that over time the timestamps end up with ever
>> increasing
>> >> gaps in them.  So imagine the following timestamps received from a
>> >> broadcasting stream:
>> >>
>> >> Type: Timestamp
>> >> Audio: 0 (abs)
>> >> Video: 0 (abs)
>> >> Audio: 47 (rel)
>> >> Video: 75 (rel)
>> >> Audio: 46 (rel)
>> >> Video: 150 (abs)
>> >> Audio: 47 (rel)
>> >>
>> >> In theory the timestamps should stay the same when someone "subscribes"
>> to
>> >> the stream, but instead Red5 adjusts them incorrect as follows:
>> >> Audio: 0 (abs)
>> >> Video: 0 (abs)
>> >> Audio: 47 (rel)
>> >> Video: 122(rel)
>> >> Audio: 168 (rel)
>> >> Video: 168 (abs) // due to a hack in ClientBroadcastStream
>> >> Audio: 215 (rel)
>> >>
>> >> How could this bug be in Red5 for so long?  Because the Flash Player
>> works
>> >> around it:
>> >> 1) If the stream is Audio-Only, Flash really ignores the timestamps and
>> then
>> >> just plays the samples as it decodes them.  It figures (not too
>> >> unreasonably) that for live streams if audio has shown up, and other
>> audio
>> >> is currently playing, that it should just append this audio after the
>> >> playing audio.
>> >> 2) If the stream is Audio and Video, Flash will use the actual samples
>> to
>> >> determine when to play audio, and will use the timestamp value in the
>> >> samples as a relative clock for video syncing.  This means that even
>> though
>> >> Red5 has constantly increasing timestamp deltas, Flash still still get
>> it
>> >> right.
>> >> 3) If the stream is Video Only, Flash will display the frame as soon as
>> it
>> >> arrives if it has an empty frame buffer, and otherwise will get the
>> timing
>> >> slightly wrong.  However, the Flash RTMP broadcasters tend to send
>> video
>> >> packets with absolute timestamps about every 3 seconds, which then
>> corrects
>> >> the error for video-only streams so it's hard to notice that those
>> >> inter-frames were off by 50-100 ms.
>> >> 4) And the nature of the original bug is that the error introduced by
>> Red5,
>> >> while increasing, remains constant across all streams, so Flash's
>> >> workarounds can correct for it in most cases.
>> >>
>> >> It fact, the bug really only shows up if you have an audio-video
>> streams
>> >> that drops audio (silence detection, or network issues), in which case
>> once
>> >> audio resumes you'll sometimes see audio and video out of sync, and
>> >> sometimes see the video "speed up" on the client to account for wrong
>> time
>> >> stamps.  The bug will also show up if you're trying to mix live audio
>> >> between two streams (which is how I found it), but that's advanced
>> stuff :)
>> >>
>> >> The fix:
>> >>
>> >> For live streams, we still set relative timestamps, but each timestamp
>> is
>> >> now relative to the previous timestamp OF THE SAME data type.  That
>> means
>> >> our subscribing streams get the same timestamps as the broadcast
>> streams!
>> >>
>> >> Like I said, I think this works, but I'm a little nervous because it's
>> such
>> >> a fundamental change that I'd like to have the core red5 devs check it
>> out
>> >> before it goes into trunk.  It's a deceptively simple change, but I
>> spent
>> >> four days on it trying to figure out why nellymoser audio samples at
>> 22khz
>> >> had timestamp values increasing by arbitrary amounts >= 46
>> milliseconds,
>> >> when they should actually increase by either 46 or 47 milliseconds
>> assuming
>> >> contiguous packets.  With this fix, audio samples output to subscribers
>> have
>> >> the correct duration (which starts to make it possible to sync-audio
>> across
>> >> streams).
>> >>
>> >> (By the way, I also determined that Red5 incorrectly computes
>> timestamps on
>> >> streams longer than 4.6 hours, and I'll patch that next).
>> >>
>> >> Finally, this doesn't fix all cases of video speeding up (some of which
>> are
>> >> done on the Flash client in response to network delay, and there's
>> nothing
>> >> Red5 can do about it), but it should help.
>> >>
>> >> - Art
>> >>
>> >> --
>> >> http://www.xuggle.com/
>> >> xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and
>> >> video.
>> >>
>> >> Use Xuggle to get the power of FFmpeg in Java.
>> >>
>> >
>> >
>> >
>> > --
>> > http://www.xuggle.com/
>> > xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and
>> video.
>> >
>> > Use Xuggle to get the power of FFmpeg in Java.
>> >
>>
>>
>>
>> --
>> http://www.xuggle.com/
>> xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and
>> video.
>>
>> Use Xuggle to get the power of FFmpeg in Java.
>>
>> _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>


-- 
http://www.xuggle.com/
xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and
video.

Use Xuggle to get the power of FFmpeg in Java.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://osflash.org/pipermail/red5devs_osflash.org/attachments/20091019/59e7021a/attachment-0001.html>

More information about the Red5devs mailing list