[Red5] Bandwidth throttling within Red5.

Walter Tak walter at waltertak.com
Mon Dec 10 15:13:36 PST 2007 

I'd love to see a developer look at that specific piece of code since this 
is affecting all of us.

To be honest ; we also experienced cleints disconnecting from our apps with 
streams as simple as 300 kbit while each
client has at least 1mbit of bandwidth available and the server has a 100 
mbit NIC and is in a bomb-proof datacentre with plenty of gigabits of 
backbonebandwidth.

However I can only reproduce the 'insufficientbw' errors myself when I use a 
tool like NetLimiter to reduce my bandwidth to a value that is just
10-20% higher than the size of the incoming stream (e.g. limiting to 300 
kbit whent the stream is 250 kbit). Only in that scenario those error-events
get triggered and if I don't pause the video the server will disconnect the 
client with a msg like

12-07 18:00:29 [WARN] 232430830 DefaultQuartzScheduler_Worker-2:( 
RTMPConnection.execute ) Closing RTMPMinaConnection from 88.211.153.23:1442 
to beachhead15.tentacle.nl (in: 3571, out: 203815) due to too much 
inactivity (1197046824274).

However, since I'm running with an "old" 0.6.3. version (no trunk) (on this 
particular server) I don't complain too hard ;
I expect the above problem (Quartz) to be related to the old version.

----- Original Message ----- 
From: "Sales Department" <sales at interalab.com>
To: <red5 at osflash.org>
Sent: Monday, December 10, 2007 2:27 PM
Subject: Re: [Red5] Bandwidth throttling within Red5.


>I agree with Mr Seb.  We have no bandwidth or hardware limitations.  I
> have been wrong in the past, but I have seen nothing in our testing to
> indicate that it is a client-side issue.  In fact, just the opposite.
> BTW, the dynamic buffer code does not have any effect on this problem.
> That's because it's not a buffering problem.  As I said before, the
> player stays nicely in it's buffering range as specified in the
> Actionscript (using the dynamic buffer code as linked by Walter or
> without).  It never gets delayed more than the specified buffer and
> audio is never out of sync, even after hours of play. The client machine
> is idle except for this player and is a high-end desktop (although we've
> tested using a variety of machines, in and outside of a firewall - rtmpt
> is worse)
>
> The code change we posted appears to resolve the symptom, so I'm hoping
> that the focus of this thread can stay on reviewing the code change we
> made to see if it will cause other unintended consequences.
>
> Bill
>
> Mr Seb wrote:
>> Well, that's just it - there is plenty of bandwidth.
>>
>> There is no frames dropped, and the bandwidth on the link is more than
>> sufficient (100mbit both sides), but anything over 350kbs seems to empty 
>> the
>> buffer and then cause insufficientBW errors.
>>
>> That's server-side. It has to be.
>>
>> And something in that code block that is being avoided with that 'fix'
>> posted earlier is allowing streams over 350kbs.
>>
>> I guess there's some erroneous code somewhere, or some balancing thingee 
>> is
>> only activated for recognised codecs -- I don't know the code well enough 
>> to
>> guess, sadly :)
>>
>> -S
>>
>> Walter Tak:
>>
>>> If you don't pay attention to "insufficientbandwidth" errors your client
>>> WILL be disconnected by the server and that's a worst case scenario.
>>>
>>
>>
>>
>>> ----- Original Message -----
>>> From: "Ben Weekes" <ben.weekes at requestec.com>
>>> To: <red5 at osflash.org>
>>> Sent: Monday, December 10, 2007 1:43 AM
>>> Subject: Re: [Red5] Bandwidth throttling within Red5.
>>>
>>>
>>>
>>>> try testing in poorer circumstants.
>>>> the thing is, if the reciever detects delay, it must tell the other end
>>>>
>>> to
>>>
>>>> SLOW down.
>>>> failure to do so will result in undefined behaviour at best.
>>>>
>>>> Ben
>>>> ----- Original Message -----
>>>> From: "Interalab" <sales at interalab.com>
>>>> To: <red5 at osflash.org>
>>>> Sent: Sunday, December 09, 2007 11:47 PM
>>>> Subject: Re: [Red5] Bandwidth throttling within Red5.
>>>>
>>>>
>>>>
>>>>> My concern with our 'fix' is this:  The reason the InsufficientBW
>>>>> notification is thrown is that there are more than a certain number of
>>>>> frames/packets/whatever waiting to be delivered.  I think we're being
>>>>> warned that it's getting behind in delivering the video.  When that
>>>>> happens in the unmodified version of Red5, the extra frames get 
>>>>> dropped
>>>>> and the notification is thrown.  In our modified version, the frames 
>>>>> are
>>>>> not  dropped and the notification isn't thrown.  What happens to the
>>>>> extra frames?  Do they build up over time and cause other problems? 
>>>>> The
>>>>> clients never see any added delay (client buffering still works as
>>>>> expected, limiting delay from growing), no audio sync problems occur
>>>>> either.
>>>>>
>>>>> Bill
>>>>>
>>>>> Mr Seb wrote:
>>>>>
>>>>>> Wow, your problems are almost identical to mine!
>>>>>>
>>>>>> Just re: the other points raised by other people - it's not hardware
>>>>>> related, I'm almost 100% certain of that. CPU load is very low, no
>>>>>> frames
>>>>>> are dropped, and I am only using 40-50% of my internet connection.
>>>>>>
>>>>>> Also, it pops up with 'InsufficientBW' on multiple people watching 
>>>>>> the
>>>>>> same
>>>>>> stream, at the same time! So again, this means it's either my home
>>>>>> connection, or the server -- ruling out the download bandwidth at
>>>>>>
>>> least.
>>>
>>>>>> Anyway, I will try the fix below... Will report back later!
>>>>>>
>>>>>> -S
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On
>>>>>>> Behalf
>>>>>>> Of Interalab
>>>>>>> Sent: 09 December 2007 21:57
>>>>>>> To: red5 at osflash.org
>>>>>>> Subject: Re: [Red5] Bandwidth throttling within Red5.
>>>>>>>
>>>>>>> The problem is not with the screen video as far as we can tell. This
>>>>>>>
>>> is
>>>
>>>>>>> a problem that has been there for a long time and seems to be 
>>>>>>> brought
>>>>>>> up
>>>>>>> in various contexts and circumstances. Here's what we see as the
>>>>>>> consistent factors:
>>>>>>>
>>>>>>> - using VP6 (whether it's FME or a custom VP6 encoder as in our 
>>>>>>> case)
>>>>>>> - live video encoding/streaming, not VOD.
>>>>>>> - higher quality settings on the encoder (we haven't pegged the
>>>>>>> bitrate,
>>>>>>> but our problems are occurring with a 400kbps stream at 15 fps)
>>>>>>> - no load on the server memory or CPU or bandwidth (we're on a 
>>>>>>> gigabit
>>>>>>> internal and external)
>>>>>>>
>>>>>>> To solve it, we have modified the Red5 server code (as posted a few
>>>>>>> days
>>>>>>> ago for comment). See below.
>>>>>>>
>>>>>>> As far as we can see, Red5 doesn't understand what the VP6 codec is,
>>>>>>>
>>> so
>>>
>>>>>>> videoCodec is seen as null. Using the following modification we make
>>>>>>>
>>> it
>>>
>>>>>>> do nothing if videoCodec is null and it seems to have fixed our
>>>>>>> problem,
>>>>>>> but I think it might have just masked some other issue.
>>>>>>>
>>>>>>> Comments welcome . . .
>>>>>>>
>>>>>>> --- .svn/text-base/PlaylistSubscriberStream.java.svn-base 2007-12-05
>>>>>>> 05:27:04.000000000 -0500
>>>>>>> +++ PlaylistSubscriberStream.java 2007-12-06 
>>>>>>> 11:37:58.000000000 -0500
>>>>>>> @@ -1889,6 +1889,8 @@
>>>>>>> }
>>>>>>>
>>>>>>> if (videoCodec == null ||
>>>>>>> videoCodec.canDropFrames()) {
>>>>>>> + if(videoCodec == null) {
>>>>>>> + } else {
>>>>>>> if (state == State.PAUSED) {
>>>>>>> // The subscriber
>>>>>>>
>>>>>>>
>>>>>> paused the
>>>>>>
>>>>>>
>>>>>>> video
>>>>>>>
>>>>>>> videoFrameDropper.dropPacket(rtmpMessage);
>>>>>>> @@ -1925,6 +1927,7 @@
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> videoFrameDropper.sendPacket(rtmpMessage);
>>>>>>> + }
>>>>>>> }
>>>>>>> } else if (body instanceof AudioData) {
>>>>>>> if (!receiveAudio && sendBlankAudio)
>>>>>>>
>>>>>>>
>>>>>> {
>>>>>>
>>>>>>
>>>>>>> Andy wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Try this screen video broadcaster....Of course point it to your 
>>>>>>>> red5
>>>>>>>> server ;)
>>>>>>>>
>>>>>>>> I have no trouble at high bandwidth with screen video. FME hasn't
>>>>>>>> given me satisfying results... yet.
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> *From:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org]
>>>>>>>>
>>> *On
>>>
>>>>>>>> Behalf Of *Mr Seb
>>>>>>>> *Sent:* Sunday, December 09, 2007 12:15 PM
>>>>>>>> *To:* red5 at osflash.org
>>>>>>>> *Subject:* Re: [Red5] Bandwidth throttling within Red5.
>>>>>>>>
>>>>>>>> I'm using FME.
>>>>>>>>
>>>>>>>> The output from FME is a steady 15 FPS (or whatever I set it to),
>>>>>>>> using whatever bit rate I set it to.
>>>>>>>>
>>>>>>>> Anything over the 350kbs preset and I suddenly start hitting
>>>>>>>> InsufficientBW errors.
>>>>>>>>
>>>>>>>> I'm running the Red5 server remotely - on a linux box. But it's
>>>>>>>> collocated on 100mbits, and the CPU is not under any load.
>>>>>>>>
>>>>>>>> As far as I can tell, there's no hardware limitations in the
>>>>>>>>
>>> equation.
>>>
>>>>>>>> I haven't tried running it locally - I guess I can try that. But I
>>>>>>>> don't know what that proves, if there is plenty of bandwidth
>>>>>>>>
>>> available
>>>
>>>>>>>> on the remote box!
>>>>>>>>
>>>>>>>> I was wondering if perhaps the default buffer sizes can't cope with 
>>>>>>>> a
>>>>>>>> high-resolution (720*576) stream at 15 FPS. but. I really don't 
>>>>>>>> know
>>>>>>>> :)
>>>>>>>>
>>>>>>>> The content is the live, stream of the display of my computer. No
>>>>>>>> audio (yet).
>>>>>>>>
>>>>>>>> -S
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>> ---
>>>
>>>>>>>> *From:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org]
>>>>>>>>
>>> *On
>>>
>>>>>>>> Behalf Of *Andy
>>>>>>>> *Sent:* 09 December 2007 20:10
>>>>>>>> *To:* red5 at osflash.org
>>>>>>>> *Subject:* Re: [Red5] Bandwidth throttling within Red5.
>>>>>>>>
>>>>>>>> There are none.
>>>>>>>>
>>>>>>>> Do you get the same response running it locally?
>>>>>>>>
>>>>>>>> Or are you already?
>>>>>>>>
>>>>>>>> Are you using FME? I find it spurts where the screen video capture 
>>>>>>>> is
>>>>>>>> a steady dribble.
>>>>>>>>
>>>>>>>> I have better luck live, with screen video.
>>>>>>>>
>>>>>>>> andy
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> *From:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org]
>>>>>>>>
>>> *On
>>>
>>>>>>>> Behalf Of *Mr Seb
>>>>>>>> *Sent:* Sunday, December 09, 2007 11:56 AM
>>>>>>>> *To:* red5 at osflash.org
>>>>>>>> *Subject:* [Red5] Bandwidth throttling within Red5.
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Could someone walk me through how to ensure there is absolutely no
>>>>>>>> bandwidth/stream bit rate limits in place?
>>>>>>>>
>>>>>>>> I'm pretty sure the default red5-trunk install has none in place, 
>>>>>>>> but
>>>>>>>> I'm really out of guesses on why the oflaDemo won't let me stream
>>>>>>>> anything over 350kbit/sec without a lot of NetStream.InsufficientBW
>>>>>>>> errors popping up.
>>>>>>>>
>>>>>>>> Of course, if someone has managed to make an 'unlimited' oflaDemo,
>>>>>>>>
>>> I'm
>>>
>>>>>>>> interested in that too!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> -S
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>> ---
>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>> --
>>> I am using the free version of SPAMfighter for private users.
>>> It has removed 13781 spam emails to date.
>>> Paying users do not have this message in their emails.
>>> Get the free SPAMfighter here: http://www.spamfighter.com/len
>>>
>>>
>>>
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
> 


-- 
I am using the free version of SPAMfighter for private users.
It has removed 13782 spam emails to date.
Paying users do not have this message in their emails.
Get the free SPAMfighter here: http://www.spamfighter.com/len

More information about the Red5 mailing list