[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)
Dan Rossi
spam at electroteque.org
Wed Jan 30 06:06:50 PST 2008
"Weird. I have tested the server with 1s delay and 100 connections.
The CPU
load is very low (below 15%). The high load only occurs at 0s delay
with 100
connections."
Can you give me the platform you are working on how you bootup red5,
what webapp you are using and what file you are trying to stream with
the test and i can see if i can match it. Im not understanding the top
stats for the dual core as the user % is much smaller than the process
cpu stats maybe you can shed some light there usually it adds up.
On 29/01/2008, at 12:46 PM, Dan Rossi wrote:
> I have it back on the old copy and 48 requests gets about 22%. The
> problem is once it gets to 80 it also starts to max out. The new
> copies i keep trying to test max out much earlier than that. Its an
> issue because I cannot make a system using a custom webapp live
> until its somehow corrected because it uses alot of the logback
> changes for a custom access logging system for bandwidth billing of
> streams. I cannot seem to get the logback stuff to work in this
> version.
>
> On 29/01/2008, at 12:16 PM, Steven Gong wrote:
>
>>
>> On Jan 29, 2008 9:10 AM, Dan Rossi <spam at electroteque.org> wrote:
>> I have the latest svn on a server with flags set to false it seemed
>> to reach a higher cpu at 18 requests than with it set to true, im
>> seeing peaks at 90% with 34 requests, at 35 requests i got 100%.
>> This doesnt add up sorry. Maybe it has something to do with network
>> storage ?
>> Yes, maybe if you are getting the VOD via network. How do you read
>> the files?
>>
>>
>> On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
>>
>>> With flag set to false and 100 requests
>>>
>>>
>>> top - 09:29:58 up 53 min, 2 users, load average: 9.25, 5.00, 4.40
>>> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
>>> Cpu0 : 40.3%us, 30.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi,
>>> 1.0%si, 0.0%st
>>> Cpu1 : 17.3%us, 22.3%sy, 0.0%ni, 44.7%id, 0.0%wa, 8.7%hi,
>>> 7.0%si, 0.0%st
>>> Mem: 1029420k total, 999888k used, 29532k free, 372k
>>> buffers
>>> Swap: 1959888k total, 0k used, 1959888k free, 558692k
>>> cached
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 7521 root 17 0 505m 136m 10m S 126 13.6 1:31.58 java
>>>
>>> With flag set to true and 100 requests
>>>
>>> 90 users 100% on one core of the cpu 20% overall.
>>>
>>> top - 09:25:42 up 49 min, 2 users, load average: 5.74, 3.23, 3.91
>>> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
>>> Cpu0 : 17.5%us, 31.1%sy, 0.0%ni, 50.7%id, 0.0%wa, 0.0%hi,
>>> 0.7%si, 0.0%st
>>> Cpu1 : 15.8%us, 22.8%sy, 0.0%ni, 47.2%id, 0.0%wa, 7.6%hi,
>>> 6.6%si, 0.0%st
>>> Mem: 1029420k total, 1010608k used, 18812k free, 372k
>>> buffers
>>> Swap: 1959888k total, 0k used, 1959888k free, 558836k
>>> cached
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 7189 root 17 0 502m 146m 10m S 101 14.6 4:02.54 java
>>>
>>>
>>> I don't see much of a difference :\
>>>
>>>
>>> On 29/01/2008, at 3:02 AM, Mondain wrote:
>>>
>>>> There no "fix" yet, but I think you may be able to set
>>>> generateMetadata to false in the interim. In red5-common.xml look
>>>> for this bean:
>>>> <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
>>>> <property name="generateMetadata" value="true"/>
>>>> </bean>
>>>> Simply setting it to false may prevent your CPU spikes.
>>>>
>>>> Paul
>>>>
>>>> On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
>>>> Is the possible fixes committed at all ? I just tried on my
>>>> little ubuntu machine which is a 2.2G AMD X2 and reached 100% at
>>>> 100 users and was averaging at 85% the whole way through with the
>>>> loader tester app.
>>>>
>>>> On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
>>>>
>>>>> I knew thats where it may have happened when the cache changed.
>>>>> The problem with this is has to identify files of same names
>>>>> inside different paths. I believe there was a clash there !
>>>>> Please let me know how you go, some frustrated bosses on my end
>>>>> wanting to bite the bullet and go with FMS and 1 and a bit years
>>>>> nearly wasted.
>>>>>
>>>>> Please let me know and I can test.
>>>>>
>>>>> On 26/01/2008, at 8:46 AM, Chris Allen wrote:
>>>>>
>>>>>> Guys,
>>>>>>
>>>>>> Let's get cranking on reverse engineering the H.264 stuff.
>>>>>>
>>>>>> http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
>>>>>>
>>>>>> :-)
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>>
>>>>>> On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com>
>>>>>>> wrote:
>>>>>>> Nice Steven! So, are you going to implement this fix?
>>>>>>> Yes, I will provide the fix before the release.
>>>>>>>
>>>>>>> -Chris
>>>>>>>
>>>>>>> On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> I found the root cause of big CPU load on VOD streaming. It
>>>>>>>> is caused by the loading of key frame metadata from metadata
>>>>>>>> cache. A simple solution to this is to enhance the metadata
>>>>>>>> cache implementation to cache the loaded metadata into memory
>>>>>>>> for FLV rather than load it again from the file system each
>>>>>>>> time.
>>>>>>>>
>>>>>>>> On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
>>>>>>>> The load tester (src) is checked into the trunk btw..
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net >
>>>>>>>> wrote:
>>>>>>>> Hi Steven
>>>>>>>>
>>>>>>>> Yes please do.
>>>>>>>>
>>>>>>>> - Luke
>>>>>>>>
>>>>>>>>
>>>>>>>> On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote: Luke,
>>>>>>>> If you need Paul's load tester for the test, I can send a
>>>>>>>> copy to you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>
>>>>>>>> wrote:
>>>>>>>> Paul's load tester flex app is a good one to use but i dont
>>>>>>>> think its accurate enough. I have tested the same app against
>>>>>>>> FMS and Wowza and there was only 5% cpu used maximum.
>>>>>>>>
>>>>>>>> On 21/01/2008, at 7:43 PM, Steven Gong wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>
>>>>>>>>> wrote:
>>>>>>>>> I will try to test with a profiler see what is eating all
>>>>>>>>> that CPU.
>>>>>>>>> This is for VOD right, not live? Or both?
>>>>>>>>> This is the VOD streaming and can be reproduced by a simple
>>>>>>>>> load test VOD playback.
>>>>>>>>>
>>>>>>>>> - Luke
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
>>>>>>>>> Less than that before its reaching 100% cpu. aparantly i go
>>>>>>>>> away on a break and everything goes down. It might be
>>>>>>>>> looking like we might have to jump onto the wowza bandwagon :\
>>>>>>>>>
>>>>>>>>> On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
>>>>>>>>>
>>>>>>>>>> Dan:
>>>>>>>>>>
>>>>>>>>>> are you saying that the core can only handle 50-80
>>>>>>>>>> concurrent connections?
>>>>>>>>>>
>>>>>>>>>> On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
>>>>>>>>>>
>>>>>>>>>>> Great, i added the preliminary task in there but didnt get
>>>>>>>>>>> much responses about it at first :D Should we add back in
>>>>>>>>>>> the option to either compile javadoc in a full
>>>>>>>>>>> distribution task or not, that was added in as a
>>>>>>>>>>> suggestion from steve but was removed. It does reduce the
>>>>>>>>>>> recompile time alot.
>>>>>>>>>>>
>>>>>>>>>>> I believe the major one that i know of is the performance
>>>>>>>>>>> problems somewhere in the core, its maxing out at 50-80
>>>>>>>>>>> concurrents.
>>>>>>>>>>>
>>>>>>>>>>> On 19/01/2008, at 3:15 AM, Mondain wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If we plan to inform users of the edge/origin features,
>>>>>>>>>>>> we will also need to get those parts of the build script
>>>>>>>>>>>> ironed-out before the release as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Paul
>>>>>>>>>>>>
>>>>>>>>>>>> On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hey Joachim,
>>>>>>>>>>>>
>>>>>>>>>>>> Welcome back!!
>>>>>>>>>>>>
>>>>>>>>>>>> Glad to see your return.
>>>>>>>>>>>>
>>>>>>>>>>>> Hope things are well with these days.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Lenny
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 1/17/08, Dominick Accattato < daccattato at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Team:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I want to give Joachim a warm welcome back. We missed
>>>>>>>>>>>> ya! Everyone has done a great job pushing Red5 forward
>>>>>>>>>>>> from the 0.6 release. In preparation of the release of
>>>>>>>>>>>> 0.7, I'd like to get a few more fixes in. If we can,
>>>>>>>>>>>> lets try to get this stuff done asap so that we can get
>>>>>>>>>>>> 0.7 out the door. Here is what I find to be of
>>>>>>>>>>>> importance for this release.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> • http://jira.red5.org/browse/APPSERVER-184
>>>>>>>>>>>> • amf3 synchronization issue with SharedObjects (I will
>>>>>>>>>>>> provide Joachim with a test app)
>>>>>>>>>>>> • fix our tests since Paul noticed many of them are
>>>>>>>>>>>> failing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If anyone can think of any other important fixes, list
>>>>>>>>>>>> them so that we can look into their priority. Thanks team!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> It is difficult to free fools from the chains they
>>>>>>>>>>>> revere. - Voltaire
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Luke Hubbard
>>>>>>>>> codegent | coding for the people
>>>>>>>>> http://www.codegent.com
>>>>>>>>>
>>>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>>>> http://www.codegent.com/top100/
>>>>>>>>>
>>>>>>>>> want to know more?
>>>>>>>>> http://www.codegent.com/showreel/
>>>>>>>>>
>>>>>>>>> This e-mail may contain information which is privileged,
>>>>>>>>> confidential and protected from disclosure. If you are not
>>>>>>>>> the intended recipient of this e-mail, or any part of it,
>>>>>>>>> please delete this email and any attachments immediately on
>>>>>>>>> receipt. You should not disclose the contents to any other
>>>>>>>>> person or take copies. Any views expressed in this message
>>>>>>>>> are those of the individual sender, except where the sender
>>>>>>>>> specifically states them to be the views of codegent limited.
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards
>>>>>>>>> Steven Gong
>>>>>>>>>
>>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>>
>>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>>> ~steven/Home
>>>>>>>>>
>>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Luke Hubbard
>>>>>>>> codegent | coding for the people
>>>>>>>> http://www.codegent.com
>>>>>>>>
>>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>>> http://www.codegent.com/top100/
>>>>>>>>
>>>>>>>> want to know more?
>>>>>>>> http://www.codegent.com/showreel/
>>>>>>>>
>>>>>>>> This e-mail may contain information which is privileged,
>>>>>>>> confidential and protected from disclosure. If you are not
>>>>>>>> the intended recipient of this e-mail, or any part of it,
>>>>>>>> please delete this email and any attachments immediately on
>>>>>>>> receipt. You should not disclose the contents to any other
>>>>>>>> person or take copies. Any views expressed in this message
>>>>>>>> are those of the individual sender, except where the sender
>>>>>>>> specifically states them to be the views of codegent limited.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> It is difficult to free fools from the chains they revere. -
>>>>>>>> Voltaire
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>> accomplishment. -- Ricky Jay
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>> _______________________________________________
>>>>>
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> It is difficult to free fools from the chains they revere. -
>>>> Voltaire _______________________________________________
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>> _______________________________________________
>>>
>>> Red5devs mailing list
>>> Red5devs at osflash.org
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>> _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>>
>>
>> --
>> Best Regards
>> Steven Gong
>>
>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>
>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>> ~steven/Home
>>
>> Modesty is an overrated quality in men of no great accomplishment.
>> -- Ricky Jay _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
More information about the Red5devs
mailing list