[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)
Dan Rossi
spam at electroteque.org
Mon Jan 28 17:10:03 PST 2008
I have the latest svn on a server with flags set to false it seemed to
reach a higher cpu at 18 requests than with it set to true, im seeing
peaks at 90% with 34 requests, at 35 requests i got 100%. This doesnt
add up sorry. Maybe it has something to do with network storage ?
On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
> With flag set to false and 100 requests
>
>
> top - 09:29:58 up 53 min, 2 users, load average: 9.25, 5.00, 4.40
> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> Cpu0 : 40.3%us, 30.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi,
> 1.0%si, 0.0%st
> Cpu1 : 17.3%us, 22.3%sy, 0.0%ni, 44.7%id, 0.0%wa, 8.7%hi,
> 7.0%si, 0.0%st
> Mem: 1029420k total, 999888k used, 29532k free, 372k
> buffers
> Swap: 1959888k total, 0k used, 1959888k free, 558692k
> cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7521 root 17 0 505m 136m 10m S 126 13.6 1:31.58 java
>
> With flag set to true and 100 requests
>
> 90 users 100% on one core of the cpu 20% overall.
>
> top - 09:25:42 up 49 min, 2 users, load average: 5.74, 3.23, 3.91
> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> Cpu0 : 17.5%us, 31.1%sy, 0.0%ni, 50.7%id, 0.0%wa, 0.0%hi,
> 0.7%si, 0.0%st
> Cpu1 : 15.8%us, 22.8%sy, 0.0%ni, 47.2%id, 0.0%wa, 7.6%hi,
> 6.6%si, 0.0%st
> Mem: 1029420k total, 1010608k used, 18812k free, 372k
> buffers
> Swap: 1959888k total, 0k used, 1959888k free, 558836k
> cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7189 root 17 0 502m 146m 10m S 101 14.6 4:02.54 java
>
>
> I don't see much of a difference :\
>
>
> On 29/01/2008, at 3:02 AM, Mondain wrote:
>
>> There no "fix" yet, but I think you may be able to set
>> generateMetadata to false in the interim. In red5-common.xml look
>> for this bean:
>> <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
>> <property name="generateMetadata" value="true"/>
>> </bean>
>> Simply setting it to false may prevent your CPU spikes.
>>
>> Paul
>>
>> On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
>> Is the possible fixes committed at all ? I just tried on my little
>> ubuntu machine which is a 2.2G AMD X2 and reached 100% at 100 users
>> and was averaging at 85% the whole way through with the loader
>> tester app.
>>
>> On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
>>
>>> I knew thats where it may have happened when the cache changed.
>>> The problem with this is has to identify files of same names
>>> inside different paths. I believe there was a clash there ! Please
>>> let me know how you go, some frustrated bosses on my end wanting
>>> to bite the bullet and go with FMS and 1 and a bit years nearly
>>> wasted.
>>>
>>> Please let me know and I can test.
>>>
>>> On 26/01/2008, at 8:46 AM, Chris Allen wrote:
>>>
>>>> Guys,
>>>>
>>>> Let's get cranking on reverse engineering the H.264 stuff.
>>>>
>>>> http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
>>>>
>>>> :-)
>>>>
>>>> -Chris
>>>>
>>>>
>>>> On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
>>>>
>>>>>
>>>>>
>>>>> On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com>
>>>>> wrote:
>>>>> Nice Steven! So, are you going to implement this fix?
>>>>> Yes, I will provide the fix before the release.
>>>>>
>>>>> -Chris
>>>>>
>>>>> On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I found the root cause of big CPU load on VOD streaming. It is
>>>>>> caused by the loading of key frame metadata from metadata
>>>>>> cache. A simple solution to this is to enhance the metadata
>>>>>> cache implementation to cache the loaded metadata into memory
>>>>>> for FLV rather than load it again from the file system each time.
>>>>>>
>>>>>> On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
>>>>>> The load tester (src) is checked into the trunk btw..
>>>>>>
>>>>>>
>>>>>> On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net >
>>>>>> wrote:
>>>>>> Hi Steven
>>>>>>
>>>>>> Yes please do.
>>>>>>
>>>>>> - Luke
>>>>>>
>>>>>>
>>>>>> On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote:
>>>>>> Luke,
>>>>>> If you need Paul's load tester for the test, I can send a copy
>>>>>> to you.
>>>>>>
>>>>>>
>>>>>> On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>
>>>>>> wrote:
>>>>>> Paul's load tester flex app is a good one to use but i dont
>>>>>> think its accurate enough. I have tested the same app against
>>>>>> FMS and Wowza and there was only 5% cpu used maximum.
>>>>>>
>>>>>> On 21/01/2008, at 7:43 PM, Steven Gong wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>
>>>>>>> wrote:
>>>>>>> I will try to test with a profiler see what is eating all that
>>>>>>> CPU.
>>>>>>> This is for VOD right, not live? Or both?
>>>>>>> This is the VOD streaming and can be reproduced by a simple
>>>>>>> load test VOD playback.
>>>>>>>
>>>>>>> - Luke
>>>>>>>
>>>>>>>
>>>>>>> On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
>>>>>>> Less than that before its reaching 100% cpu. aparantly i go
>>>>>>> away on a break and everything goes down. It might be looking
>>>>>>> like we might have to jump onto the wowza bandwagon :\
>>>>>>>
>>>>>>> On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
>>>>>>>
>>>>>>>> Dan:
>>>>>>>>
>>>>>>>> are you saying that the core can only handle 50-80 concurrent
>>>>>>>> connections?
>>>>>>>>
>>>>>>>> On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
>>>>>>>>
>>>>>>>>> Great, i added the preliminary task in there but didnt get
>>>>>>>>> much responses about it at first :D Should we add back in
>>>>>>>>> the option to either compile javadoc in a full distribution
>>>>>>>>> task or not, that was added in as a suggestion from steve
>>>>>>>>> but was removed. It does reduce the recompile time alot.
>>>>>>>>>
>>>>>>>>> I believe the major one that i know of is the performance
>>>>>>>>> problems somewhere in the core, its maxing out at 50-80
>>>>>>>>> concurrents.
>>>>>>>>>
>>>>>>>>> On 19/01/2008, at 3:15 AM, Mondain wrote:
>>>>>>>>>
>>>>>>>>>> If we plan to inform users of the edge/origin features, we
>>>>>>>>>> will also need to get those parts of the build script
>>>>>>>>>> ironed-out before the release as well.
>>>>>>>>>>
>>>>>>>>>> Paul
>>>>>>>>>>
>>>>>>>>>> On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> Hey Joachim,
>>>>>>>>>>
>>>>>>>>>> Welcome back!!
>>>>>>>>>>
>>>>>>>>>> Glad to see your return.
>>>>>>>>>>
>>>>>>>>>> Hope things are well with these days.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Lenny
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 1/17/08, Dominick Accattato < daccattato at gmail.com> wrote:
>>>>>>>>>> Team:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I want to give Joachim a warm welcome back. We missed ya!
>>>>>>>>>> Everyone has done a great job pushing Red5 forward from the
>>>>>>>>>> 0.6 release. In preparation of the release of 0.7, I'd
>>>>>>>>>> like to get a few more fixes in. If we can, lets try to
>>>>>>>>>> get this stuff done asap so that we can get 0.7 out the
>>>>>>>>>> door. Here is what I find to be of importance for this
>>>>>>>>>> release.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://jira.red5.org/browse/APPSERVER-184
>>>>>>>>>> amf3 synchronization issue with SharedObjects (I will
>>>>>>>>>> provide Joachim with a test app)
>>>>>>>>>> fix our tests since Paul noticed many of them are failing
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If anyone can think of any other important fixes, list them
>>>>>>>>>> so that we can look into their priority. Thanks team!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> It is difficult to free fools from the chains they revere.
>>>>>>>>>> - Voltaire _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Luke Hubbard
>>>>>>> codegent | coding for the people
>>>>>>> http://www.codegent.com
>>>>>>>
>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>> http://www.codegent.com/top100/
>>>>>>>
>>>>>>> want to know more?
>>>>>>> http://www.codegent.com/showreel/
>>>>>>>
>>>>>>> This e-mail may contain information which is privileged,
>>>>>>> confidential and protected from disclosure. If you are not the
>>>>>>> intended recipient of this e-mail, or any part of it, please
>>>>>>> delete this email and any attachments immediately on receipt.
>>>>>>> You should not disclose the contents to any other person or
>>>>>>> take copies. Any views expressed in this message are those of
>>>>>>> the individual sender, except where the sender specifically
>>>>>>> states them to be the views of codegent limited.
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>> accomplishment. -- Ricky Jay
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> Steven Gong
>>>>>>
>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>
>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>> ~steven/Home
>>>>>>
>>>>>> Modesty is an overrated quality in men of no great
>>>>>> accomplishment. -- Ricky Jay
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luke Hubbard
>>>>>> codegent | coding for the people
>>>>>> http://www.codegent.com
>>>>>>
>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>> http://www.codegent.com/top100/
>>>>>>
>>>>>> want to know more?
>>>>>> http://www.codegent.com/showreel/
>>>>>>
>>>>>> This e-mail may contain information which is privileged,
>>>>>> confidential and protected from disclosure. If you are not the
>>>>>> intended recipient of this e-mail, or any part of it, please
>>>>>> delete this email and any attachments immediately on receipt.
>>>>>> You should not disclose the contents to any other person or
>>>>>> take copies. Any views expressed in this message are those of
>>>>>> the individual sender, except where the sender specifically
>>>>>> states them to be the views of codegent limited.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> It is difficult to free fools from the chains they revere. -
>>>>>> Voltaire
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> Steven Gong
>>>>>>
>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>
>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>> ~steven/Home
>>>>>>
>>>>>> Modesty is an overrated quality in men of no great
>>>>>> accomplishment. -- Ricky Jay
>>>>>> _______________________________________________
>>>>>>
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> Steven Gong
>>>>>
>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>
>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>> ~steven/Home
>>>>>
>>>>> Modesty is an overrated quality in men of no great
>>>>> accomplishment. -- Ricky Jay
>>>>> _______________________________________________
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>> _______________________________________________
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>> _______________________________________________
>>>
>>> Red5devs mailing list
>>> Red5devs at osflash.org
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>> _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -
>> Voltaire _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5devs_osflash.org/attachments/20080129/191dcb2c/attachment-0001.html
More information about the Red5devs
mailing list