[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)

Dan Rossi spam at electroteque.org
Mon Jan 28 17:31:59 PST 2008 

Im using the filenamegenerator bean, ive sent a refactored version of  
our webapp offlist with the customisation in the filename generator to  
also create 'virtual' directories to mimmick FMS.

webapp.playbackPath=//videoStorageControllerMachine/NASSharePath/ 
FlashVideo/

Its linking using Windows UNC paths as the java service wrapper does  
not know a mapped network drive in the administrator login.


On 29/01/2008, at 12:16 PM, Steven Gong wrote:

>
> On Jan 29, 2008 9:10 AM, Dan Rossi <spam at electroteque.org> wrote:
> I have the latest svn on a server with flags set to false it seemed  
> to reach a higher cpu at 18 requests than with it set to true, im  
> seeing peaks at 90% with 34 requests, at 35 requests i got 100%.  
> This doesnt add up sorry. Maybe it has something to do with network  
> storage ?
> Yes, maybe if you are getting the VOD via network. How do you read  
> the files?
>
>
> On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
>
>> With flag set to false and 100 requests
>>
>>
>> top - 09:29:58 up 53 min,  2 users,  load average: 9.25, 5.00, 4.40
>> Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
>> Cpu0  : 40.3%us, 30.0%sy,  0.0%ni, 28.7%id,  0.0%wa,  0.0%hi,   
>> 1.0%si,  0.0%st
>> Cpu1  : 17.3%us, 22.3%sy,  0.0%ni, 44.7%id,  0.0%wa,  8.7%hi,   
>> 7.0%si,  0.0%st
>> Mem:   1029420k total,   999888k used,    29532k free,      372k  
>> buffers
>> Swap:  1959888k total,        0k used,  1959888k free,   558692k  
>> cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>  7521 root      17   0  505m 136m  10m S  126 13.6   1:31.58 java
>>
>> With flag set to true and 100 requests
>>
>> 90 users 100% on one core of the cpu 20% overall.
>>
>> top - 09:25:42 up 49 min,  2 users,  load average: 5.74, 3.23, 3.91
>> Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
>> Cpu0  : 17.5%us, 31.1%sy,  0.0%ni, 50.7%id,  0.0%wa,  0.0%hi,   
>> 0.7%si,  0.0%st
>> Cpu1  : 15.8%us, 22.8%sy,  0.0%ni, 47.2%id,  0.0%wa,  7.6%hi,   
>> 6.6%si,  0.0%st
>> Mem:   1029420k total,  1010608k used,    18812k free,      372k  
>> buffers
>> Swap:  1959888k total,        0k used,  1959888k free,   558836k  
>> cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>  7189 root      17   0  502m 146m  10m S  101 14.6   4:02.54 java
>>
>>
>> I don't see much of a difference :\
>>
>>
>> On 29/01/2008, at 3:02 AM, Mondain wrote:
>>
>>> There no "fix" yet, but I think you may be able to set  
>>> generateMetadata to false in the interim. In red5-common.xml look  
>>> for this bean:
>>> <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
>>>     <property name="generateMetadata" value="true"/>
>>> </bean>
>>> Simply setting it to false may prevent your CPU spikes.
>>>
>>> Paul
>>>
>>> On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
>>> Is the possible fixes committed at all ? I just tried on my little  
>>> ubuntu machine which is a 2.2G AMD X2 and reached 100% at 100  
>>> users and was averaging at 85% the whole way through with the  
>>> loader tester app.
>>>
>>> On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
>>>
>>>> I knew thats where it may have happened when the cache changed.  
>>>> The problem with this is has to identify files of same names  
>>>> inside different paths. I believe there was a clash there !  
>>>> Please let me know how you go, some frustrated bosses on my end  
>>>> wanting to bite the bullet and go with FMS and 1 and a bit years  
>>>> nearly wasted.
>>>>
>>>> Please let me know and I can test.
>>>>
>>>> On 26/01/2008, at 8:46 AM, Chris Allen wrote:
>>>>
>>>>> Guys,
>>>>>
>>>>> Let's get cranking on reverse engineering the H.264 stuff.
>>>>>
>>>>> http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
>>>>>
>>>>> :-)
>>>>>
>>>>> -Chris
>>>>>
>>>>>
>>>>> On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com>  
>>>>>> wrote:
>>>>>> Nice Steven! So, are you going to implement this fix?
>>>>>> Yes, I will provide the fix before the release.
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>> On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>> I found the root cause of big CPU load on VOD streaming. It is  
>>>>>>> caused by the loading of key frame metadata from metadata  
>>>>>>> cache. A simple solution to this is to enhance the metadata  
>>>>>>> cache implementation to cache the loaded metadata into memory  
>>>>>>> for FLV rather than load it again from the file system each  
>>>>>>> time.
>>>>>>>
>>>>>>> On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
>>>>>>> The load tester (src) is checked into the trunk btw..
>>>>>>>
>>>>>>>
>>>>>>> On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net >  
>>>>>>> wrote:
>>>>>>> Hi Steven
>>>>>>>
>>>>>>> Yes please do.
>>>>>>>
>>>>>>> - Luke
>>>>>>>
>>>>>>>
>>>>>>> On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote:
>>>>>>> Luke,
>>>>>>> If you need Paul's load tester for the test, I can send a copy  
>>>>>>> to you.
>>>>>>>
>>>>>>>
>>>>>>> On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>  
>>>>>>> wrote:
>>>>>>> Paul's load tester flex app is a good one to use but i dont  
>>>>>>> think its accurate enough. I have tested the same app against  
>>>>>>> FMS and Wowza and there was only 5% cpu used maximum.
>>>>>>>
>>>>>>> On 21/01/2008, at 7:43 PM, Steven Gong wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>  
>>>>>>>> wrote:
>>>>>>>> I will try to test with a profiler see what is eating all  
>>>>>>>> that CPU.
>>>>>>>> This is for VOD right, not live? Or both?
>>>>>>>> This is the VOD streaming and can be reproduced by a simple  
>>>>>>>> load test VOD playback.
>>>>>>>>
>>>>>>>> - Luke
>>>>>>>>
>>>>>>>>
>>>>>>>> On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
>>>>>>>> Less than that before its reaching 100% cpu. aparantly i go  
>>>>>>>> away on a break and everything goes down. It might be looking  
>>>>>>>> like we might have to jump onto the wowza bandwagon :\
>>>>>>>>
>>>>>>>> On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
>>>>>>>>
>>>>>>>>> Dan:
>>>>>>>>>
>>>>>>>>> are you saying that the core can only handle 50-80  
>>>>>>>>> concurrent connections?
>>>>>>>>>
>>>>>>>>> On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
>>>>>>>>>
>>>>>>>>>> Great, i added the preliminary task in there but didnt get  
>>>>>>>>>> much responses about it at first :D Should we add back in  
>>>>>>>>>> the option to either compile javadoc in a full distribution  
>>>>>>>>>> task or not, that was added in as a suggestion from steve  
>>>>>>>>>> but was removed. It does reduce the recompile time alot.
>>>>>>>>>>
>>>>>>>>>> I believe the major one that i know of is the performance  
>>>>>>>>>> problems somewhere in the core, its maxing out at 50-80  
>>>>>>>>>> concurrents.
>>>>>>>>>>
>>>>>>>>>> On 19/01/2008, at 3:15 AM, Mondain wrote:
>>>>>>>>>>
>>>>>>>>>>> If we plan to inform users of the edge/origin features, we  
>>>>>>>>>>> will also need to get those parts of the build script  
>>>>>>>>>>> ironed-out before the release as well.
>>>>>>>>>>>
>>>>>>>>>>> Paul
>>>>>>>>>>>
>>>>>>>>>>> On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>  
>>>>>>>>>>> wrote:
>>>>>>>>>>> Hey Joachim,
>>>>>>>>>>>
>>>>>>>>>>> Welcome back!!
>>>>>>>>>>>
>>>>>>>>>>> Glad to see your return.
>>>>>>>>>>>
>>>>>>>>>>> Hope things are well with these days.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Lenny
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 1/17/08, Dominick Accattato < daccattato at gmail.com>  
>>>>>>>>>>> wrote:
>>>>>>>>>>> Team:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I want to give Joachim a warm welcome back.  We missed  
>>>>>>>>>>> ya!  Everyone has done a great job pushing Red5 forward  
>>>>>>>>>>> from the 0.6 release.  In preparation of the release of  
>>>>>>>>>>> 0.7, I'd like to get a few more fixes in.  If we can, lets  
>>>>>>>>>>> try to get this stuff done asap so that we can get 0.7 out  
>>>>>>>>>>> the door.  Here is what I find to be of importance for  
>>>>>>>>>>> this release.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://jira.red5.org/browse/APPSERVER-184
>>>>>>>>>>> amf3 synchronization issue with SharedObjects (I will  
>>>>>>>>>>> provide Joachim with a test app)
>>>>>>>>>>> fix our tests since Paul noticed many of them are failing
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If anyone can think of any other important fixes, list  
>>>>>>>>>>> them so that we can look into their priority.  Thanks team!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> It is difficult to free fools from the chains they revere.  
>>>>>>>>>>> - Voltaire _______________________________________________
>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Luke Hubbard
>>>>>>>> codegent | coding for the people
>>>>>>>> http://www.codegent.com
>>>>>>>>
>>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>>> http://www.codegent.com/top100/
>>>>>>>>
>>>>>>>> want to know more?
>>>>>>>> http://www.codegent.com/showreel/
>>>>>>>>
>>>>>>>> This e-mail may contain information which is privileged,  
>>>>>>>> confidential and protected from disclosure. If you are not  
>>>>>>>> the intended recipient of this e-mail, or any part of it,  
>>>>>>>> please delete this email and any attachments immediately on  
>>>>>>>> receipt. You should not disclose the contents to any other  
>>>>>>>> person or take copies. Any views expressed in this message  
>>>>>>>> are those of the individual sender, except where the sender  
>>>>>>>> specifically states them to be the views of codegent limited.
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>>> accomplishment.  -- Ricky Jay  
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>> accomplishment.  -- Ricky Jay
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Luke Hubbard
>>>>>>> codegent | coding for the people
>>>>>>> http://www.codegent.com
>>>>>>>
>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>> http://www.codegent.com/top100/
>>>>>>>
>>>>>>> want to know more?
>>>>>>> http://www.codegent.com/showreel/
>>>>>>>
>>>>>>> This e-mail may contain information which is privileged,  
>>>>>>> confidential and protected from disclosure. If you are not the  
>>>>>>> intended recipient of this e-mail, or any part of it, please  
>>>>>>> delete this email and any attachments immediately on receipt.  
>>>>>>> You should not disclose the contents to any other person or  
>>>>>>> take copies. Any views expressed in this message are those of  
>>>>>>> the individual sender, except where the sender specifically  
>>>>>>> states them to be the views of codegent limited.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> It is difficult to free fools from the chains they revere. -  
>>>>>>> Voltaire
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>> accomplishment.  -- Ricky Jay  
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Best Regards
>>>>>> Steven Gong
>>>>>>
>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>>
>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>> ~steven/Home
>>>>>>
>>>>>> Modesty is an overrated quality in men of no great  
>>>>>> accomplishment.  -- Ricky Jay  
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>> _______________________________________________
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>> _______________________________________________
>>>>
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>>
>>> _______________________________________________
>>> Red5devs mailing list
>>> Red5devs at osflash.org
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>>
>>>
>>>
>>> -- 
>>> It is difficult to free fools from the chains they revere. -  
>>> Voltaire _______________________________________________
>>> Red5devs mailing list
>>> Red5devs at osflash.org
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>> _______________________________________________
>>
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
>
>
> -- 
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
> ~steven/Home
>
> Modesty is an overrated quality in men of no great accomplishment.   
> -- Ricky Jay _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5devs_osflash.org/attachments/20080129/a16d0ccc/attachment-0001.html

More information about the Red5devs mailing list