[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)
Steven Gong
steven.gong at gmail.com
Tue Jan 29 06:28:17 PST 2008
Hi Dan,
I committed some fixes to the VOD playback performance issue. The metadata
issue is not checked in but I think this should not impact the CPU usage
during the playback as metadata generation will only occur at the beginning.
Could you please try to test the trunk and tell me the result? Thanks.
On Jan 29, 2008 9:46 AM, Dan Rossi <spam at electroteque.org> wrote:
> I have it back on the old copy and 48 requests gets about 22%. The problem
> is once it gets to 80 it also starts to max out. The new copies i keep
> trying to test max out much earlier than that. Its an issue because I cannot
> make a system using a custom webapp live until its somehow corrected
> because it uses alot of the logback changes for a custom access logging
> system for bandwidth billing of streams. I cannot seem to get the logback
> stuff to work in this version.
> On 29/01/2008, at 12:16 PM, Steven Gong wrote:
>
>
> On Jan 29, 2008 9:10 AM, Dan Rossi <spam at electroteque.org> wrote:
>
> > I have the latest svn on a server with flags set to false it seemed to
> > reach a higher cpu at 18 requests than with it set to true, im seeing peaks
> > at 90% with 34 requests, at 35 requests i got 100%. This doesnt add up
> > sorry. Maybe it has something to do with network storage ?
> >
> Yes, maybe if you are getting the VOD via network. How do you read the
> files?
>
> >
> >
> > On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
> >
> > With flag set to false and 100 requests
> >
> >
> > top - 09:29:58 up 53 min, 2 users, load average: 9.25, 5.00, 4.40
> > Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 40.3%us, 30.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi, 1.0%si,
> > 0.0%st
> > Cpu1 : 17.3%us, 22.3%sy, 0.0%ni, 44.7%id, 0.0%wa, 8.7%hi, 7.0%si,
> > 0.0%st
> > Mem: 1029420k total, 999888k used, 29532k free, 372k buffers
> > Swap: 1959888k total, 0k used, 1959888k free, 558692k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >
> > 7521 root 17 0 505m 136m 10m S 126 13.6 1:31.58 java
> >
> > With flag set to true and 100 requests
> >
> > 90 users 100% on one core of the cpu 20% overall.
> >
> > top - 09:25:42 up 49 min, 2 users, load average: 5.74, 3.23, 3.91
> > Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 17.5%us, 31.1%sy, 0.0%ni, 50.7%id, 0.0%wa, 0.0%hi, 0.7%si,
> > 0.0%st
> > Cpu1 : 15.8%us, 22.8%sy, 0.0%ni, 47.2%id, 0.0%wa, 7.6%hi, 6.6%si,
> > 0.0%st
> > Mem: 1029420k total, 1010608k used, 18812k free, 372k buffers
> > Swap: 1959888k total, 0k used, 1959888k free, 558836k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >
> > 7189 root 17 0 502m 146m 10m S 101 14.6 4:02.54 java
> >
> >
> >
> > I don't see much of a difference :\
> >
> >
> > On 29/01/2008, at 3:02 AM, Mondain wrote:
> >
> > There no "fix" yet, but I think you may be able to set *generateMetadata
> > *to *false *in the interim. In red5-common.xml look for this bean:
> > <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
> > <property name="generateMetadata" value="true"/>
> > </bean>
> > Simply setting it to false may prevent your CPU spikes.
> >
> > Paul
> >
> > On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
> >
> > > Is the possible fixes committed at all ? I just tried on my little
> > > ubuntu machine which is a 2.2G AMD X2 and reached 100% at 100 users
> > > and was averaging at 85% the whole way through with the loader tester app.
> > > On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
> > >
> > > I knew thats where it may have happened when the cache changed. The
> > > problem with this is has to identify files of same names inside different
> > > paths. I believe there was a clash there ! Please let me know how you go,
> > > some frustrated bosses on my end wanting to bite the bullet and go with FMS
> > > and 1 and a bit years nearly wasted.
> > > Please let me know and I can test.
> > >
> > > On 26/01/2008, at 8:46 AM, Chris Allen wrote:
> > >
> > > Guys,
> > > Let's get cranking on reverse engineering the H.264 stuff.
> > >
> > > http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
> > >
> > > :-)
> > >
> > > -Chris
> > >
> > >
> > > On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
> > >
> > >
> > >
> > > On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com> wrote:
> > >
> > > > Nice Steven! So, are you going to implement this fix?
> > > >
> > > Yes, I will provide the fix before the release.
> > >
> > > >
> > > > -Chris
> > > >
> > > > On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
> > > >
> > > > Hi all,
> > > > I found the root cause of big CPU load on VOD streaming. It is
> > > > caused by the loading of key frame metadata from metadata cache. A simple
> > > > solution to this is to enhance the metadata cache implementation to cache
> > > > the loaded metadata into memory for FLV rather than load it again from the
> > > > file system each time.
> > > >
> > > > On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
> > > >
> > > > > The load tester (src) is checked into the trunk btw..
> > > > >
> > > > >
> > > > > On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net > wrote:
> > > > >
> > > > > > Hi Steven
> > > > > >
> > > > > > Yes please do.
> > > > > >
> > > > > > - Luke
> > > > > >
> > > > > >
> > > > > > On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote:
> > > > > > >
> > > > > > > Luke,
> > > > > > > If you need Paul's load tester for the test, I can send a copy
> > > > > > > to you.
> > > > > > >
> > > > > > > On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Paul's load tester flex app is a good one to use but i dont
> > > > > > > > think its accurate enough. I have tested the same app against FMS and Wowza
> > > > > > > > and there was only 5% cpu used maximum.
> > > > > > > > On 21/01/2008, at 7:43 PM, Steven Gong wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I will try to test with a profiler see what is eating all
> > > > > > > > > that CPU.
> > > > > > > > > This is for VOD right, not live? Or both?
> > > > > > > > >
> > > > > > > > This is the VOD streaming and can be reproduced by a simple
> > > > > > > > load test VOD playback.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > - Luke
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
> > > > > > > > > >
> > > > > > > > > > Less than that before its reaching 100% cpu. aparantly
> > > > > > > > > > i go away on a break and everything goes down. It might be looking like we
> > > > > > > > > > might have to jump onto the wowza bandwagon :\
> > > > > > > > > > On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
> > > > > > > > > >
> > > > > > > > > > Dan:
> > > > > > > > > >
> > > > > > > > > > are you saying that the core can only handle 50-80
> > > > > > > > > > concurrent connections?
> > > > > > > > > >
> > > > > > > > > > On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
> > > > > > > > > >
> > > > > > > > > > Great, i added the preliminary task in there but didnt
> > > > > > > > > > get much responses about it at first :D Should we add back in the option to
> > > > > > > > > > either compile javadoc in a full distribution task or not, that was added in
> > > > > > > > > > as a suggestion from steve but was removed. It does reduce the recompile
> > > > > > > > > > time alot.
> > > > > > > > > > I believe the major one that i know of is the
> > > > > > > > > > performance problems somewhere in the core, its maxing out at 50-80
> > > > > > > > > > concurrents.
> > > > > > > > > >
> > > > > > > > > > On 19/01/2008, at 3:15 AM, Mondain wrote:
> > > > > > > > > >
> > > > > > > > > > If we plan to inform users of the edge/origin features,
> > > > > > > > > > we will also need to get those parts of the build script ironed-out before
> > > > > > > > > > the release as well.
> > > > > > > > > >
> > > > > > > > > > Paul
> > > > > > > > > >
> > > > > > > > > > On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Joachim,
> > > > > > > > > > >
> > > > > > > > > > > Welcome back!!
> > > > > > > > > > >
> > > > > > > > > > > Glad to see your return.
> > > > > > > > > > >
> > > > > > > > > > > Hope things are well with these days.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Lenny
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 1/17/08, Dominick Accattato < daccattato at gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Team:
> > > > > > > > > > > >
> > > > > > > > > > > > I want to give Joachim a warm welcome back. We
> > > > > > > > > > > > missed ya! Everyone has done a great job pushing Red5 forward from the
> > > > > > > > > > > > 0.6 release. In preparation of the release of 0.7,
> > > > > > > > > > > > I'd like to get a few more fixes in. If we can, lets try to get this stuff
> > > > > > > > > > > > done asap so that we can get 0.7 out the door. Here
> > > > > > > > > > > > is what I find to be of importance for this release.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > - http://jira.red5.org/browse/APPSERVER-184
> > > > > > > > > > > > - amf3 synchronization issue with
> > > > > > > > > > > > SharedObjects (I will provide Joachim with a test app)
> > > > > > > > > > > > - fix our tests since Paul noticed many of
> > > > > > > > > > > > them are failing
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If anyone can think of any other important fixes,
> > > > > > > > > > > > list them so that we can look into their priority. Thanks team!
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > Red5devs mailing list
> > > > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > > >
> > > > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Red5devs mailing list
> > > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > It is difficult to free fools from the chains they
> > > > > > > > > > revere. - Voltaire _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Luke Hubbard
> > > > > > > > > codegent | coding for the people
> > > > > > > > > http://www.codegent.com
> > > > > > > > >
> > > > > > > > > NMA Top 100 Interactive Agencies - Ones to watch!
> > > > > > > > > http://www.codegent.com/top100/
> > > > > > > > >
> > > > > > > > > want to know more?
> > > > > > > > > http://www.codegent.com/showreel/
> > > > > > > > >
> > > > > > > > > This e-mail may contain information which is privileged,
> > > > > > > > > confidential and protected from disclosure. If you are not the intended
> > > > > > > > > recipient of this e-mail, or any part of it, please delete this email and
> > > > > > > > > any attachments immediately on receipt. You should not disclose the contents
> > > > > > > > > to any other person or take copies. Any views expressed in this message are
> > > > > > > > > those of the individual sender, except where the sender specifically states
> > > > > > > > > them to be the views of codegent limited.
> > > > > > > > > _______________________________________________
> > > > > > > > > Red5devs mailing list
> > > > > > > > > Red5devs at osflash.org
> > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Regards
> > > > > > > > Steven Gong
> > > > > > > >
> > > > > > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > > > > > steven at infrared5.com
> > > > > > > >
> > > > > > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > > > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > > > > > >
> > > > > > > > Modesty is an overrated quality in men of no great
> > > > > > > > accomplishment. -- Ricky Jay
> > > > > > > > _______________________________________________
> > > > > > > > Red5devs mailing list
> > > > > > > > Red5devs at osflash.org
> > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Red5devs mailing list
> > > > > > > > Red5devs at osflash.org
> > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > > Steven Gong
> > > > > > >
> > > > > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > > > > steven at infrared5.com
> > > > > > >
> > > > > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > > > > >
> > > > > > > Modesty is an overrated quality in men of no great
> > > > > > > accomplishment. -- Ricky Jay
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Red5devs mailing list
> > > > > > > Red5devs at osflash.org
> > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Luke Hubbard
> > > > > > codegent | coding for the people
> > > > > > http://www.codegent.com
> > > > > >
> > > > > > NMA Top 100 Interactive Agencies - Ones to watch!
> > > > > > http://www.codegent.com/top100/
> > > > > >
> > > > > > want to know more?
> > > > > > http://www.codegent.com/showreel/
> > > > > >
> > > > > > This e-mail may contain information which is privileged,
> > > > > > confidential and protected from disclosure. If you are not the intended
> > > > > > recipient of this e-mail, or any part of it, please delete this email and
> > > > > > any attachments immediately on receipt. You should not disclose the contents
> > > > > > to any other person or take copies. Any views expressed in this message are
> > > > > > those of the individual sender, except where the sender specifically states
> > > > > > them to be the views of codegent limited.
> > > > > >
> > > > > > _______________________________________________
> > > > > > Red5devs mailing list
> > > > > > Red5devs at osflash.org
> > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > It is difficult to free fools from the chains they revere. -
> > > > > Voltaire
> > > > >
> > > > > _______________________________________________
> > > > > Red5devs mailing list
> > > > > Red5devs at osflash.org
> > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > > Steven Gong
> > > >
> > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > steven at infrared5.com
> > > >
> > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > >
> > > > Modesty is an overrated quality in men of no great accomplishment.
> > > > -- Ricky Jay _______________________________________________
> > > > Red5devs mailing list
> > > > Red5devs at osflash.org
> > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Red5devs mailing list
> > > > Red5devs at osflash.org
> > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > > Steven Gong
> > >
> > > InfraRed5 Red5 Consultant: http://www.infrared5.com,
> > > steven at infrared5.com
> > >
> > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > >
> > > Modesty is an overrated quality in men of no great accomplishment. --
> > > Ricky Jay _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> >
> >
> > --
> > It is difficult to free fools from the chains they revere. - Voltaire
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
> >
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
>
>
> --
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5,
> http://jira.red5.org/confluence/display/~steven/Home<http://jira.red5.org/confluence/display/%7Esteven/Home>
>
> Modesty is an overrated quality in men of no great accomplishment. --
> Ricky Jay _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
--
Best Regards
Steven Gong
InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
Red5 Developer: http://osflash.org/red5,
http://jira.red5.org/confluence/display/~steven/Home
Modesty is an overrated quality in men of no great accomplishment. -- Ricky
Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5devs_osflash.org/attachments/20080129/d6c75033/attachment-0001.html
More information about the Red5devs
mailing list