[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)
Steven Gong
steven.gong at gmail.com
Wed Jan 30 06:08:06 PST 2008
OK, looks like we still need to do more profiling to fix the hot spots. Here
are some results from the profiling when 100 clients were playing VOD:
Channel.write() 33%
FileProvider.pullMessage() 29%
doPushMessage() 40%
encodePacket() 20%
The CPU usage of one function call might cover another call.
On Jan 30, 2008 7:26 PM, Dan Rossi <spam at electroteque.org> wrote:
> Here is top after using the loadtest tool on a live stream
> top - 22:28:04 up 3:49, 2 users, load average: 1.79, 0.57, 0.20
> Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie
> Cpu0 : 5.6%us, 8.3%sy, 0.0%ni, 85.7%id, 0.0%wa, 0.0%hi, 0.3%si,
> 0.0%st
> Cpu1 : 9.6%us, 4.0%sy, 0.0%ni, 80.1%id, 0.0%wa, 3.0%hi, 3.3%si,
> 0.0%st
> Mem: 1029420k total, 982700k used, 46720k free, 16k buffers
> Swap: 1959888k total, 20752k used, 1939136k free, 631824k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 7493 root 18 0 501m 114m 10m S 33 11.4 0:36.52 java
>
> I'll try starting red5 with GC flags and see what happens.
>
> On 29/01/2008, at 12:46 PM, Dan Rossi wrote:
>
> I have it back on the old copy and 48 requests gets about 22%. The problem
> is once it gets to 80 it also starts to max out. The new copies i keep
> trying to test max out much earlier than that. Its an issue because I cannot
> make a system using a custom webapp live until its somehow corrected
> because it uses alot of the logback changes for a custom access logging
> system for bandwidth billing of streams. I cannot seem to get the logback
> stuff to work in this version.
> On 29/01/2008, at 12:16 PM, Steven Gong wrote:
>
>
> On Jan 29, 2008 9:10 AM, Dan Rossi <spam at electroteque.org> wrote:
>
> > I have the latest svn on a server with flags set to false it seemed to
> > reach a higher cpu at 18 requests than with it set to true, im seeing peaks
> > at 90% with 34 requests, at 35 requests i got 100%. This doesnt add up
> > sorry. Maybe it has something to do with network storage ?
> >
> Yes, maybe if you are getting the VOD via network. How do you read the
> files?
>
> >
> >
> > On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
> >
> > With flag set to false and 100 requests
> >
> >
> > top - 09:29:58 up 53 min, 2 users, load average: 9.25, 5.00, 4.40
> > Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 40.3%us, 30.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi, 1.0%si,
> > 0.0%st
> > Cpu1 : 17.3%us, 22.3%sy, 0.0%ni, 44.7%id, 0.0%wa, 8.7%hi, 7.0%si,
> > 0.0%st
> > Mem: 1029420k total, 999888k used, 29532k free, 372k buffers
> > Swap: 1959888k total, 0k used, 1959888k free, 558692k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >
> > 7521 root 17 0 505m 136m 10m S 126 13.6 1:31.58 java
> >
> > With flag set to true and 100 requests
> >
> > 90 users 100% on one core of the cpu 20% overall.
> >
> > top - 09:25:42 up 49 min, 2 users, load average: 5.74, 3.23, 3.91
> > Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
> > Cpu0 : 17.5%us, 31.1%sy, 0.0%ni, 50.7%id, 0.0%wa, 0.0%hi, 0.7%si,
> > 0.0%st
> > Cpu1 : 15.8%us, 22.8%sy, 0.0%ni, 47.2%id, 0.0%wa, 7.6%hi, 6.6%si,
> > 0.0%st
> > Mem: 1029420k total, 1010608k used, 18812k free, 372k buffers
> > Swap: 1959888k total, 0k used, 1959888k free, 558836k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >
> > 7189 root 17 0 502m 146m 10m S 101 14.6 4:02.54 java
> >
> >
> >
> > I don't see much of a difference :\
> >
> >
> > On 29/01/2008, at 3:02 AM, Mondain wrote:
> >
> > There no "fix" yet, but I think you may be able to set *generateMetadata
> > *to *false *in the interim. In red5-common.xml look for this bean:
> > <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
> > <property name="generateMetadata" value="true"/>
> > </bean>
> > Simply setting it to false may prevent your CPU spikes.
> >
> > Paul
> >
> > On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
> >
> > > Is the possible fixes committed at all ? I just tried on my little
> > > ubuntu machine which is a 2.2G AMD X2 and reached 100% at 100 users
> > > and was averaging at 85% the whole way through with the loader tester app.
> > > On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
> > >
> > > I knew thats where it may have happened when the cache changed. The
> > > problem with this is has to identify files of same names inside different
> > > paths. I believe there was a clash there ! Please let me know how you go,
> > > some frustrated bosses on my end wanting to bite the bullet and go with FMS
> > > and 1 and a bit years nearly wasted.
> > > Please let me know and I can test.
> > >
> > > On 26/01/2008, at 8:46 AM, Chris Allen wrote:
> > >
> > > Guys,
> > > Let's get cranking on reverse engineering the H.264 stuff.
> > >
> > > http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
> > >
> > > :-)
> > >
> > > -Chris
> > >
> > >
> > > On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
> > >
> > >
> > >
> > > On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com> wrote:
> > >
> > > > Nice Steven! So, are you going to implement this fix?
> > > >
> > > Yes, I will provide the fix before the release.
> > >
> > > >
> > > > -Chris
> > > >
> > > > On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
> > > >
> > > > Hi all,
> > > > I found the root cause of big CPU load on VOD streaming. It is
> > > > caused by the loading of key frame metadata from metadata cache. A simple
> > > > solution to this is to enhance the metadata cache implementation to cache
> > > > the loaded metadata into memory for FLV rather than load it again from the
> > > > file system each time.
> > > >
> > > > On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
> > > >
> > > > > The load tester (src) is checked into the trunk btw..
> > > > >
> > > > >
> > > > > On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net > wrote:
> > > > >
> > > > > > Hi Steven
> > > > > >
> > > > > > Yes please do.
> > > > > >
> > > > > > - Luke
> > > > > >
> > > > > >
> > > > > > On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote:
> > > > > > >
> > > > > > > Luke,
> > > > > > > If you need Paul's load tester for the test, I can send a copy
> > > > > > > to you.
> > > > > > >
> > > > > > > On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Paul's load tester flex app is a good one to use but i dont
> > > > > > > > think its accurate enough. I have tested the same app against FMS and Wowza
> > > > > > > > and there was only 5% cpu used maximum.
> > > > > > > > On 21/01/2008, at 7:43 PM, Steven Gong wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I will try to test with a profiler see what is eating all
> > > > > > > > > that CPU.
> > > > > > > > > This is for VOD right, not live? Or both?
> > > > > > > > >
> > > > > > > > This is the VOD streaming and can be reproduced by a simple
> > > > > > > > load test VOD playback.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > - Luke
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
> > > > > > > > > >
> > > > > > > > > > Less than that before its reaching 100% cpu. aparantly
> > > > > > > > > > i go away on a break and everything goes down. It might be looking like we
> > > > > > > > > > might have to jump onto the wowza bandwagon :\
> > > > > > > > > > On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
> > > > > > > > > >
> > > > > > > > > > Dan:
> > > > > > > > > >
> > > > > > > > > > are you saying that the core can only handle 50-80
> > > > > > > > > > concurrent connections?
> > > > > > > > > >
> > > > > > > > > > On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
> > > > > > > > > >
> > > > > > > > > > Great, i added the preliminary task in there but didnt
> > > > > > > > > > get much responses about it at first :D Should we add back in the option to
> > > > > > > > > > either compile javadoc in a full distribution task or not, that was added in
> > > > > > > > > > as a suggestion from steve but was removed. It does reduce the recompile
> > > > > > > > > > time alot.
> > > > > > > > > > I believe the major one that i know of is the
> > > > > > > > > > performance problems somewhere in the core, its maxing out at 50-80
> > > > > > > > > > concurrents.
> > > > > > > > > >
> > > > > > > > > > On 19/01/2008, at 3:15 AM, Mondain wrote:
> > > > > > > > > >
> > > > > > > > > > If we plan to inform users of the edge/origin features,
> > > > > > > > > > we will also need to get those parts of the build script ironed-out before
> > > > > > > > > > the release as well.
> > > > > > > > > >
> > > > > > > > > > Paul
> > > > > > > > > >
> > > > > > > > > > On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Joachim,
> > > > > > > > > > >
> > > > > > > > > > > Welcome back!!
> > > > > > > > > > >
> > > > > > > > > > > Glad to see your return.
> > > > > > > > > > >
> > > > > > > > > > > Hope things are well with these days.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Lenny
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 1/17/08, Dominick Accattato < daccattato at gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Team:
> > > > > > > > > > > >
> > > > > > > > > > > > I want to give Joachim a warm welcome back. We
> > > > > > > > > > > > missed ya! Everyone has done a great job pushing Red5 forward from the
> > > > > > > > > > > > 0.6 release. In preparation of the release of 0.7,
> > > > > > > > > > > > I'd like to get a few more fixes in. If we can, lets try to get this stuff
> > > > > > > > > > > > done asap so that we can get 0.7 out the door. Here
> > > > > > > > > > > > is what I find to be of importance for this release.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > - http://jira.red5.org/browse/APPSERVER-184
> > > > > > > > > > > > - amf3 synchronization issue with
> > > > > > > > > > > > SharedObjects (I will provide Joachim with a test app)
> > > > > > > > > > > > - fix our tests since Paul noticed many of
> > > > > > > > > > > > them are failing
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If anyone can think of any other important fixes,
> > > > > > > > > > > > list them so that we can look into their priority. Thanks team!
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > Red5devs mailing list
> > > > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > > >
> > > > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Red5devs mailing list
> > > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > It is difficult to free fools from the chains they
> > > > > > > > > > revere. - Voltaire _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Red5devs mailing list
> > > > > > > > > > Red5devs at osflash.org
> > > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Luke Hubbard
> > > > > > > > > codegent | coding for the people
> > > > > > > > > http://www.codegent.com
> > > > > > > > >
> > > > > > > > > NMA Top 100 Interactive Agencies - Ones to watch!
> > > > > > > > > http://www.codegent.com/top100/
> > > > > > > > >
> > > > > > > > > want to know more?
> > > > > > > > > http://www.codegent.com/showreel/
> > > > > > > > >
> > > > > > > > > This e-mail may contain information which is privileged,
> > > > > > > > > confidential and protected from disclosure. If you are not the intended
> > > > > > > > > recipient of this e-mail, or any part of it, please delete this email and
> > > > > > > > > any attachments immediately on receipt. You should not disclose the contents
> > > > > > > > > to any other person or take copies. Any views expressed in this message are
> > > > > > > > > those of the individual sender, except where the sender specifically states
> > > > > > > > > them to be the views of codegent limited.
> > > > > > > > > _______________________________________________
> > > > > > > > > Red5devs mailing list
> > > > > > > > > Red5devs at osflash.org
> > > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best Regards
> > > > > > > > Steven Gong
> > > > > > > >
> > > > > > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > > > > > steven at infrared5.com
> > > > > > > >
> > > > > > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > > > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > > > > > >
> > > > > > > > Modesty is an overrated quality in men of no great
> > > > > > > > accomplishment. -- Ricky Jay
> > > > > > > > _______________________________________________
> > > > > > > > Red5devs mailing list
> > > > > > > > Red5devs at osflash.org
> > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Red5devs mailing list
> > > > > > > > Red5devs at osflash.org
> > > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > > Steven Gong
> > > > > > >
> > > > > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > > > > steven at infrared5.com
> > > > > > >
> > > > > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > > > > >
> > > > > > > Modesty is an overrated quality in men of no great
> > > > > > > accomplishment. -- Ricky Jay
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Red5devs mailing list
> > > > > > > Red5devs at osflash.org
> > > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Luke Hubbard
> > > > > > codegent | coding for the people
> > > > > > http://www.codegent.com
> > > > > >
> > > > > > NMA Top 100 Interactive Agencies - Ones to watch!
> > > > > > http://www.codegent.com/top100/
> > > > > >
> > > > > > want to know more?
> > > > > > http://www.codegent.com/showreel/
> > > > > >
> > > > > > This e-mail may contain information which is privileged,
> > > > > > confidential and protected from disclosure. If you are not the intended
> > > > > > recipient of this e-mail, or any part of it, please delete this email and
> > > > > > any attachments immediately on receipt. You should not disclose the contents
> > > > > > to any other person or take copies. Any views expressed in this message are
> > > > > > those of the individual sender, except where the sender specifically states
> > > > > > them to be the views of codegent limited.
> > > > > >
> > > > > > _______________________________________________
> > > > > > Red5devs mailing list
> > > > > > Red5devs at osflash.org
> > > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > It is difficult to free fools from the chains they revere. -
> > > > > Voltaire
> > > > >
> > > > > _______________________________________________
> > > > > Red5devs mailing list
> > > > > Red5devs at osflash.org
> > > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > > Steven Gong
> > > >
> > > > InfraRed5 Red5 Consultant: http://www.infrared5.com ,
> > > > steven at infrared5.com
> > > >
> > > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > > >
> > > > Modesty is an overrated quality in men of no great accomplishment.
> > > > -- Ricky Jay _______________________________________________
> > > > Red5devs mailing list
> > > > Red5devs at osflash.org
> > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Red5devs mailing list
> > > > Red5devs at osflash.org
> > > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > > Steven Gong
> > >
> > > InfraRed5 Red5 Consultant: http://www.infrared5.com,
> > > steven at infrared5.com
> > >
> > > Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/~steven/Home
> > > <http://jira.red5.org/confluence/display/%7Esteven/Home>
> > >
> > > Modesty is an overrated quality in men of no great accomplishment. --
> > > Ricky Jay _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> > >
> > > _______________________________________________
> > > Red5devs mailing list
> > > Red5devs at osflash.org
> > > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> > >
> > >
> >
> >
> > --
> > It is difficult to free fools from the chains they revere. - Voltaire
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
> >
> > _______________________________________________
> > Red5devs mailing list
> > Red5devs at osflash.org
> > http://osflash.org/mailman/listinfo/red5devs_osflash.org
> >
> >
>
>
> --
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5,
> http://jira.red5.org/confluence/display/~steven/Home<http://jira.red5.org/confluence/display/%7Esteven/Home>
>
> Modesty is an overrated quality in men of no great accomplishment. --
> Ricky Jay _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
--
Best Regards
Steven Gong
InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
Red5 Developer: http://osflash.org/red5,
http://jira.red5.org/confluence/display/~steven/Home
Modesty is an overrated quality in men of no great accomplishment. -- Ricky
Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5devs_osflash.org/attachments/20080130/b532b725/attachment-0001.html
More information about the Red5devs
mailing list