[Red5] performance questions

Dan Rossi spam at electroteque.org
Wed Jan 2 05:28:57 PST 2008 

Ive updated with trunk, both servers say 50 connections each in  
adminPanel though im pretty sure there is loads more than that playing  
files. About 12-99% on the cpu. It jumps up and down constantly. At  
times it will just sit on 100% and then fail connections.

Still getting these errors after 5 mins bringing the server up, i  
believe when this happens the flash player fails to connect !

INFO   | jvm 1    | 2008/01/02 08:20:44 | 2008-01-02 08:20:44,058  
[pool-5-thread-18] ERROR  
org.red5.server.stream.PlaylistSubscriberStream - Error while pulling  
message.
INFO   | jvm 1    | 2008/01/02 08:20:44 |  
java.util.concurrent.RejectedExecutionException
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java.util.concurrent.ThreadPoolExecutor 
$AbortPolicy.rejectedExecution(Unknown Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java 
.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown  
Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java 
.util 
.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(Unknown  
Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.red5.server.stream.PlaylistSubscriberStream 
$PlayEngine.ensurePullAndPushRunning(PlaylistSubscriberStream.java:1402)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.red5.server.stream.PlaylistSubscriberStream 
$PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1464)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access 
$400(PlaylistSubscriberStream.java:792)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org 
.red5 
.server 
.stream.PlaylistSubscriberStream.written(PlaylistSubscriberStream.java: 
599)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org 
.red5.server.net.rtmp.BaseRTMPHandler.messageSent(BaseRTMPHandler.java: 
229)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org 
.red5 
.server.net.rtmp.RTMPMinaIoHandler.messageSent(RTMPMinaIoHandler.java: 
188)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.apache.mina.common.support.AbstractIoFilterChain 
$TailFilter.messageSent(AbstractIoFilterChain.java:579)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org 
.apache 
.mina 
.common 
.support 
.AbstractIoFilterChain.callNextMessageSent(AbstractIoFilterChain.java: 
320)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.apache.mina.common.support.AbstractIoFilterChain.access 
$1200(AbstractIoFilterChain.java:53)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl 
$1.messageSent(AbstractIoFilterChain.java:653)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org 
.apache 
.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java: 
222)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
org.apache.mina.filter.executor.ExecutorFilter 
$ProcessEventsRunnable.run(ExecutorFilter.java:264)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
java.lang.Thread.run(Unknown Source)


Ive had to revert to the old distribution for now ill try again  
tommorow, still not working unfortunately.

What trunk version do I need to try and checkout ?


On 02/01/2008, at 11:56 PM, Steven Gong wrote:

> Maybe you can try an earliest verson with logback stuff checkin.
>
> On Jan 2, 2008 8:42 PM, Dan Rossi <spam at electroteque.org> wrote:
> Sorry i gave it a try its not possible, im in too deep with the  
> logback stuff now I cant get it to load on the old versions even  
> copying over the logback and sl4j files complains about missing sl4j  
> classes.
>
> On 02/01/2008, at 11:04 PM, Dan Rossi wrote:
>
>> thats a bit of a problem the webapps need a bit of refactoring for  
>> 0.6.3 ill let you know.
>>
>> On 02/01/2008, at 10:24 PM, Steven Gong wrote:
>>
>>> Dan,
>>>
>>> On Jan 2, 2008 7:16 PM, Dan Rossi < spam at electroteque.org> wrote:
>>> Steve I sent you the custom webapp aswell as the GC flags I  
>>> believe the garbage collection flags were partially causing the  
>>> high CPU, just setting the min and max memory settings doesnt  
>>> really improve the cpu though. In our busy periods its peaking at  
>>> 100% , im not really sure what im suppose to increase here sorry  
>>> about that or debug this.
>>>
>>> I believe there was a change in trunk recently about an update to  
>>> mina, could this have been the cause for the streaming issues  
>>> under load ?
>>> Don't know if the mina lib update is the cause. But the update is  
>>> very recent change. Can you try the load on 0.6.3 to see if it  
>>> exists? I also saw the high CPU usage here with the trunk and will  
>>> use profiling tool to see where is the bottleneck...
>>>
>>>
>>> On 30/12/2007, at 1:20 PM, Steven Gong wrote:
>>>
>>>>
>>>>
>>>> On Dec 30, 2007 10:04 AM, Dan Rossi < spam at electroteque.org> wrote:
>>>> Steve this is from an old trunk or previous versions. I cannot  
>>>> get the latest changes working yet re previous emails where  
>>>> stream will cut out. I'm guessing i need to resolve that problem,  
>>>> get them working and then review performance then I think.
>>>>
>>>> How can we resolve the stream cutting out issue ?
>>>> Well, it's hard to answer this question as I need to find a way  
>>>> to reproduce this issue. Can you help to describe how I can  
>>>> reproduce this problem?
>>>>
>>>>
>>>> On 30/12/2007, at 11:20 AM, Steven Gong wrote:
>>>>
>>>>> Dan,
>>>>> Thanks for the info. Can you submit this to JIRA?
>>>>>
>>>>> On Dec 29, 2007 10:01 PM, Dan Rossi < spam at electroteque.org>  
>>>>> wrote:
>>>>> Sorry about that not up with the technical terms. I think its  
>>>>> been a problem for a while. I run them inside a java service  
>>>>> wrapper.
>>>>>
>>>>> So the entire config is
>>>>>
>>>>> 4 x
>>>>>
>>>>> Processor: 	P4 3.0Ghz
>>>>> RAM:	 	2GB
>>>>> Drives:	  	1  160 GB SATA (RAID: None)
>>>>>
>>>>> Windows 2003 Server
>>>>>
>>>>> Java6 with java service wrapper and GC java arguments
>>>>>
>>>>> Gigabit ethernet on public and lan
>>>>>
>>>>> Single storage controller machine with Dell MD1000 NAS rack  
>>>>> attached for 3TB of storage, all machines get the content over  
>>>>> the gigabit lan and shared drive.
>>>>>
>>>>> It will start to peak after a certain amount of connections it  
>>>>> seems, at about 20 connections it will display about up to 30%.   
>>>>> Windows media has hundreds of connections and 1% cpu load.
>>>>>
>>>>> We are due to switch a system over to default to flash instead  
>>>>> so may become a problem at hundreds of connections.
>>>>>
>>>>> On 30/12/2007, at 12:32 AM, Steven Gong wrote:
>>>>>
>>>>>> Dan,
>>>>>> I mean whether it's a regression bug or a new issue you found.
>>>>>>
>>>>>> On Dec 29, 2007 6:41 PM, Dan Rossi < spam at electroteque.org>  
>>>>>> wrote:
>>>>>>
>>>>>> On 29/12/2007, at 12:05 PM, Steven Gong wrote:
>>>>>>
>>>>>>> Dan,
>>>>>>> Is it a regression?
>>>>>>
>>>>>> Sorry steve what do you mean by regression ? Its just normal  
>>>>>> traffic, I dont have any such tool to try and do it manually.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Dec 29, 2007 2:39 AM, Dan Rossi <spam at electroteque.org >  
>>>>>>> wrote:
>>>>>>> Hi there after looking at the connections in the adminPanel im
>>>>>>> noticing that our machines are under quite a bit of stress for  
>>>>>>> VOD
>>>>>>> streams with less than 100 concurrent connections. What mina  
>>>>>>> settings
>>>>>>> should i tweak to make it possible to fix these issues in the  
>>>>>>> JVM ?
>>>>>>> Its always been like this but is now going from 60-100% on one  
>>>>>>> machine
>>>>>>> in particular.
>>>>>>>
>>>>>>> Let me know thanks.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>> accomplishment.  -- Ricky Jay  
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Best Regards
>>>>>> Steven Gong
>>>>>>
>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>
>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>> ~steven/Home
>>>>>>
>>>>>> Modesty is an overrated quality in men of no great  
>>>>>> accomplishment.  -- Ricky Jay  
>>>>>> _______________________________________________
>>>>>>
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> Best Regards
>>>>> Steven Gong
>>>>>
>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>
>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>> ~steven/Home
>>>>>
>>>>> Modesty is an overrated quality in men of no great  
>>>>> accomplishment.  -- Ricky Jay  
>>>>> _______________________________________________
>>>>>
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> Best Regards
>>>> Steven Gong
>>>>
>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>
>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>> ~steven/Home
>>>>
>>>> Modesty is an overrated quality in men of no great  
>>>> accomplishment.  -- Ricky Jay  
>>>> _______________________________________________
>>>>
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>>
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>>
>>>
>>>
>>> -- 
>>> Best Regards
>>> Steven Gong
>>>
>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>
>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>> ~steven/Home
>>>
>>> Modesty is an overrated quality in men of no great  
>>> accomplishment.  -- Ricky Jay  
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
>
> -- 
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
> ~steven/Home
>
> Modesty is an overrated quality in men of no great accomplishment.   
> -- Ricky Jay _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20080103/352ca512/attachment-0001.html

More information about the Red5 mailing list