[Red5] performance questions

Wed Jan 2 05:58:35 PST 2008

sorry forgot to mention, tested with standalone build, im waiting for  
the new server to be added to the lan to test 2 x edge and single  
origin. Mind you with slave machines connecting to a master would all  
the load then be directed to the master server ? When i tested briefly  
i noticed this was the case.

On 03/01/2008, at 12:28 AM, Dan Rossi wrote:

> Ive updated with trunk, both servers say 50 connections each in  
> adminPanel though im pretty sure there is loads more than that  
> playing files. About 12-99% on the cpu. It jumps up and down  
> constantly. At times it will just sit on 100% and then fail  
> connections.
>
> Still getting these errors after 5 mins bringing the server up, i  
> believe when this happens the flash player fails to connect !
>
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 2008-01-02 08:20:44,058  
> [pool-5-thread-18] ERROR  
> org.red5.server.stream.PlaylistSubscriberStream - Error while  
> pulling message.
> INFO   | jvm 1    | 2008/01/02 08:20:44 |  
> java.util.concurrent.RejectedExecutionException
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java.util.concurrent.ThreadPoolExecutor 
> $AbortPolicy.rejectedExecution(Unknown Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java 
> .util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown  
> Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java 
> .util 
> .concurrent 
> .ScheduledThreadPoolExecutor.scheduleWithFixedDelay(Unknown Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.red5.server.stream.PlaylistSubscriberStream 
> $PlayEngine.ensurePullAndPushRunning(PlaylistSubscriberStream.java: 
> 1402)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.red5.server.stream.PlaylistSubscriberStream 
> $PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1464)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access 
> $400(PlaylistSubscriberStream.java:792)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org 
> .red5 
> .server 
> .stream 
> .PlaylistSubscriberStream.written(PlaylistSubscriberStream.java:599)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org 
> .red5 
> .server.net.rtmp.BaseRTMPHandler.messageSent(BaseRTMPHandler.java:229)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org 
> .red5 
> .server 
> .net.rtmp.RTMPMinaIoHandler.messageSent(RTMPMinaIoHandler.java:188)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.apache.mina.common.support.AbstractIoFilterChain 
> $TailFilter.messageSent(AbstractIoFilterChain.java:579)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org 
> .apache 
> .mina 
> .common 
> .support 
> .AbstractIoFilterChain 
> .callNextMessageSent(AbstractIoFilterChain.java:320)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.apache.mina.common.support.AbstractIoFilterChain.access 
> $1200(AbstractIoFilterChain.java:53)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl 
> $1.messageSent(AbstractIoFilterChain.java:653)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org 
> .apache 
> .mina 
> .filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:222)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> org.apache.mina.filter.executor.ExecutorFilter 
> $ProcessEventsRunnable.run(ExecutorFilter.java:264)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> INFO   | jvm 1    | 2008/01/02 08:20:44 | 	at  
> java.lang.Thread.run(Unknown Source)
>
>
> Ive had to revert to the old distribution for now ill try again  
> tommorow, still not working unfortunately.
>
> What trunk version do I need to try and checkout ?
>
>
> On 02/01/2008, at 11:56 PM, Steven Gong wrote:
>
>> Maybe you can try an earliest verson with logback stuff checkin.
>>
>> On Jan 2, 2008 8:42 PM, Dan Rossi <spam at electroteque.org> wrote:
>> Sorry i gave it a try its not possible, im in too deep with the  
>> logback stuff now I cant get it to load on the old versions even  
>> copying over the logback and sl4j files complains about missing  
>> sl4j classes.
>>
>> On 02/01/2008, at 11:04 PM, Dan Rossi wrote:
>>
>>> thats a bit of a problem the webapps need a bit of refactoring for  
>>> 0.6.3 ill let you know.
>>>
>>> On 02/01/2008, at 10:24 PM, Steven Gong wrote:
>>>
>>>> Dan,
>>>>
>>>> On Jan 2, 2008 7:16 PM, Dan Rossi < spam at electroteque.org> wrote:
>>>> Steve I sent you the custom webapp aswell as the GC flags I  
>>>> believe the garbage collection flags were partially causing the  
>>>> high CPU, just setting the min and max memory settings doesnt  
>>>> really improve the cpu though. In our busy periods its peaking at  
>>>> 100% , im not really sure what im suppose to increase here sorry  
>>>> about that or debug this.
>>>>
>>>> I believe there was a change in trunk recently about an update to  
>>>> mina, could this have been the cause for the streaming issues  
>>>> under load ?
>>>> Don't know if the mina lib update is the cause. But the update is  
>>>> very recent change. Can you try the load on 0.6.3 to see if it  
>>>> exists? I also saw the high CPU usage here with the trunk and  
>>>> will use profiling tool to see where is the bottleneck...
>>>>
>>>>
>>>> On 30/12/2007, at 1:20 PM, Steven Gong wrote:
>>>>
>>>>>
>>>>>
>>>>> On Dec 30, 2007 10:04 AM, Dan Rossi < spam at electroteque.org>  
>>>>> wrote:
>>>>> Steve this is from an old trunk or previous versions. I cannot  
>>>>> get the latest changes working yet re previous emails where  
>>>>> stream will cut out. I'm guessing i need to resolve that  
>>>>> problem, get them working and then review performance then I  
>>>>> think.
>>>>>
>>>>> How can we resolve the stream cutting out issue ?
>>>>> Well, it's hard to answer this question as I need to find a way  
>>>>> to reproduce this issue. Can you help to describe how I can  
>>>>> reproduce this problem?
>>>>>
>>>>>
>>>>> On 30/12/2007, at 11:20 AM, Steven Gong wrote:
>>>>>
>>>>>> Dan,
>>>>>> Thanks for the info. Can you submit this to JIRA?
>>>>>>
>>>>>> On Dec 29, 2007 10:01 PM, Dan Rossi < spam at electroteque.org>  
>>>>>> wrote:
>>>>>> Sorry about that not up with the technical terms. I think its  
>>>>>> been a problem for a while. I run them inside a java service  
>>>>>> wrapper.
>>>>>>
>>>>>> So the entire config is
>>>>>>
>>>>>> 4 x
>>>>>>
>>>>>> Processor: 	P4 3.0Ghz
>>>>>> RAM:	 	2GB
>>>>>> Drives:	  	1  160 GB SATA (RAID: None)
>>>>>>
>>>>>> Windows 2003 Server
>>>>>>
>>>>>> Java6 with java service wrapper and GC java arguments
>>>>>>
>>>>>> Gigabit ethernet on public and lan
>>>>>>
>>>>>> Single storage controller machine with Dell MD1000 NAS rack  
>>>>>> attached for 3TB of storage, all machines get the content over  
>>>>>> the gigabit lan and shared drive.
>>>>>>
>>>>>> It will start to peak after a certain amount of connections it  
>>>>>> seems, at about 20 connections it will display about up to  
>>>>>> 30%.  Windows media has hundreds of connections and 1% cpu load.
>>>>>>
>>>>>> We are due to switch a system over to default to flash instead  
>>>>>> so may become a problem at hundreds of connections.
>>>>>>
>>>>>> On 30/12/2007, at 12:32 AM, Steven Gong wrote:
>>>>>>
>>>>>>> Dan,
>>>>>>> I mean whether it's a regression bug or a new issue you found.
>>>>>>>
>>>>>>> On Dec 29, 2007 6:41 PM, Dan Rossi < spam at electroteque.org>  
>>>>>>> wrote:
>>>>>>>
>>>>>>> On 29/12/2007, at 12:05 PM, Steven Gong wrote:
>>>>>>>
>>>>>>>> Dan,
>>>>>>>> Is it a regression?
>>>>>>>
>>>>>>> Sorry steve what do you mean by regression ? Its just normal  
>>>>>>> traffic, I dont have any such tool to try and do it manually.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Dec 29, 2007 2:39 AM, Dan Rossi <spam at electroteque.org >  
>>>>>>>> wrote:
>>>>>>>> Hi there after looking at the connections in the adminPanel im
>>>>>>>> noticing that our machines are under quite a bit of stress  
>>>>>>>> for VOD
>>>>>>>> streams with less than 100 concurrent connections. What mina  
>>>>>>>> settings
>>>>>>>> should i tweak to make it possible to fix these issues in the  
>>>>>>>> JVM ?
>>>>>>>> Its always been like this but is now going from 60-100% on  
>>>>>>>> one machine
>>>>>>>> in particular.
>>>>>>>>
>>>>>>>> Let me know thanks.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>>> accomplishment.  -- Ricky Jay  
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great  
>>>>>>> accomplishment.  -- Ricky Jay  
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Best Regards
>>>>>> Steven Gong
>>>>>>
>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>
>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>>> ~steven/Home
>>>>>>
>>>>>> Modesty is an overrated quality in men of no great  
>>>>>> accomplishment.  -- Ricky Jay  
>>>>>> _______________________________________________
>>>>>>
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> Best Regards
>>>>> Steven Gong
>>>>>
>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>
>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>>> ~steven/Home
>>>>>
>>>>> Modesty is an overrated quality in men of no great  
>>>>> accomplishment.  -- Ricky Jay  
>>>>> _______________________________________________
>>>>>
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> Best Regards
>>>> Steven Gong
>>>>
>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>
>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>>>> ~steven/Home
>>>>
>>>> Modesty is an overrated quality in men of no great  
>>>> accomplishment.  -- Ricky Jay  
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>>
>> -- 
>> Best Regards
>> Steven Gong
>>
>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>
>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
>> ~steven/Home
>>
>> Modesty is an overrated quality in men of no great accomplishment.   
>> -- Ricky Jay _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20080103/116f9dec/attachment-0001.html 