[Red5] Memory leak questions

Daniel Rossi spam at electroteque.org
Tue Jun 24 01:05:45 PDT 2008 

thats a java version of cacti i believe.  Or does it need a special  
service on each server for the monitoring ?  I cant find examples of  
graphing the JVM stuff as you would see in jconsole or other apps like  
that.


On 24/06/2008, at 5:44 PM, Steven Gong wrote:

> There are also other NMS tools like Zenoss and OpenNMS that might be  
> helpful for system monitoring.
>
> On Tue, Jun 24, 2008 at 3:32 PM, Daniel Rossi  
> <spam at electroteque.org> wrote:
> So nagio can observe the JVM data directly then ? Is there a plugin  
> for this ?
>
> On 24/06/2008, at 5:21 PM, Tony Su wrote:
>
>> The client app running on the machine being monitored can be  
>> written in any language and even run in a sandbox (like Java). The  
>> only thing  that's important is <how> the app is able to collect  
>> data.
>>
>> So, although I'm not familiar with how any particular client  
>> running on a *NIX machine might do this,
>>
>> On a Windows machine today almost everything must run in a  
>> virtualized environment (32-bit or 64-bit virtual machines) called  
>> Protected Mode. The only notable exception is video. Only older OS  
>> like WinME, Win98/95 permit "real mode" runtimes which grant direct  
>> access to the hardware.
>>
>> But, on a Windows box, Microsoft has enabled several API that come  
>> pretty close to delivering data as close to real mode as possible,  
>> and you write your applications to those specified interfaces with  
>> their controlled options.
>>
>> Things weren't always this way. I remember comparing benchmark apps  
>> in Win98, you'd always be getting different figures whether the app  
>> ran in real or protected (virtualized) mode.
>>
>> Although I haven't studied Linux in as much detail, my general  
>> impression is that although most apps run in Protected Mode in  
>> Linux, the barriers to run in Real Mode aren't as great as they are  
>> in Windows.
>>
>> Tony
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On  
>> Behalf Of Daniel Rossi
>> Sent: Monday, June 23, 2008 8:33 PM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>> No java can export some JMX data via SNMP for monitoring and  
>> graphing in supported software for JVM specific information which  
>> is what is needed I'm guessing ? Does the OS display the JVM stuff  
>> aswell ?
>>
>> On 24/06/2008, at 8:08 AM, Tony Su wrote:
>>
>>
>> That's what makes *NIX harder sometimes…
>> It doesn't necessarily have built in interfaces to access specific  
>> functions.
>>
>> Nagios doesn't just monitor ports, it monitors just about anything  
>> imaginable… It can even monitor individual threads and processes if  
>> you know which ones to watch. It can also be set up to monitor  
>> environmental, not just a computer but even things like building  
>> HVAC.
>>
>> When you use something like SNMP, that only refers to the network  
>> transport… it's the capabilities of the client running on the  
>> machine and how deeply it can dig into/access information that  
>> determines what is possible.
>>
>> Tony
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On  
>> Behalf Of Daniel Rossi
>> Sent: Monday, June 23, 2008 7:07 AM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>> The snmp OS level things are too highly level to determine if there  
>> is a memory leak issue. And nagios is a port monitor isnt it ?  I'm  
>> currently investigating into the SNMP connector for JMX to extract  
>> interesting information for it to graph into something like cacti.
>>
>>
>> On 23/06/2008, at 4:24 PM, Tony Su wrote:
>>
>>
>>
>> If anyone doing dev really needs to determine what is going on when  
>> a problem like this occurs, I'm willing to deploy a special  
>> instance of Nagios to determine what is going on.
>>
>> Over the next few days, I'm also going to be building a  
>> distributable VM for GroundworkOpensource (which is a web frontend  
>> for Nagios). It's something I've been doing for the last 3 years,  
>> whenever SuSE releases a new OS I've re-built and updated a VMware  
>> Appliance anyone is welcome to use.
>>
>> I'll deploy one of those for restricted Red5 dev use(probably only  
>> for a day or few at a time), all you'd generally have to do is  
>> publish your server to the Internet and point my Nagios to your  
>> machine.
>>
>> I generally recommend Active Checks, which means that checks are  
>> configured on the Server. If you do Active Checks, with any luck  
>> what you need should already be installed on my server for whatever  
>> you want to investigate. If you implement a Passive Check, that  
>> would mean you'd have to install and configure a special client on  
>> your machine which isn't always easy to do. If you're developing on  
>> Windows, my distribution probably has the building blocks to  
>> monitor anything you'd want, but if you're developing on Linux only  
>> basic checks are included… I'm not as familiar monitoring services  
>> and processes specific to an application on Linux.
>>
>> So, yes for example my distribution already includes all the  
>> possibilities mentioned in this thread, but when running on a  
>> Windows box… Memory utilization, database locks, contention,  
>> connections, whether the /3GB switch might make a difference (re- 
>> partitioning the 32-bit 4gb memory map), and so on…
>>
>> Anyone who might want to take a stab at this over the next week  
>> might peruse the following things you can check on whatever OS…
>> http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Check_Plugins%2FOperating_Systems%2Findex.html;d=1
>>
>> Don't be misled about the fewer checks for Windows, many of them  
>> are written to APIs and interfaces like WMI, Perfmon (my  
>> preference) the Windows Event Logger and more which lets you dig  
>> deep.
>>
>> Tony
>>
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On  
>> Behalf Of Daniel Rossi
>> Sent: Friday, June 20, 2008 11:57 AM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>>
>> On 21/06/2008, at 4:32 AM, Mondain wrote:
>>
>>
>>
>>
>> If the memory doesnt grow towards your max, then I would assume  
>> your problem is not a memory leak. Maybe you have thread locking or  
>> database issues?
>>
>> No it grows if the service isnt restarted daily ;) Im not sure what  
>> was happening with that 'lockup' disconnection issue  but it  
>> resolved after a restart.
>>
>>
>>
>>
>>
>>
>>
>> Paul
>> On Fri, Jun 20, 2008 at 11:25 AM, Daniel Rossi  
>> <spam at electroteque.org> wrote:
>> Yes ive used all of these flags before but made performance worse  
>> on p4 we have xeon now so i will add them in, will it make a  
>> difference though ?  We have max set as -Xmx3500M, if the server  
>> gets rebooted daily it will stay in the 1GB range and never grow  
>> anywhere close to 3.5GB.
>>
>> On 21/06/2008, at 4:10 AM, Mondain wrote:
>>
>> I recently provided some JVM options for a group experiencing  
>> memory issues like you describe, here are the params (similar to  
>> what is in red5-hiperf.bat):
>>
>> -Xrs -Xms512M -Xmx1024M -Xss128K -XX:NewSize=256m - 
>> XX:SurvivorRatio=16 -XX:MinHeapFreeRatio=20 -XX:+AggressiveHeap -XX: 
>> +ExplicitGCInvokesConcurrent -XX:+UseConcMarkSweepGC - 
>> Dsun.rmi.dgc.client.gcInterval=990000 - 
>> Dsun.rmi.dgc.server.gcInterval=990000 - 
>> Djava.net.preferIPv4Stack=true -Xverify:none
>>
>> Set the Xms to your minimum starting memory and Xmx for the maximum  
>> you want to allot to the JVM. The other options can be researched  
>> here: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
>> I believe there may be problem internally with handling  
>> "unexpected" disconnection of clients (browser close, crash,  
>> whatever); this is hard to locate but I will look into it soon...
>> The ServiceUtils fix was found because they provided me with a log  
>> which allowed me to track down the problem.
>>
>> Paul
>> On Fri, Jun 20, 2008 at 10:22 AM, Daniel Rossi  
>> <spam at electroteque.org> wrote:
>> What can i do to track down an internal issue if it happens again.  
>> The playback of the vod stream was locking up this time this time  
>> after a few seconds of playback after buffering. I had to reboot  
>> the service for it to start playing. Im guessing the 'lockup' was  
>> probably a disconnection ?
>>
>> If i dont reboot the services daily in some circumstances the  
>> memory will go over the allocated memory and get outofmemory  
>> problems. Ie for 4GB of memory i set the max to 3.5GB and after a  
>> few days of uptime it will use it all up and start failing. I'm not  
>> so keen to stop the reboot schedule because i will expect calls in  
>> a few days of problems. How can i replicate all of these things  
>> locally.
>>
>>
>> On 21/06/2008, at 2:15 AM, Mondain wrote:
>>
>> I'm not aware of the memory leak you guys are referring to, but I  
>> did fix two potential leaks in ServiceUtils about a week or so ago..
>>
>> Paul
>> On Fri, Jun 20, 2008 at 7:04 AM, Daniel Rossi  
>> <spam at electroteque.org> wrote:
>>
>> On 20/06/2008, at 11:44 PM, Prabhu Tamilarasan (omNovia) wrote:
>>
>> > Sorry I don't have an answer to your question.  I actually just
>> > recently
>> > upgraded thinking it has been fixed based on Steven Gong's email
>> > about the
>> > RTMPMinaConnection memory leaks.  Is this a different one or  
>> related?
>> >
>> > For next time, try this in your /etc/init.d/red5 script to backup
>> > the logs
>> > for later review when you restart/start:
>> > start()
>> > {
>> >       mv /var/log/red5.log /var/log/red5_$(date +%Y-%m-%d-%H-%M- 
>> %S).log
>> >       ...
>> > }
>> I use the java service wrapper on that particular machine. Firedeamon
>> on the others. They are windows machines.
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -  
>> Voltaire _______________________________________________
>>
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -  
>> Voltaire _______________________________________________
>>
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> -- 
>> It is difficult to free fools from the chains they revere. -  
>> Voltaire _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
>
> -- 
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/ 
> ~steven/Home
>
> Javaflash Project Founder and Maintainer: http://osflash.org/javaflash
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20080624/093fc6a5/attachment-0001.html

More information about the Red5 mailing list