[Red5] Memory leak questions
Daniel Rossi
spam at electroteque.org
Tue Jun 24 01:14:19 PDT 2008
Looks like some run their own 'snmp' deamons to observe data. Still
can't find a way to discover the jvm information. We have cacti
handling all the servers from one location, and they are bsd servers
so not too keen to run java on them because of the hack port issues.
The vid servers are win2k3 however.
So it's not advisable then to expose the snmp layer of the jmx
management in java and use some tool to poll it ?
On 24/06/2008, at 5:44 PM, Steven Gong wrote:
> There are also other NMS tools like Zenoss and OpenNMS that might be
> helpful for system monitoring.
>
> On Tue, Jun 24, 2008 at 3:32 PM, Daniel Rossi
> <spam at electroteque.org> wrote:
> So nagio can observe the JVM data directly then ? Is there a plugin
> for this ?
>
> On 24/06/2008, at 5:21 PM, Tony Su wrote:
>
>> The client app running on the machine being monitored can be
>> written in any language and even run in a sandbox (like Java). The
>> only thing that's important is <how> the app is able to collect
>> data.
>>
>> So, although I'm not familiar with how any particular client
>> running on a *NIX machine might do this,
>>
>> On a Windows machine today almost everything must run in a
>> virtualized environment (32-bit or 64-bit virtual machines) called
>> Protected Mode. The only notable exception is video. Only older OS
>> like WinME, Win98/95 permit "real mode" runtimes which grant direct
>> access to the hardware.
>>
>> But, on a Windows box, Microsoft has enabled several API that come
>> pretty close to delivering data as close to real mode as possible,
>> and you write your applications to those specified interfaces with
>> their controlled options.
>>
>> Things weren't always this way. I remember comparing benchmark apps
>> in Win98, you'd always be getting different figures whether the app
>> ran in real or protected (virtualized) mode.
>>
>> Although I haven't studied Linux in as much detail, my general
>> impression is that although most apps run in Protected Mode in
>> Linux, the barriers to run in Real Mode aren't as great as they are
>> in Windows.
>>
>> Tony
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On
>> Behalf Of Daniel Rossi
>> Sent: Monday, June 23, 2008 8:33 PM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>> No java can export some JMX data via SNMP for monitoring and
>> graphing in supported software for JVM specific information which
>> is what is needed I'm guessing ? Does the OS display the JVM stuff
>> aswell ?
>>
>> On 24/06/2008, at 8:08 AM, Tony Su wrote:
>>
>>
>> That's what makes *NIX harder sometimes…
>> It doesn't necessarily have built in interfaces to access specific
>> functions.
>>
>> Nagios doesn't just monitor ports, it monitors just about anything
>> imaginable… It can even monitor individual threads and processes if
>> you know which ones to watch. It can also be set up to monitor
>> environmental, not just a computer but even things like building
>> HVAC.
>>
>> When you use something like SNMP, that only refers to the network
>> transport… it's the capabilities of the client running on the
>> machine and how deeply it can dig into/access information that
>> determines what is possible.
>>
>> Tony
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On
>> Behalf Of Daniel Rossi
>> Sent: Monday, June 23, 2008 7:07 AM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>> The snmp OS level things are too highly level to determine if there
>> is a memory leak issue. And nagios is a port monitor isnt it ? I'm
>> currently investigating into the SNMP connector for JMX to extract
>> interesting information for it to graph into something like cacti.
>>
>>
>> On 23/06/2008, at 4:24 PM, Tony Su wrote:
>>
>>
>>
>> If anyone doing dev really needs to determine what is going on when
>> a problem like this occurs, I'm willing to deploy a special
>> instance of Nagios to determine what is going on.
>>
>> Over the next few days, I'm also going to be building a
>> distributable VM for GroundworkOpensource (which is a web frontend
>> for Nagios). It's something I've been doing for the last 3 years,
>> whenever SuSE releases a new OS I've re-built and updated a VMware
>> Appliance anyone is welcome to use.
>>
>> I'll deploy one of those for restricted Red5 dev use(probably only
>> for a day or few at a time), all you'd generally have to do is
>> publish your server to the Internet and point my Nagios to your
>> machine.
>>
>> I generally recommend Active Checks, which means that checks are
>> configured on the Server. If you do Active Checks, with any luck
>> what you need should already be installed on my server for whatever
>> you want to investigate. If you implement a Passive Check, that
>> would mean you'd have to install and configure a special client on
>> your machine which isn't always easy to do. If you're developing on
>> Windows, my distribution probably has the building blocks to
>> monitor anything you'd want, but if you're developing on Linux only
>> basic checks are included… I'm not as familiar monitoring services
>> and processes specific to an application on Linux.
>>
>> So, yes for example my distribution already includes all the
>> possibilities mentioned in this thread, but when running on a
>> Windows box… Memory utilization, database locks, contention,
>> connections, whether the /3GB switch might make a difference (re-
>> partitioning the 32-bit 4gb memory map), and so on…
>>
>> Anyone who might want to take a stab at this over the next week
>> might peruse the following things you can check on whatever OS…
>> http://www.nagiosexchange.org/cgi-bin/page.cgi?g=Check_Plugins%2FOperating_Systems%2Findex.html;d=1
>>
>> Don't be misled about the fewer checks for Windows, many of them
>> are written to APIs and interfaces like WMI, Perfmon (my
>> preference) the Windows Event Logger and more which lets you dig
>> deep.
>>
>> Tony
>>
>>
>> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On
>> Behalf Of Daniel Rossi
>> Sent: Friday, June 20, 2008 11:57 AM
>> To: red5 at osflash.org
>> Subject: Re: [Red5] Memory leak questions
>>
>>
>> On 21/06/2008, at 4:32 AM, Mondain wrote:
>>
>>
>>
>>
>> If the memory doesnt grow towards your max, then I would assume
>> your problem is not a memory leak. Maybe you have thread locking or
>> database issues?
>>
>> No it grows if the service isnt restarted daily ;) Im not sure what
>> was happening with that 'lockup' disconnection issue but it
>> resolved after a restart.
>>
>>
>>
>>
>>
>>
>>
>> Paul
>> On Fri, Jun 20, 2008 at 11:25 AM, Daniel Rossi
>> <spam at electroteque.org> wrote:
>> Yes ive used all of these flags before but made performance worse
>> on p4 we have xeon now so i will add them in, will it make a
>> difference though ? We have max set as -Xmx3500M, if the server
>> gets rebooted daily it will stay in the 1GB range and never grow
>> anywhere close to 3.5GB.
>>
>> On 21/06/2008, at 4:10 AM, Mondain wrote:
>>
>> I recently provided some JVM options for a group experiencing
>> memory issues like you describe, here are the params (similar to
>> what is in red5-hiperf.bat):
>>
>> -Xrs -Xms512M -Xmx1024M -Xss128K -XX:NewSize=256m -
>> XX:SurvivorRatio=16 -XX:MinHeapFreeRatio=20 -XX:+AggressiveHeap -XX:
>> +ExplicitGCInvokesConcurrent -XX:+UseConcMarkSweepGC -
>> Dsun.rmi.dgc.client.gcInterval=990000 -
>> Dsun.rmi.dgc.server.gcInterval=990000 -
>> Djava.net.preferIPv4Stack=true -Xverify:none
>>
>> Set the Xms to your minimum starting memory and Xmx for the maximum
>> you want to allot to the JVM. The other options can be researched
>> here: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
>> I believe there may be problem internally with handling
>> "unexpected" disconnection of clients (browser close, crash,
>> whatever); this is hard to locate but I will look into it soon...
>> The ServiceUtils fix was found because they provided me with a log
>> which allowed me to track down the problem.
>>
>> Paul
>> On Fri, Jun 20, 2008 at 10:22 AM, Daniel Rossi
>> <spam at electroteque.org> wrote:
>> What can i do to track down an internal issue if it happens again.
>> The playback of the vod stream was locking up this time this time
>> after a few seconds of playback after buffering. I had to reboot
>> the service for it to start playing. Im guessing the 'lockup' was
>> probably a disconnection ?
>>
>> If i dont reboot the services daily in some circumstances the
>> memory will go over the allocated memory and get outofmemory
>> problems. Ie for 4GB of memory i set the max to 3.5GB and after a
>> few days of uptime it will use it all up and start failing. I'm not
>> so keen to stop the reboot schedule because i will expect calls in
>> a few days of problems. How can i replicate all of these things
>> locally.
>>
>>
>> On 21/06/2008, at 2:15 AM, Mondain wrote:
>>
>> I'm not aware of the memory leak you guys are referring to, but I
>> did fix two potential leaks in ServiceUtils about a week or so ago..
>>
>> Paul
>> On Fri, Jun 20, 2008 at 7:04 AM, Daniel Rossi
>> <spam at electroteque.org> wrote:
>>
>> On 20/06/2008, at 11:44 PM, Prabhu Tamilarasan (omNovia) wrote:
>>
>> > Sorry I don't have an answer to your question. I actually just
>> > recently
>> > upgraded thinking it has been fixed based on Steven Gong's email
>> > about the
>> > RTMPMinaConnection memory leaks. Is this a different one or
>> related?
>> >
>> > For next time, try this in your /etc/init.d/red5 script to backup
>> > the logs
>> > for later review when you restart/start:
>> > start()
>> > {
>> > mv /var/log/red5.log /var/log/red5_$(date +%Y-%m-%d-%H-%M-
>> %S).log
>> > ...
>> > }
>> I use the java service wrapper on that particular machine. Firedeamon
>> on the others. They are windows machines.
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -
>> Voltaire _______________________________________________
>>
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -
>> Voltaire _______________________________________________
>>
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>>
>> --
>> It is difficult to free fools from the chains they revere. -
>> Voltaire _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
>
> --
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>
> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
> ~steven/Home
>
> Javaflash Project Founder and Maintainer: http://osflash.org/javaflash
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20080624/2218143c/attachment-0001.html
More information about the Red5
mailing list