[Red5] Java heap space OutOfMemoryError
Daniel Rossi
electroteque at gmail.com
Fri Oct 17 20:34:22 PDT 2008
Interesting, which java objects are you referring should be cleared on
client disconnect, anything assigned during the connection in the
webapp ?
Already tried to observe what happens when flooding the server with my
bench client, stayed in the 1-2GB range, but the outofmemory error
happens when it increases to 3.2GB after a few days which is the max
setting.
On 18/10/2008, at 2:06 PM, Walter Tak wrote:
> These problems aren't limited to Red5. On FMS2 we had servers use
> more and more memory , on a lineair basis , and eventually hitting
> the limit after 3-4 days of operation. We solved this by resetting
> the servers every 3-4 days before they would crash.
>
> However we also had 24/7 services and could not afford to hard reset
> the service.
>
> 1. we had a pool of 3 servers
> 2. new users were routed to one of the three available servers
> 3. if a server had an uptime of more than 2 days that server was
> removed from the pool for new users
> 4. after 12-24 hours there would be no more activity on the server
> with , at that point , near 3 days of uptime and around 80% memory
> usage which would crash anytime now
> 5. when the last user would have been disconnected, of after 4 days
> uptime, the server would be restarted automatically
> 6. the server was added to the pool of available servers again
>
> That's one possible solution.
>
> The other is to find the source of your problem.
>
> Without knowing how much users are connected (at any given moment or
> per hour/day/week) you're guessing ofcourse so first make sure
> you know more about what's happening.
>
> Then go test and see why your server is eating memory and not
> releasing it. Try to connect 100.000 times and disconnect. See if
> memory is released. If not ; why ; make sure you have no open
> references or have made 100.000 schedulers and not removed them. You
> have to clean up anything that might not be cleaned up automatically
> by the garbage collector.
>
> If you user rooms ; don't use the same room over and over again ;
> make random rooms, have a scheduler run once every hour and let it
> walk through all rooms and if a room is empty, remove it.
>
> Check your logs for exceptions ; if there are exceptions chances are
> big that some object was made and was never removed especially if
> the exception is inside the appDisconnect/roomDisconnect/roomLeave
> events/methods.
>
> Etc.
>
> We tried this for the FMS2 application as well and ended up setting
> near EVERY object to NULL when the user left the room or exitted a
> function (yeah that's back-to-the-stone-age type of programming, but
> it solved 80% of the crashes) ; you want want to explicit remove/
> clean/nullify objects in java if you're not using them anymore.
> Don't wait for the GC to come by, it might never come by or not in
> time to save you from a out-of-memory crash.
>
> Walter
>
>
> ----- Original Message -----
> From: Sebastien Bicais
> To: red5 at osflash.org
> Sent: Friday, 17 October 2008 16:28
> Subject: Re: [Red5] Java heap space OutOfMemoryError
>
> Ok, that’s more a workaround. I can’t afford to reboot the server
> everyday, I’ve got users connected all day throughout.
>
> Should we log this issue against JIRA then ? I saw some tickets with
> the similar error, but they all have been closed.
>
>
> Cheers,
> seb
>
> From: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] On
> Behalf Of Daniel Rossi
> Sent: 17 October 2008 13:53
> To: red5 at osflash.org
> Subject: Re: [Red5] Java heap space OutOfMemoryError
>
> I have the servers rebooting daily at 12am to try and solve these
> problems, maybe it's still happening. Try and reboot the service at
> 12am each day and see how it goes ?
>
> On 17/10/2008, at 10:39 PM, Sebastien Bicais wrote:
>
>
> Hi there,
>
> here is the error I'm getting:
>
>
> Exception in thread "pool-5-thread-584" java.lang.OutOfMemoryError:
> Java heap space
> Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError:
> Java heap space
> 2008-10-14 11:13:27,814 [Thread-1] INFO
> org.red5.server.jmx.JMXAgent - Shutting down JMX agent
> java.lang.ClassCastException:
> org.springframework.beans.factory.CannotLoadBeanClassException
> cannot be cast to
> org.springframework.web.context.ConfigurableWebApplicationContext
> at
> org
> .red5
> .server.war.WarLoaderServlet.contextDestroyed(WarLoaderServlet.java:
> 219)
> at
> org
> .apache
> .catalina.core.StandardContext.listenerStop(StandardContext.java:3882)
> at
> org.apache.catalina.core.StandardContext.stop(StandardContext.java:
> 4516)
> at
> org
> .apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:
> 924)
> at
> org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:
> 1189)
> at
> org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1160)
> at
> org
> .apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:
> 313)
> at
> org
> .apache
> .catalina
> .util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
> at
> org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1086)
> at
> org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
> at
> org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
> at
> org.apache.catalina.core.StandardService.stop(StandardService.java:
> 584)
> at
> org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
> at
> org.apache.catalina.startup.Catalina.stop(Catalina.java:628)
> at
> org.apache.catalina.startup.Catalina.start(Catalina.java:603)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>
>
> this is happening on a production server, after what seems to be a
> fairly heavy load (I don't know how many connections/stream I got
> though).
>
> Just before the error, I got this info:
> 2008-10-13 21:30:25,502 [pool-4-thread-4] INFO
> org.red5.server.jmx.JMXFactory - Object name:
> org
> .red5.server:type=RTMPMinaConnection,connectionType=persistent,host=www.mycompany.com
> ,port=1935,clientId=1985
>
>
> The clientID on the days I experienced the error seems quite high;
> I'm not sure it is meant to go that high.
>
> Our application is implemented pretty much like the oflaDemo. We are
> creating a connection for each stream and then closing them.
> Potentially, one user could
> send several streams (up to 3). But we are creating a connection for
> each of them. Is that alright ? Is the oflaDemo the best practice
> with opening/closing
> connections ?
>
> Thanks,
>
> seb
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> No virus found in this incoming message.
> Checked by AVG.
> Version: 7.5.549 / Virus Database: 270.8.1/1731 - Release Date:
> 17-10-08 19:01
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20081018/648e23f0/attachment-0001.html
More information about the Red5
mailing list