[Red5] R: R: R: OUT of memory!

Mon Oct 19 11:36:24 PDT 2009

Are you trying to access a local server or a remote one ? Since 170 x 256 = 
more than 40 mbit of traffic.

Are those 170 connections real world users or simulated ones on local lan 
machines accessing a local lan based server ?

If so ; is your lan limited to 100 mbit or 1 gbit ? And is your server 
connected to the lan at 100 or 1 gbit ?

If you try to access a remote server with 170 local simulated clients then 
your tests go bananas due to the lack of bandwidth between your lan and the 
remote server, probably.

Furthermore oflaDemo isn't the best example in the world ; I'd try to create 
a very simple custom application with logging information in the major 
events like appStart, appConnect, roomStart roomConnect appDisconnect etc so 
you can monitor using the logfiles what the clients are doing.

W.

----- Original Message ----- 
From: "Giusanny" <giusanny at tiscali.it>
To: <red5 at osflash.org>
Sent: Monday, 19 October 2009 19:02
Subject: [Red5] R: R: R: OUT of memory!

Mmmh I use oflaDemo as application to stream live...anything strange in the
appConnect method! ..and I didn't change it.

New test:
- 170 clients at 256kbps
- I see only a lot of handshake problem (but probably, caused by my
provider..because all the connections come from my computers in the same lan
:D)
- I see a lot of pending messages in the oflaDemo application (using admin
application to monitor)
- after 170 clients, the server stopped working and all the connection were
refused
- no more Out of Memory

Any ideas?

Thanks
Giuseppe

-----Messaggio originale-----
Da: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Per conto di
Andy Shaules
Inviato: lunedì 19 ottobre 2009 18.37
A: red5 at osflash.org
Oggetto: Re: [Red5] R: R: OUT of memory!

NO, its not the que, its probably one or two things.

Your webapp is single threaded application adapter, AND / OR you have
problem with your application code, either in its length of time to execute
being too long, or you have bad logic to handle the appConnect method.

----- Original Message -----
From: "Giusanny" <giusanny at tiscali.it>
To: <red5 at osflash.org>
Sent: Monday, October 19, 2009 8:51 AM
Subject: [Red5] R: R: OUT of memory!

Ok, reading the red5-highperf.sh I tryed with...

export JAVA_OPTS "-Xrs -Xms512M -Xmx1024M -Xss128K -XX:NewSize=256m"
and then execute red5.sh

I have no more out of memory, but if about 10 clients connect at almost the
same time the server refuse all the connection and stops working with
nothing in the error.log! (than I have to restart)

Could be a problem with the queue?
I have rtmp.event_threads_queue=0 in the re5.properties
What do you think?

Thanks
Giuseppe

-----Messaggio originale-----
Da: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Per conto di
Hugo Flambó
Inviato: lunedì 19 ottobre 2009 15.57
A: red5 at osflash.org
Oggetto: Re: [Red5] R: OUT of memory!

just set env var ...

export JAVA_OPTS=-Xms256M -Xmx512M -XX:MaxPermSize=128m

and it should work!!!

BR

On Mon, Oct 19, 2009 at 2:03 PM, Giusanny <giusanny at tiscali.it> wrote:
> Ok thanks, but how works red5-highperf.sh?
> It just set the JVM garbage collector?
>
> So if I want to use it I have just to modify red5-highperf.sh like
> "-Xms256M -Xmx512M -XX:MaxPermSize=128m ..."
> and then run red5-highperf.sh and than red5.sh?
> It's right?
>
> P.S. I see that if I start a live stream and then let 10 client
> connect in almost the same time, the server crashes (well doesn't
> crash but doesn't allow any other connection even http connections)!
> But if I let the client connect one per time it works for 150 and more
> client...could be a problem with the queue?
> Now I have rtmp.event_threads_queue=0 in the re5.properties.
>
> Any ideas?
>
> Thanks a lot!
> Giuseppe
>
> -----Messaggio originale-----
> Da: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Per
> conto di Hugo Flambó
> Inviato: lunedì 19 ottobre 2009 14.24
> A: red5 at osflash.org
> Oggetto: Re: [Red5] OUT of memory!
>
> don't know if solves your prob but maybe you can set JVM garbage
> collector options like:
>
> -Xms256M -Xmx512M -XX:MaxPermSize=128m
>
> On Mon, Oct 19, 2009 at 12:10 PM, Walter Tak <walter at waltertak.com> wrote:
>> There have been excellent threads about this subject a few weeks and
>> months ago. Most important thing ; get jconsole working and start
>> monitoring your application in real time so you can actually see it's
>> memory usage on minute-by-minute basis and then adjust the
>> commandline
> options accordingly.
>> Perhaps your java-app is only using 1 Gb of memory and won't ever
>> touch the remaining 2.
>>
>> Unfortunately Java , by default , used very fixed amounts of memory
>> (max heap space) so you need to tweak that manually. It's not really
>> a
>> Red5 problem. A problem of Red5 IS that it's using quite some
>> resources per VOD-connection. After a while that will be improved but
>> so far your best bet is to finetune it manually and make sure you
>> don't do odd (memory-related) things if you have a custom webapplication.
>>
>> You can check the archives at
>> http://osflash.org/pipermail/red5_osflash.org/ and use Google for
>> easiers searching like:
>>
>> "site:osflash.org/pipermail/red5_osflash.org/ heap space" ; this will
>> search the archives in a convenient way for keywords "heap space".
>>
>> Walter
>>
>> ----- Original Message -----
>> From: Giusanny
>> To: red5 at osflash.org
>> Sent: Monday, 19 October 2009 11:54
>> Subject: [Red5] OUT of memory!
>>
>> Hi,
>>
>> yesterday I was streaming live with 150 connections at 256kbps
>> everything ok till a problem that blocked my server (after only 30
> minutes)!
>>
>> I didn't have time to get the cause and I rebooted.I continued for
>> 1,5 hour with other 150 connections without problems.
>>
>>
>>
>> Today I was trying to recreate the problem and I was a JAVA OUT OF
> MEMORY!!!
>> (I'm scared about it :D being a java developer!)
>>
>> I think I could recreate the problem doing about 30 connection almost
>> in the same time.
>>
>>
>>
>> I'm not using red5-highperf.sh but just red5.sh? Shoud I use highperf?
>> I have a 2GB ram dedicated server how could be the beast way to use
>> the
> ram?
>> And how can I use (the command line options) the red5-highperf.sh?
>>
>>
>>
>> Thanks a lot
>>
>> Giuseppe
>>
>>
>>
>> P.s. I got the logs.
>>
>> 2009-10-19 11:13:25,612 [Red5_Scheduler_Worker-4] ERROR
>> org.quartz.core.JobRunShell - Job DEFAULT.ScheduledJob_2386 threw an
>> unhandled Exception:
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>> at java.util.Arrays.copyOf(Arrays.java:2882)
>> [na:1.6.0_16]
>>
>> at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.
>> j
>> ava:100)
>> [na:1.6.0_16]
>>
>> at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390
>> )
>> [na:1.6.0_16]
>>
>> at
>> java.lang.StringBuilder.append(StringBuilder.java:119)
>> [na:1.6.0_16]
>>
>> at java.lang.Object.toString(Object.java:219)
>> [na:1.6.0_16]
>>
>> at
>> org.red5.server.scheduling.QuartzSchedulingServiceJob.execute(QuartzS
>> c
>> hedulingServiceJob.java:61)
>> [red5.jar:na]
>>
>> at
>> org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>> [quartz-1.6.5.jar:1.6.5]
>>
>> at
>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
>> a
>> va:525)
>> [quartz-1.6.5.jar:1.6.5]
>>
>> 2009-10-19 11:13:26,806 [NioProcessor-1] ERROR
>> o.r.s.m.InMemoryPushPushPipe
>> - exception when pushing message to consumer
>>
>> java.lang.OutOfMemoryError: Java heap space
>>
>>
>>
>>
>>
>> ________________________________
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>

_______________________________________________
Red5 mailing list
Red5 at osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5 at osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5 at osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org

_______________________________________________
Red5 mailing list
Red5 at osflash.org
http://osflash.org/mailman/listinfo/red5_osflash.org