[Red5] RES: 2,000 connections

Thu Oct 22 04:49:04 PDT 2009

Could this just be a bunch of IO buffers somewhere building up once there
isn't enough bandwidth to handle what red5 is trying to send?

2009/10/22 Giusanny <giusanny at tiscali.it>

>  …continue
>
>
>
> After 200 conncetions, I can see in my jconsole output the same error for a
> lot of clients...
>
> ….
>
> Error calling isBroadcaster: org.red5.server:type=Client,id=235
>
> Error calling isBroadcaster:
> org.red5.server:type=RTMPMinaConnection,connectionType=persistent,host=myserveIP,port=1935,clientId=235
>
> …
>
>
>
> Could be that pending connections cause a lot of memory usage? (remember
> that I make all the client connection from the same lan with limited
> bandwith)
>
>
>
> Thanks
>
> Giuseppe
>
> *Da:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *Per
> conto di *Giusanny
> *Inviato:* giovedì 22 ottobre 2009 12.59
>
> *A:* red5 at osflash.org
> *Oggetto:* Re: [Red5] RES: 2,000 connections
>
>
>
> Ok New tests with jconsole
>
>
>
> Same configuration (hugepages 1,5GB and heap 1,45GB)
>
>
>
> Red5 activated and waiting for connections
>
> GC every 46MB and 15 minutes (cpu goes to 3% every GC)
>
> live threads 37-38
>
> classes 4335-4336
>
> cpu 0,1%
>
>
>
> Start LIVE publisher (FME3)  whit no clients connected to the stream
>
> GC every 10MB-250MB and 15 minutes (cpu goes to 3% every GC)
>
> live threads 39-38
>
> classes 4336-4349
>
> cpu 0,2%-0.5%
>
>
>
> Connect first client to the stream
>
> GC every 10MB-250MB and 15 minutes (cpu goes to 3% every GC)
>
> live threads 39-40
>
> classes 4349
>
> cpu 0.4%-0.5%
>
>
>
> 10 client LIVE connections (connected in almost the same time)
>
> GC every 50MB-250MB and 1 minute (cpu goes to 3-10% every GC)
>
> live threads 40-42
>
> classes 4651-4657
>
> cpu 3%-10%
>
>
>
> 50 client LIVE connections (connected in almost the same time)
>
> GC every 1GB-1,4GB and 1 minutes (cpu goes to 3-30% every GC)
>
> live threads 41-43
>
> classes 4650-4660
>
> cpu 3%-10%
>
>
>
> With only 50 clients connctions 1,4GB…strange! I don’t understand how I got
> 260 yesterday!
>
>
>
> Any idea?
>
>
>
> Could be that pending connections cause a lot of memory usage? (remember
> that I make all the client connection from the same lan with limited
> bandwith)
>
>
>
> Giuseppe
>
>
>
> *Da:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *Per
> conto di *Mondain
> *Inviato:* giovedì 22 ottobre 2009 4.42
> *A:* red5 at osflash.org
> *Oggetto:* Re: [Red5] RES: 2,000 connections
>
>
>
> I have not read this whole thread but I would like to state the following:
>
> I have simulated between 500 to 700 VOD users with all the code available
> in our repository (loadtester) on my local box so for one thing I'm sure I
> didn't max out my NIC card. I believe I used nothing more than the
> highperf.bat to startup even though I have 4 Gb of RAM on this Windows XP
> x64 (Dual Xeons) box. I have not attempted this in the 0.9 branch since I
> have had other priorities but I would assume that Mina 2 works better than
> Mina 1.x.
>
> And like Walter says, 5 mb per viewer seems really high..
>
>
>
> Paul
>
>
>
> On Wed, Oct 21, 2009 at 5:35 PM, Walter Tak <walter at waltertak.com> wrote:
>
> Good you're breaking personal records , how about trying to minimise the
> memory usage of the OS and giving Java near 1.7 Gb mem ?
>
> You're actually nearing the maximum throughput of your 100 mbit NIC with
> 262 x 256 = +/- 70 Mbit of videodata anyway.
>
>
>
> If the maximum amount of connections scale linear then you'd be able to
> gain 20% more connections if you'd use 1.7 Gb heap versus 1.4 Gb thus
> increasing the maximum concurrent streams from 262 to 318 and increasing the
> NIC throughput from 70 to 85 Mbit. You normally don't want to hit the 100
> mbit limit since that would give a lot of high latency for all connected
> clients.
>
>
>
> If you'd watch this from the other side then you'd be able to stream to
> 1500 users if you'd put 8 Gb of memory in that server and 8 Gb really isn't
> that expensive compared to the cost of 1500 users x 256 kbit = 384 Mbit
> continous streaming.
>
>
>
> ( I do think that 5 Mb of mem usage per VOD is a bit high, perhaps some
> people are able to make some memory-footprint-analysis for a single VOD
> connection to see where the excessive usage of memory is occuring in the
> sources ? )
>
>
>
> W.
>
>  ----- Original Message -----
>
> *From:* Giusanny <giusanny at tiscali.it>
>
> *To:* red5 at osflash.org
>
> *Sent:* Wednesday, 21 October 2009 19:16
>
> *Subject:* Re: [Red5] RES: 2,000 connections
>
>
>
> New tests
>
> Using always default red.properties
>
>
>
> Having a 2GB ram server I used 1,5 GB for Hugepages (
> http://www.peuss.de/node/67) and 1,4GB for heap space…
>
> Note that I’m using the same value for XMS and XMM parameters in java_opts
> as Walter suggested (thanks)
>
> JAVA_OPTS=-Dpython.home=lib -Dred5.root=/usr/local/red5_9
> -Dlogback.ContextSelector=org.red5.logging.LoggingContextSelector
> -Dcatalina.useNaming=true -Djava.security.debug=failure
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9088
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false -Xrs -Xms1450M -Xmx1450M
> -Xss128K -XX:NewSize=256m -XX:+UseLargePages -XX:SurvivorRatio=16
> -XX:MinHeapFreeRatio=20 -XX:+AggressiveHeap -XX:+ExplicitGCInvokesConcurrent
> -Dsun.rmi.dgc.client.gcInterval=990000
> -Dsun.rmi.dgc.server.gcInterval=990000 -Djava.net.preferIPv4Stack=true
> -Xverify:none
>
>
>
> 262 users LIVE at 256kbps after that the heap grows too much and the server
> slows down too much and OUT of Memory!
>
> But I’m happy yesterday were only 170 for an out of memory! :D
>
>
>
> Using Jconsole,what can I post here to help to better understand?
>
>
>
> Giuseppe
>
>
>
>
>
> *Da:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *Per
> conto di *Rafael Franco Carvalho
> *Inviato:* mercoledì 21 ottobre 2009 16.11
> *A:* red5 at osflash.org
> *Oggetto:* Re: [Red5] RES: 2,000 connections
>
>
>
> Hi,
>
>
>
> How is your red5.properties?
>
>
>
> I’m using default configurations:
>
>
>
> rtmp.event_threads_core=16
>
> rtmp.event_threads_max=64
>
> # event threads queue: -1 unbounded, 0 direct (no queue), n bounded queue
>
> rtmp.event_threads_queue=0
>
>
>
> Thanks,
>
> Rafael Franco
>
>
>  ------------------------------
>
> *From:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *On
> Behalf Of *Giusanny
> *Sent:* Wednesday, October 21, 2009 9:21 AM
> *To:* red5 at osflash.org
> *Subject:* Re: [Red5] RES: 2,000 connections
>
>
>
> Thanks Dominick,
>
> I used http://www.peuss.de/node/67 very useful, so I changed the
> red5-highperf.sh …
>
>
>
> export JAVA_OPTS="-Xrs -Xms512M -Xmx1024M -Xss128K -XX:NewSize=256m -server
> -XX:+UseLargePages -XX:SurvivorRatio=16 -XX:MinHeapFreeRatio=20
> -XX:+AggressiveHeap -XX:+ExplicitGCInvokesConcurrent
> -Dsun.rmi.dgc.client.gcInterval=990000
> -Dsun.rmi.dgc.server.gcInterval=990000 -Djava.net.preferIPv4Stack=true
> -Xverify:none"
>
>
>
> After 200 streams LIVE (with FME3) at 256Kbps
>
> I got these error
>
>
>
> Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: GC
> overhead limit exceeded
>
>         at java.util.HashMap.newKeyIterator(HashMap.java:840)
>
>         at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>
>         at java.util.HashSet.iterator(HashSet.java:153)
>
>         at
> sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>         at
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:288)
>
>         at
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:402)
>
>         at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
>         at java.lang.Thread.run(Thread.java:619)
>
> Exception in thread "Red5_Scheduler_QuartzSchedulerThread" [ERROR]
> [NioProcessor-1] org.red5.server.messaging.InMemoryPushPushPipe - exception
> when pushing message to consumer
>
>
>
>
>
> Any idea?
>
> I could disable the feature by adding the option -XX:-UseGCOverheadLimit to
> the command line but the server still slows down!
>
> The final resolution could be to add more RAM?
>
>
>
> Could be caused by the client connection bandwith limit? All the 200
> clients came from the same lan and they have not too much bandwith (all
> togher 6Mbps), so it could be that the clients don’t have too much bandwith
> and cause an increase of pending messages from the server so an Out of
> Memory?
>
>
>
> Please answer me about this..I’m getting crazy!
>
>
>
> Thanks in advance
>
> Giuseppe
>
>
>
> *Da:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *Per
> conto di *Dominick Accattato
> *Inviato:* martedì 20 ottobre 2009 20.03
> *A:* red5 at osflash.org
> *Oggetto:* Re: [Red5] RES: 2,000 connections
>
>
>
> It's great that everyone wants to get some attention here.  Can someone
> spearhead this issue by documenting up on trac?  There currently isn't that
> much up on trac for performance:
>
> http://trac.red5.org/search?q=performance
>
> On Tue, Oct 20, 2009 at 12:51 PM, Giusanny <giusanny at tiscali.it> wrote:
>
> Doing my test I saw that when the pending messages number grows up, the ram
> usage grows up till the max so the server slows down
>
>
>
> Could be that, cause I make 150 connection from the same lan (with limited
> bandwith), the server can’t send all the messages to all the connections and
> cause growing up of connections and more ram usage?
>
>
>
>
>
> *Da:* red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] *Per
> conto di *Dan Daemon
> *Inviato:* martedì 20 ottobre 2009 18.20
> *A:* red5 at osflash.org
> *Oggetto:* Re: [Red5] RES: 2,000 connections
>
>
>
> Try to optimize your Application.java file.
>
>
>
> I have 3 clusters with 5 edges in each + 1 master.
>
> And I can connect up to 1650 concurrent connection per one cluster
>
> after I did some optimizations in my Application.java class.
>
>
>
> Anyway, if you wish to transfer live streams I'm not sure that
>
> red5 can handle more.
>
>
>
> At least we still did not get any results like guys told that
>
> 5 edges + 1 origin can handle 20000 concurrent connections.
>
>
>
> Seems it's possible only for FLV streaming but not for live
>
> streams.
>
>
>
> I need help too, but still did not get any solution...
>
>
>
> On Tue, Oct 20, 2009 at 4:23 PM, Giusanny <giusanny at tiscali.it> wrote:
>
> I have the same problem testing it with about 170 true connections at
> 256kbps.
>
> After these connection the server starts to disconnect all old and new
> connections writing in the log the same thing you have posted in your mail.
>
> I have a 2GM Ram Server so I use
>
> export JAVA_OPTS="-Xrs -Xms768M -Xmx1536M -Xss256K -XX:NewSize=512m
> -XX:SurvivorRatio=16 -XX:MinHeapFreeRatio=20 -XX:+AggressiveHeap
> -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=990000
> -Dsun.rmi.dgc.server.gcInterval=990000 -Djava.net.preferIPv4Stack=true
> -Xverify:none"
>
> then I run "re5.sh" so almost 80% of RAM (1536MB) is used with 170
> connections, means that the JVM use all the RAM that I set in the parameter
> -Xmx1536M
>
> My Idea? We need more RAM or we need to set some parameters somewhere!
> Please help us :D
>
> Giuseppe
>
>
> -----Messaggio originale-----
> Da: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Per conto
> di
> Rafael Franco Carvalho
>
> Inviato: martedì 20 ottobre 2009 13.52
> A: red5 at osflash.org
> Oggetto: [Red5] RES: 2,000 connections
>
>
> Hello,
>
> Thank you for your attention.
>
> After 500 connections Red5 show these lines:
> 2009-10-20 08:54:51,838 [Red5_Scheduler_Worker-5] WARN
> o.r.server.net.rtmp.RTMPConnection - Closing RTMPMinaConnection from
> 200.221.10.69 : 49787 to null (in: 0 out 0 ), with id 782357 due to long
> handshake 2009-10-20 08:55:37,483 [Red5_Scheduler_Worker-11] WARN
> o.r.server.net.rtmp.RTMPConnection - Closing RTMPMinaConnection from
> 200.221.10.69 : 49709 to 200.221.10.69:443 (in: 3709 out 3393 ), with id
> 2377175, due to too much inactivity (60002ms), last ping sent 10001ms ago
>
> Do you have any idea?
>
> Thanks,
>
> Rafael Franco
>
> -----Mensagem original-----
> De: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Em nome de
> Giusanny Enviada em: terça-feira, 20 de outubro de 2009 06:28
> Para: red5 at osflash.org
> Assunto: Re: [Red5] 2,000 connections
>
> Hi,
> after 500 connections what appends?
>
>
> -----Messaggio originale-----
> Da: red5-bounces at osflash.org [mailto:red5-bounces at osflash.org] Per conto
> di
> Rafael Franco Carvalho
> Inviato: lunedì 19 ottobre 2009 22.31
> A: red5 at osflash.org
> Oggetto: [Red5] 2,000 connections
>
> Hello,
>
> I'm trying to connect plus 2,000 users to Red5 using BenchmarkProfiler, but
> I couldn't reach this number of connections. The maximum clients that I
> have
> connected were 500. I'm using standard Red5 configurations.
>
> There is a special configuration for this? Where can find information about
> red.properties configurations? Increasing rtmp.event_threads_core and
> rtmp.event_threads_max should help?
>
>
> Please help me.
> Thanks in advance.
>
> Rafael Franco
>
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>   ------------------------------
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
>
> --
> http://gregoire.org/
> http://code.google.com/p/red5/
> http://code.google.com/p/blue5/
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://osflash.org/pipermail/red5_osflash.org/attachments/20091022/a167ee27/attachment-0001.html>