[Red5] [Red5 0.9.0 final] Terracotta support

Tue Feb 9 14:45:36 PST 2010

Hey this last message seems particularly cool! Do you know why
modifiying that has so much impact?

What is the first resource that saturates on your tests? CPU or bandwidth?

How are you testing concurrency of live streams?

2010/2/8, Dan Daemon <dan.daemon at gmail.com>:
> Hello guys...
>
> Good news here...
>
> We deleted this configuration from red5.sh file:
> ---
> -Dsun.rmi.dgc.client.gcInterval=990000
> -Dsun.rmi.dgc.server.gcInterval=990000
> ---
>
> And increased amount of concurrent connections from 300 to 800 with live
> high quality video stream and audio 22 Kilobits... Interesting...
>
> On Mon, Feb 8, 2010 at 5:29 PM, Dan Daemon <dan.daemon at gmail.com> wrote:
>
>> I tried to add some stuff in the code... but server went to unstable mode
>> :o)...
>> I tried to extend origin server to get read and written bytes.
>>
>>
>> On Mon, Feb 8, 2010 at 4:57 PM, Dan Daemon <dan.daemon at gmail.com> wrote:
>>
>>> Hello Dan,
>>>
>>> On Mon, Feb 8, 2010 at 1:11 PM, Dan Rossi <electroteque at gmail.com> wrote:
>>>
>>>>
>>>> On 08/02/2010, at 8:18 PM, Dan Daemon wrote:
>>>>
>>>> Hello Dan,
>>>>
>>>> Yep, I already use it last 6-8 months.
>>>>
>>>> I activated EhCache on all edges first, then I activated EhCache setup
>>>> on
>>>> all origins. It helps a little bit.
>>>>
>>>>
>>>> Is ehcache actually working I thought it was taken out. There was a bug
>>>> where it was caching on filenames regardless of paths, maybe that was
>>>> fixed
>>>> ?
>>>>
>>> At least it works for me... It helps a little bit for live streaming, at
>>> least it's more stable.
>>>
>>>
>>>>
>>>>
>>>> But no difference on origin server with using CPU or RAM. It's
>>>> absolutely
>>>> same with edges and without edges. That is not good :(
>>>>
>>>>
>>>> Yep thats what I mean, there is no load difference it has to be fixed I
>>>> guess.
>>>>
>>> Yes, I would like to help with this part too... but I dunno where to
>>> start. Now
>>> I'm investigating the RED5 architecture.
>>>
>>>
>>>>
>>>>
>>>> Plus we're trying to rewrite some parts of code for origin and edges to
>>>> get packets statistics, information about bytes sent/received.
>>>>
>>>>
>>>> Getting statistics for the mrtmp transfer stufff might be good.
>>>>
>>> yes. origin-edge architecture needs it... where to start?
>>>
>>>
>>>>
>>>>
>>>> I wanna help with testing and optimizing, but how :)
>>>>
>>>>
>>>> Please do.
>>>>
>>> I will :) Let me know if I can do something.
>>>
>>>
>>>>
>>>>
>>>> On Mon, Feb 8, 2010 at 10:27 AM, Dan Rossi
>>>> <electroteque at gmail.com>wrote:
>>>>
>>>>>
>>>>> On 08/02/2010, at 7:17 PM, Dan Daemon wrote:
>>>>>
>>>>> Hello Walter,
>>>>>
>>>>> Yes, currently I have 4 clusters:
>>>>> 1 - cluster #1: 1 origin + 3 edges
>>>>> 2 - cluster #2: 1 origin + 3 edges
>>>>> 3 - cluster #3: 1 origin + 3 edges
>>>>> 4 - cluster #4: 1 origin + 5 edges
>>>>>
>>>>> each cluster can handle about 1500 connections not much. The cluster #4
>>>>> with 5 edges can handle
>>>>> same 1500 connections like clusters #1, #2, #3. This is point ONE.
>>>>>
>>>>> Point No TWO: When I delete all edges and keep working only 4 origin
>>>>> servers and activate
>>>>> RTMP of them.... These all 4 servers can handle 1500 connection each...
>>>>>
>>>>> The questions:
>>>>> 1. Why 4 servers can handle 1500 connections without edges and with
>>>>> edges?
>>>>> 2. What sense to use edges if server can handle 1500 connections
>>>>> without
>>>>> edges?
>>>>> 3. What scalable advantages to use edges if origin server works
>>>>> absolutely same
>>>>>     with edges and without edges...
>>>>>
>>>>> :o)
>>>>>
>>>>> And common questions :))))
>>>>>
>>>>> How to extend my 4 clusters to handle much more connections... now it
>>>>> can handle 4x1500=6000max...
>>>>>
>>>>> Maybe somebody has some suggestions with cluster optimization?
>>>>>
>>>>>
>>>>> Farout thats a killer setup, so you are already using it. Any noticable
>>>>> difference in load and resources ? All the network load is placed on
>>>>> the
>>>>> origin and because there is no caching its also file reading. On my
>>>>> tests
>>>>> the origin had the same load as the edges :)
>>>>>
>>>>>
>>>>> On Mon, Feb 8, 2010 at 6:08 AM, Dan Rossi
>>>>> <electroteque at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>> On 08/02/2010, at 9:27 AM, Walter Tak wrote:
>>>>>>
>>>>>> Not sure if I understood that remark but 1500 publishing concurrent
>>>>>> connections to any server is actually pretty demanding for any server.
>>>>>>
>>>>>> The only option is to reduce the amount of incoming connections to one
>>>>>> server by adding servers. Creating more clusters (IIRC your setup
>>>>>> correct)
>>>>>> and using more nodes but smaller nodes so in the end each node only
>>>>>> has to
>>>>>> server 200-400 clients.
>>>>>>
>>>>>> If you're able to scaleout your application that way you can easily
>>>>>> have hundreds of Red5 VM's serving streams to subscribers.
>>>>>>
>>>>>> I'd like to try to keep things simple, perhaps that's not possible in
>>>>>> your situation, perhaps it is.
>>>>>>
>>>>>> E.g. lets assume you want to publish 1000 live streams. Each stream
>>>>>> has
>>>>>> a few subscribers, ranging from 1 just subscriber to say 20.
>>>>>>
>>>>>> Currently hardly any Red5 server can publish that many with just one
>>>>>> server so you're in nead of a cluster. However a single Origin server
>>>>>> still
>>>>>> cannot handle 1000 incoming streams. So you setup several individual
>>>>>> clusters as you already did (IIRC). You basically spread out 1000
>>>>>> livestreams over 4 Origin servers thus each server only has to deal
>>>>>> with 250
>>>>>> incoming streams. Each Origin server copies the streams to 3-4 Edge
>>>>>> servers
>>>>>> and each Edge server can easily handle say 200 subscribers each.
>>>>>>
>>>>>> That results in a nice setup of 4 x (1 + 3 ) = 12 servers. Either
>>>>>> physical or VMs (al be it very large VMs ofcourse).
>>>>>>
>>>>>> Now think of this possible scenario ; under no circumstances one live
>>>>>> stream will be watched by more than 200 subscribers. In that case you
>>>>>> don't
>>>>>> need a cluster with Edge/Origin setups but you'd only need a large
>>>>>> array of
>>>>>> single node Red5 servers ; say 12 pieces. Each server is independent
>>>>>> of it's
>>>>>> brothers and sisters in the array.
>>>>>>
>>>>>> One application-server (Java , PHP, Perl , Python, Dot.net, whatever)
>>>>>> routes incoming requests from webuser-users (who want/need to publish
>>>>>> their
>>>>>> stream) to a "free" Red5 server. After they started to publish
>>>>>> subscribers
>>>>>> show up and want to watch the stream ; the application-server
>>>>>> redirects
>>>>>> their request to the specific Red5 server et voila everything is
>>>>>> reasonable
>>>>>> fine.
>>>>>>
>>>>>> When the situation occurs that you'd need more capacity for more
>>>>>> incoming published streams, because your service is getting more
>>>>>> popular
>>>>>> then just add more servers to your array ; your application-server
>>>>>> should
>>>>>> keep a small list of servers and their load (by monitoring them ->
>>>>>> sending
>>>>>> requests each minute to learn the used amount of memory, CPU-usage,
>>>>>> bandwidth-usage etc).
>>>>>>
>>>>>>
>>>>>> Is there such a script available ? i was planning to add that
>>>>>> functionality into the clustering plugin for the flowplayer project i
>>>>>> did ;)
>>>>>>
>>>>>> http://flowplayer.org/plugins/streaming/cluster.html
>>>>>>
>>>>>>
>>>>>> That way your routing-server / application-server is the single
>>>>>> bottleneck but since it doesn't route streams, only requests, it will
>>>>>> probably be able to handle tens of thousands of streams without a
>>>>>> problem.
>>>>>> It's just traffic-cop.
>>>>>>
>>>>>> Perhaps you already have such a system in place ofcourse, this isn't
>>>>>> particulary rocketscience. Many large volume sites have the same
>>>>>> problem but
>>>>>> instead with video their problem is database / file / connection /
>>>>>> firewall-capacity. Any system that uses a single 'Origin'
>>>>>> hierachical-type
>>>>>> structure will in the end run out of capacity at the top.
>>>>>>
>>>>>> Regards,
>>>>>> Walter
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> *From:* Dan Daemon <dan.daemon at gmail.com>
>>>>>> *To:* red5 at osflash.org
>>>>>> *Sent:* Sunday, 07 February 2010 20:44
>>>>>> *Subject:* Re: [Red5] [Red5 0.9.0 final] Terracotta support
>>>>>>
>>>>>> How it possible to use VM on same computer if origin server dies after
>>>>>> 1500 concurrent connections
>>>>>> even in configuration from 4 computers (1 origin + 3 edges)
>>>>>>
>>>>>> On Sun, Feb 7, 2010 at 8:07 PM, Dan Rossi <electroteque at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yeah VM === Openvz , thats my setup, dual dual core Opteron  and 20GB
>>>>>>> of ram. I'll setup 4 instances then to act as a mock cluster setup :)
>>>>>>> My
>>>>>>> only problem is I have one IP.
>>>>>>>
>>>>>>> On 08/02/2010, at 3:24 AM, Walter Tak wrote:
>>>>>>>
>>>>>>> > Hey Dan,
>>>>>>> >
>>>>>>> > you can use VMs as well so you don't require say 6 physical
>>>>>>> > machines
>>>>>>> but just one decent dual/quad xeon with say 6-8 Gb memory to be able
>>>>>>> to run
>>>>>>> enough virtual machines with enough bandwidth to be able to emulate a
>>>>>>> network of machines as a proof of concept.
>>>>>>> >
>>>>>>> > Regards,
>>>>>>> > Walter
>>>>>>> >
>>>>>>> >
>>>>>>> > ----- Original Message ----- From: "Dan Rossi" <
>>>>>>> electroteque at gmail.com>
>>>>>>> > To: <red5 at osflash.org>
>>>>>>> > Sent: Sunday, 07 February 2010 06:16
>>>>>>> > Subject: Re: [Red5] [Red5 0.9.0 final] Terracotta support
>>>>>>> >
>>>>>>> >
>>>>>>> >> Stephen Gong was working on this ages ago in 0.8.*. I tested it
>>>>>>> >> out
>>>>>>> and it worked I think only with sharedobjects support though, but not
>>>>>>> sure
>>>>>>> how the integration is going yet I am assuming it will require more
>>>>>>> conversation with the terracotta peeps who pop their head up in the
>>>>>>> list now
>>>>>>> and then. I think for a clustering solution red5 would integrate well
>>>>>>> with
>>>>>>> terracotta. One thing I noticed with edge / origin the origin still
>>>>>>> needs to
>>>>>>> be one big fk off server because its taking the load of the edge
>>>>>>> machines
>>>>>>> still especially on the network so terracotta and file caching might
>>>>>>> help
>>>>>>> here I suppose. I was testing on dual core xeon's at the time. I
>>>>>>> think the
>>>>>>> killer here is still the metadata stuff which needs to be moved to a
>>>>>>> memory
>>>>>>> cache perhaps, its more noticable on P4 or duo core than xeon though
>>>>>>> ie 100%
>>>>>>> cpu usage compared to 25% usage for the same traffic but 25% for 100
>>>>>>> VOD
>>>>>>> streams on each frontend server is still quite high I reckon ;)
>>>>>>> >>
>>>>>>> >> Here is the diagram of the setup steve made
>>>>>>> >>
>>>>>>> >>
>>>>>>> http://trac.red5.org/wiki/Documentation/Clustering/EdgeOriginSolutiononTerracotta
>>>>>>> >>
>>>>>>> >> So thats One or two origin servers, one delegating server for
>>>>>>> terracotta and 3 or 4 edge machines behind a load balancer. Pretty
>>>>>>> expensive
>>>>>>> and beefy setup. I dont have access to such a setup anymore since a
>>>>>>> client
>>>>>>> moved from red5 to FMS when they moved into a new data centre. Im
>>>>>>> still keen
>>>>>>> on setting up some amazon instances for testing such a setup if its
>>>>>>> not too
>>>>>>> expensive I could even use my server running openvz for a dev testbed
>>>>>>> but I
>>>>>>> believe the terracotta peeps have a serious clustering setup for a
>>>>>>> real
>>>>>>> testbed :)
>>>>>>> >>
>>>>>>> >> On 07/02/2010, at 8:15 AM, david.engelmaier wrote:
>>>>>>> >>
>>>>>>> >>> Hi guys,
>>>>>>> >>>
>>>>>>> >>> First of all I would like to say big THANK YOU for the new 0.9
>>>>>>> >>> release, especially for fixing the invoke memory leak bug.
>>>>>>> >>> Somewhere I saw an announcement of Terracotta out of the box
>>>>>>> >>> clustering in 0.9, but can't find anything about it in the
>>>>>>> changelog.
>>>>>>> >>> All the posts regarding Terracotta+Red5 clustering are about a
>>>>>>> year
>>>>>>> >>> old, is there any news in the 0.9 release concerning Terracotta
>>>>>>> >>> clustering?
>>>>>>> >>>
>>>>>>> >>> Many thanks
>>>>>>> >>>
>>>>>>> >>> David Engelmaier
>>>>>>> >>>
>>>>>>> >>> _______________________________________________
>>>>>>> >>> Red5 mailing list
>>>>>>> >>> Red5 at osflash.org
>>>>>>> >>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> _______________________________________________
>>>>>>> >> Red5 mailing list
>>>>>>> >> Red5 at osflash.org
>>>>>>> >> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>> >
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > Red5 mailing list
>>>>>>> > Red5 at osflash.org
>>>>>>> > http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Red5 mailing list
>>>>> Red5 at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>
>>
>