[Red5] [Red5 0.9.0 final] Terracotta support

Tue Feb 9 10:13:12 PST 2010

Hello Walter,

No unfortunately it's same... max 400 concurrent connections! It seems RED5
limit...
I read some topics before... we need to avoid it...

On Tue, Feb 9, 2010 at 7:54 PM, Tyler Kocheran <rfkrocktk at gmail.com> wrote:

> I'm glad we're talking about this kind of stuff again :)
>
> My proposed "solution" is that we port all persistence operations (such as
> IClient.setAttribute(), ISharedObject.setAttribute(),
> IConnection.setAttribute() etc.) to use EhCache in the backend. By doing so,
> we'll almost immediately be able to cluster that information across N
> servers using EhCache's distributed cache solution. Basically, you run a
> cache cluster and an edge cluster which actually handles connections. All
> data is kept in sync via EhCache so all servers are basically working off of
> the same information seamlessly. And we can also lower memory overhead by
> overflowing to disk on single-server setups if need be. This ultimately
> leaves cache setup to the developer, giving us a lot of flexibility.
>
> That would take care of most of the "information" issues, like clustering
> SharedObjects and connection information, but it wouldn't address streaming,
> which is the reason most people use Red5.
>
> Why don't we just create an edge-origin setup where the edge servers simply
> subscribe to the stream being broadcasted through the origin server? That
> would seem to be a good option; the origin server(s) would only broadcast as
> many streams as there are edge servers asking for them. Or is this the way
> it already is working?
>
> How is Wowza using EC2? Is there any way we could imitate that sort of
> setup to get Red5 into the cloud?
>
>
> On Tue, Feb 9, 2010 at 9:37 AM, Walter Tak <walter at waltertak.com> wrote:
>
>>  Hey Dan, glad your results are improving , after a few vague messages on
>> the list ;)
>>
>> Dan (Rossi) wondered if there is some active script around which could be
>> run on an applicationserver ; alas I was just referring to a hypothetical
>> script , which isn't too hard to develop, a few days should do it.
>>
>> It's nothing more than a cron-scheduled script (e.g. PHP to make things
>> easy but might also be a java-daemon started once on the application-server)
>> .
>>
>> While it's running it should ping or test tcp-ports of all edges (or
>> normally individual nodes if you don't use an Edge/Origin setup) on a
>> regular basis. The script would have a list in a little database, or perhaps
>> in memory , of all physical nodes, and their states e.g. running or down, a
>> timestamp when the last 'ping' was sent and a number indicating the load on
>> the queried-server. (Best is to call a script on each Red5 node which would
>> return the cpu % and memory usage for accurate information).
>>
>> When a new user would request a video (VOD) then his player (or the page
>> in which the player is embedded) should first connect to the application
>> server (AS), the AS would check his list of available servers (e.g. 7 of 10
>> are up and running) and then return the ip/hostname of the least busy
>> server.
>>
>> Regards,
>> Walter
>>
>>
>> ----- Original Message -----
>> *From:* Dan Daemon <dan.daemon at gmail.com>
>> *To:* red5 at osflash.org
>> *Sent:* Monday, 08 February 2010 23:44
>> *Subject:* Re: [Red5] [Red5 0.9.0 final] Terracotta support
>>
>> Tested call was about 2 hours... without any problems... Interesting!
>> Our surrogate performance tests did not show anything before...
>>
>> On Tue, Feb 9, 2010 at 12:27 AM, Dan Daemon <dan.daemon at gmail.com> wrote:
>>
>>> Hello guys...
>>>
>>> Good news here...
>>>
>>> We deleted this configuration from red5.sh file:
>>> ---
>>> -Dsun.rmi.dgc.client.gcInterval=990000
>>> -Dsun.rmi.dgc.server.gcInterval=990000
>>> ---
>>>
>>> And increased amount of concurrent connections from 300 to 800 with live
>>> high quality video stream and audio 22 Kilobits... Interesting...
>>>
>>> On Mon, Feb 8, 2010 at 5:29 PM, Dan Daemon <dan.daemon at gmail.com> wrote:
>>>
>>>> I tried to add some stuff in the code... but server went to unstable
>>>> mode :o)...
>>>> I tried to extend origin server to get read and written bytes.
>>>>
>>>>
>>>> On Mon, Feb 8, 2010 at 4:57 PM, Dan Daemon <dan.daemon at gmail.com>wrote:
>>>>
>>>>> Hello Dan,
>>>>>
>>>>>  On Mon, Feb 8, 2010 at 1:11 PM, Dan Rossi <electroteque at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>>  On 08/02/2010, at 8:18 PM, Dan Daemon wrote:
>>>>>>
>>>>>> Hello Dan,
>>>>>>
>>>>>> Yep, I already use it last 6-8 months.
>>>>>>
>>>>>> I activated EhCache on all edges first, then I activated EhCache setup
>>>>>> on
>>>>>> all origins. It helps a little bit.
>>>>>>
>>>>>>
>>>>>> Is ehcache actually working I thought it was taken out. There was a
>>>>>> bug where it was caching on filenames regardless of paths, maybe that was
>>>>>> fixed ?
>>>>>>
>>>>> At least it works for me... It helps a little bit for live streaming,
>>>>> at least it's more stable.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> But no difference on origin server with using CPU or RAM. It's
>>>>>> absolutely
>>>>>> same with edges and without edges. That is not good :(
>>>>>>
>>>>>>
>>>>>> Yep thats what I mean, there is no load difference it has to be fixed
>>>>>> I guess.
>>>>>>
>>>>> Yes, I would like to help with this part too... but I dunno where to
>>>>> start. Now
>>>>> I'm investigating the RED5 architecture.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Plus we're trying to rewrite some parts of code for origin and edges
>>>>>> to
>>>>>> get packets statistics, information about bytes sent/received.
>>>>>>
>>>>>>
>>>>>> Getting statistics for the mrtmp transfer stufff might be good.
>>>>>>
>>>>> yes. origin-edge architecture needs it... where to start?
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> I wanna help with testing and optimizing, but how :)
>>>>>>
>>>>>>
>>>>>> Please do.
>>>>>>
>>>>> I will :) Let me know if I can do something.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 8, 2010 at 10:27 AM, Dan Rossi <electroteque at gmail.com>wrote:
>>>>>>
>>>>>>>
>>>>>>>  On 08/02/2010, at 7:17 PM, Dan Daemon wrote:
>>>>>>>
>>>>>>> Hello Walter,
>>>>>>>
>>>>>>> Yes, currently I have 4 clusters:
>>>>>>> 1 - cluster #1: 1 origin + 3 edges
>>>>>>> 2 - cluster #2: 1 origin + 3 edges
>>>>>>> 3 - cluster #3: 1 origin + 3 edges
>>>>>>> 4 - cluster #4: 1 origin + 5 edges
>>>>>>>
>>>>>>> each cluster can handle about 1500 connections not much. The cluster
>>>>>>> #4 with 5 edges can handle
>>>>>>> same 1500 connections like clusters #1, #2, #3. This is point ONE.
>>>>>>>
>>>>>>> Point No TWO: When I delete all edges and keep working only 4 origin
>>>>>>> servers and activate
>>>>>>> RTMP of them.... These all 4 servers can handle 1500 connection
>>>>>>> each...
>>>>>>>
>>>>>>> The questions:
>>>>>>> 1. Why 4 servers can handle 1500 connections without edges and with
>>>>>>> edges?
>>>>>>> 2. What sense to use edges if server can handle 1500 connections
>>>>>>> without edges?
>>>>>>> 3. What scalable advantages to use edges if origin server works
>>>>>>> absolutely same
>>>>>>>     with edges and without edges...
>>>>>>>
>>>>>>> :o)
>>>>>>>
>>>>>>> And common questions :))))
>>>>>>>
>>>>>>> How to extend my 4 clusters to handle much more connections... now it
>>>>>>> can handle 4x1500=6000max...
>>>>>>>
>>>>>>> Maybe somebody has some suggestions with cluster optimization?
>>>>>>>
>>>>>>>
>>>>>>> Farout thats a killer setup, so you are already using it. Any
>>>>>>> noticable difference in load and resources ? All the network load is placed
>>>>>>> on the origin and because there is no caching its also file reading. On my
>>>>>>> tests the origin had the same load as the edges :)
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 8, 2010 at 6:08 AM, Dan Rossi <electroteque at gmail.com>wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>  On 08/02/2010, at 9:27 AM, Walter Tak wrote:
>>>>>>>>
>>>>>>>>   Not sure if I understood that remark but 1500 publishing
>>>>>>>> concurrent connections to any server is actually pretty demanding for any
>>>>>>>> server.
>>>>>>>>
>>>>>>>> The only option is to reduce the amount of incoming connections to
>>>>>>>> one server by adding servers. Creating more clusters (IIRC your setup
>>>>>>>> correct) and using more nodes but smaller nodes so in the end each node only
>>>>>>>> has to server 200-400 clients.
>>>>>>>>
>>>>>>>> If you're able to scaleout your application that way you can easily
>>>>>>>> have hundreds of Red5 VM's serving streams to subscribers.
>>>>>>>>
>>>>>>>> I'd like to try to keep things simple, perhaps that's not possible
>>>>>>>> in your situation, perhaps it is.
>>>>>>>>
>>>>>>>> E.g. lets assume you want to publish 1000 live streams. Each stream
>>>>>>>> has a few subscribers, ranging from 1 just subscriber to say 20.
>>>>>>>>
>>>>>>>> Currently hardly any Red5 server can publish that many with just one
>>>>>>>> server so you're in nead of a cluster. However a single Origin server still
>>>>>>>> cannot handle 1000 incoming streams. So you setup several individual
>>>>>>>> clusters as you already did (IIRC). You basically spread out 1000
>>>>>>>> livestreams over 4 Origin servers thus each server only has to deal with 250
>>>>>>>> incoming streams. Each Origin server copies the streams to 3-4 Edge servers
>>>>>>>> and each Edge server can easily handle say 200 subscribers each.
>>>>>>>>
>>>>>>>> That results in a nice setup of 4 x (1 + 3 ) = 12 servers. Either
>>>>>>>> physical or VMs (al be it very large VMs ofcourse).
>>>>>>>>
>>>>>>>> Now think of this possible scenario ; under no circumstances one
>>>>>>>> live stream will be watched by more than 200 subscribers. In that case you
>>>>>>>> don't need a cluster with Edge/Origin setups but you'd only need a large
>>>>>>>> array of single node Red5 servers ; say 12 pieces. Each server is
>>>>>>>> independent of it's brothers and sisters in the array.
>>>>>>>>
>>>>>>>> One application-server (Java , PHP, Perl , Python, Dot.net,
>>>>>>>> whatever) routes incoming requests from webuser-users (who want/need to
>>>>>>>> publish their stream) to a "free" Red5 server. After they started to publish
>>>>>>>> subscribers show up and want to watch the stream ; the application-server
>>>>>>>> redirects their request to the specific Red5 server et voila everything is
>>>>>>>> reasonable fine.
>>>>>>>>
>>>>>>>> When the situation occurs that you'd need more capacity for more
>>>>>>>> incoming published streams, because your service is getting more popular
>>>>>>>> then just add more servers to your array ; your application-server should
>>>>>>>> keep a small list of servers and their load (by monitoring them -> sending
>>>>>>>> requests each minute to learn the used amount of memory, CPU-usage,
>>>>>>>> bandwidth-usage etc).
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there such a script available ? i was planning to add that
>>>>>>>> functionality into the clustering plugin for the flowplayer project i did ;)
>>>>>>>>
>>>>>>>> http://flowplayer.org/plugins/streaming/cluster.html
>>>>>>>>
>>>>>>>>
>>>>>>>> That way your routing-server / application-server is the single
>>>>>>>> bottleneck but since it doesn't route streams, only requests, it will
>>>>>>>> probably be able to handle tens of thousands of streams without a problem.
>>>>>>>> It's just traffic-cop.
>>>>>>>>
>>>>>>>> Perhaps you already have such a system in place ofcourse, this isn't
>>>>>>>> particulary rocketscience. Many large volume sites have the same problem but
>>>>>>>> instead with video their problem is database / file / connection /
>>>>>>>> firewall-capacity. Any system that uses a single 'Origin' hierachical-type
>>>>>>>> structure will in the end run out of capacity at the top.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Walter
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> *From:* Dan Daemon <dan.daemon at gmail.com>
>>>>>>>> *To:* red5 at osflash.org
>>>>>>>> *Sent:* Sunday, 07 February 2010 20:44
>>>>>>>> *Subject:* Re: [Red5] [Red5 0.9.0 final] Terracotta support
>>>>>>>>
>>>>>>>> How it possible to use VM on same computer if origin server dies
>>>>>>>> after 1500 concurrent connections
>>>>>>>> even in configuration from 4 computers (1 origin + 3 edges)
>>>>>>>>
>>>>>>>> On Sun, Feb 7, 2010 at 8:07 PM, Dan Rossi <electroteque at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yeah VM === Openvz , thats my setup, dual dual core Opteron  and
>>>>>>>>> 20GB of ram. I'll setup 4 instances then to act as a mock cluster setup :)
>>>>>>>>> My only problem is I have one IP.
>>>>>>>>>
>>>>>>>>> On 08/02/2010, at 3:24 AM, Walter Tak wrote:
>>>>>>>>>
>>>>>>>>> > Hey Dan,
>>>>>>>>> >
>>>>>>>>> > you can use VMs as well so you don't require say 6 physical
>>>>>>>>> machines but just one decent dual/quad xeon with say 6-8 Gb memory to be
>>>>>>>>> able to run enough virtual machines with enough bandwidth to be able to
>>>>>>>>> emulate a network of machines as a proof of concept.
>>>>>>>>> >
>>>>>>>>> > Regards,
>>>>>>>>> > Walter
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > ----- Original Message ----- From: "Dan Rossi" <
>>>>>>>>> electroteque at gmail.com>
>>>>>>>>> > To: <red5 at osflash.org>
>>>>>>>>> > Sent: Sunday, 07 February 2010 06:16
>>>>>>>>> > Subject: Re: [Red5] [Red5 0.9.0 final] Terracotta support
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >> Stephen Gong was working on this ages ago in 0.8.*. I tested it
>>>>>>>>> out and it worked I think only with sharedobjects support though, but not
>>>>>>>>> sure how the integration is going yet I am assuming it will require more
>>>>>>>>> conversation with the terracotta peeps who pop their head up in the list now
>>>>>>>>> and then. I think for a clustering solution red5 would integrate well with
>>>>>>>>> terracotta. One thing I noticed with edge / origin the origin still needs to
>>>>>>>>> be one big fk off server because its taking the load of the edge machines
>>>>>>>>> still especially on the network so terracotta and file caching might help
>>>>>>>>> here I suppose. I was testing on dual core xeon's at the time. I think the
>>>>>>>>> killer here is still the metadata stuff which needs to be moved to a memory
>>>>>>>>> cache perhaps, its more noticable on P4 or duo core than xeon though ie 100%
>>>>>>>>> cpu usage compared to 25% usage for the same traffic but 25% for 100 VOD
>>>>>>>>> streams on each frontend server is still quite high I reckon ;)
>>>>>>>>> >>
>>>>>>>>> >> Here is the diagram of the setup steve made
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> http://trac.red5.org/wiki/Documentation/Clustering/EdgeOriginSolutiononTerracotta
>>>>>>>>> >>
>>>>>>>>> >> So thats One or two origin servers, one delegating server for
>>>>>>>>> terracotta and 3 or 4 edge machines behind a load balancer. Pretty expensive
>>>>>>>>> and beefy setup. I dont have access to such a setup anymore since a client
>>>>>>>>> moved from red5 to FMS when they moved into a new data centre. Im still keen
>>>>>>>>> on setting up some amazon instances for testing such a setup if its not too
>>>>>>>>> expensive I could even use my server running openvz for a dev testbed but I
>>>>>>>>> believe the terracotta peeps have a serious clustering setup for a real
>>>>>>>>> testbed :)
>>>>>>>>> >>
>>>>>>>>> >> On 07/02/2010, at 8:15 AM, david.engelmaier wrote:
>>>>>>>>> >>
>>>>>>>>> >>> Hi guys,
>>>>>>>>> >>>
>>>>>>>>> >>> First of all I would like to say big THANK YOU for the new 0.9
>>>>>>>>> >>> release, especially for fixing the invoke memory leak bug.
>>>>>>>>> >>> Somewhere I saw an announcement of Terracotta out of the box
>>>>>>>>> >>> clustering in 0.9, but can't find anything about it in the
>>>>>>>>> changelog.
>>>>>>>>> >>> All the posts regarding Terracotta+Red5 clustering are about a
>>>>>>>>> year
>>>>>>>>> >>> old, is there any news in the 0.9 release concerning Terracotta
>>>>>>>>> >>> clustering?
>>>>>>>>> >>>
>>>>>>>>> >>> Many thanks
>>>>>>>>> >>>
>>>>>>>>> >>> David Engelmaier
>>>>>>>>> >>>
>>>>>>>>> >>> _______________________________________________
>>>>>>>>> >>> Red5 mailing list
>>>>>>>>> >>> Red5 at osflash.org
>>>>>>>>> >>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> _______________________________________________
>>>>>>>>> >> Red5 mailing list
>>>>>>>>> >> Red5 at osflash.org
>>>>>>>>> >> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > Red5 mailing list
>>>>>>>>> > Red5 at osflash.org
>>>>>>>>> > http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5 mailing list
>>>>>>>>> Red5 at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5 mailing list
>>>>>>>> Red5 at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5 mailing list
>>>>>>> Red5 at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5 mailing list
>>>>>> Red5 at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>  ------------------------------
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>
>
> --
> ... and they stirred up the Nazirites who had completed their days and they
> cried aloud to Heaven, saying, "What shall we do with these? Where shall we
> take them?"
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://osflash.org/pipermail/red5_osflash.org/attachments/20100209/858de4dc/attachment-0001.html>