[Red5devs] Replicating a live stream between servers
Victor
dreamland_skib2 at mail333.com
Wed Sep 12 11:00:17 PDT 2007
Steven,
thanks for the explanation!
We have had one more discussion today with my college, I think that
after the MRTMP improvement you propose (I have illustrated how I see it
- look at the attached picture) there is no need to make AV stream
replication between Origins.
I agree that Origin should store ALL STATE, manage client connections,
etc; Edge will be just a lightweight proxy, it will only decode and
encode RTMP/MRTMP packets and forward them (optionally, Edge can provide
caching and security services, as you say).
Let us assume that ALL Edges will be connected to ALL Origins in a
cluster - this way a Provider may connect to any Edge server (and will
be stickied to one Origin), his consumers may connect to some other Edge
server, but they will be forwarded to the same Origin where the
Performer's connection is managed (see the illustration).
Having this and the optimized MRTMP, seems that we can scale the cluster
as necessary in any situation - without replicating AV streams between
Origins.
I am going to analyze the mathematical side of this approach - how
scalable it is.
Victor
Steven Gong wrote:
> Victor,
>
> On 9/12/07, *Victor* <dreamland_skib2 at mail333.com
> <mailto:dreamland_skib2 at mail333.com>> wrote:
>
> Steven,
>
> I have looked at the clustering branch, especially MRTMP-related
> classes. Correct me if I am wrong - in the live streaming
> scenario, the Edge-Origin approach does not reduce the traffic
> between Origin server and clients because ALL packets for all
> clients are forwarded from Edge to Origin and from Origin to Edge.
> Yes, I see that Edge server multiplexes data from all clients and
> sends it to Origin in one TCP connection, but the number of
> packets sent between Edge and Origin is not reduced.
> In our approach, we want to minimize traffic between Edge and
> Origin. For example, if we have 1 producer and 100 consumers
> connected to the same Edge server, we would like to have only 2
> data streams between Edge and Origin because one stream could be
> sent to ALL connected consumers. Please see the illustration attached.
>
>
> Good catch! The Origin SHOULD only send out one copy of data to each
> Edge for all broadcasting cases. These include: live streaming (video
> conference or server-side streaming), SharedObject synchronization and
> invocation etc.
>
> Currently I haven't implemented this because I used the fast
> prototyping approach to make this first prototype. So I didn't do much
> refactoring and leave the server framework as is. I believe this is
> the best approach to make a working implementation: lower cost, less
> regressions and quicker demo.
>
> We might be lost in the discussions and argues for the clustering
> solution without a working implementation. I think this working copy
> can not only provide one of the solutions (needs enhancing of course)
> but also set up a basis for discussions and argues. So now, the Edge
> server is implemented only with client connection management, with
> multiplexed RTMP connection, with only RTMPT supported, without cache
> service and without security service. I am sure that these limitations
> will be fixed in the near future.
>
> Your diagram looks good. But it seems that we need to be careful to
> find a good design solution for this scenario. Are you assuming that
> the Edge server should maintain the relationship of each client? I
> don't think it's a good idea to put this burden to the Edge. The main
> responsibility of Edge server is to provide a proxy-like entity to
> proxy the request and response, to filter them for security reasons
> and to cache them for faster server response time. But the logics have
> to go to the Origin server. The relationship of clients that is
> created as the result of live streaming or shared object is, in no
> doubt, a logical thing and should be maintained on Origin servers.
>
> You might argue that we can cache the relationship on the Edge too and
> keep it synchronized between Edges and Origin. But with that, you will
> end up with the diagram as Rob provided: the Edge and Origin are bound
> together and the clustering is simply among Edge/Origin monolithic
> server because the client relationship maintenance is one of the major
> parts of Origin.
>
> One thought about sending audio/video data between Edge and
> Origin... Seems that instead of "virtual provider / virtual
> consumer" idea it is possible to look at incoming RTMP packets in
> RTMPMinaIoHandler.messageReceived(), filter them somehow and
> forward some of them to another Red5 instance (Origin, for
> example) via standart RTMP (1935) port. Of course, we need to
> filter incoming packets and forward not all of them but just a
> part; in addition, it will be necessary to react to "ping"
> messages (and maybe some others) sent in the reverse direction
> from this second Red5 node. I am going to play with this tomorrow.
>
>
> OK, I am fine with this solution as long as the virtual pipe is built
> between Origins rather than between Edges.
>
> What do you think about this?
>
> Thanks,
> Victor
>
>
>
>
> Victor wrote:
>> Thanks Steven,
>>
>> yes, the usage of both approaches together looks interesting! I
>> am trying to understand how it could be done ;)
>> Of course I will be glad to discuss the ideas and publish our
>> future clustering code, if it will be interesting.
>>
>> If I understand the Red5's "Edge-Origin" architecture correctly,
>> each incoming AV stream should be routed to an Origin server? If
>> this is true, you need several Origin servers in a cluster and
>> direct performers connections to different Origins - otherwise
>> you can reach the performance limit of one Origin server (when
>> there are thousands of performers). But do you need replication
>> of data between Origin servers? Isn't it possible to connect each
>> Edge to ALL Origins instead? How this problem with huge number of
>> performers is resolved in the "Edge-Origin" architecture?
>>
>> If, potentially, our approach could be used to replicate AV
>> streams between Origins, could not it also be applied to
>> replicate between Edge and Origin? In your architecture, after
>> the first implementation step, how are you going to send AV data
>> between Edge->Origin and Origin->Edge?
>>
>> And one technical question: where to start in Red5 code to add a
>> virtual consumer and virtual provider?
>>
>> Victor
>>
>>
>>
>> Steven Gong wrote:
>>> Hi Victor,
>>> Your design block for virtual provider/consumer looks good.
>>> Currently the Red5 clustering prototype uses terracotta to share
>>> everything in the sending buffer including AV packets and other
>>> RTMP packets. But that will be enhanced as was indicated by
>>> terracotta experts that sharing raw data on terracotta is
>>> expensive. The approach to enhance this is to put the buffer on
>>> to the Origin server. Please refer to the wiki page on
>>> Terracotta for detail:
>>> http://www.terracotta.org/confluence/display/wiki/Red5
>>> <http://www.terracotta.org/confluence/display/wiki/Red5>
>>>
>>> Back to your approach. I think that your solution does not
>>> conflict with the Red5 clustering solution we are working on.
>>> All your blocks could be put on the Origin server. Take live
>>> conference as an example, for our current design, all the
>>> connections should be routed to the same Origin server to be
>>> served. Now with the virtual tunnel between the Origin servers,
>>> we can provide more than one Origin server to serve a live
>>> conference session. That's promising, isn't it?
>>>
>>> You are welcomed to join our clustering effort to make this
>>> happen. :-)
>>>
>>> On 9/10/07, *Victor* <dreamland_skib2 at mail333.com
>>> <mailto:dreamland_skib2 at mail333.com>> wrote:
>>>
>>> Steven,
>>>
>>> I was reading your discussion about clustering with
>>> Terracotta this weekend.
>>> It's great! Maybe our clustering solution will be
>>> unnecessary soon :) Are you going to replicate video this
>>> way - using Terracotta shared storage?
>>>
>>> I believe that choice is good - that's why we continue
>>> working at our own clustering solution based on JGroups and
>>> live video replication between Red5 instances. But we need
>>> some help with Red5.
>>>
>>> If possible, please give us some ideas - where to start the
>>> integration of "virtual consumer" and "virtual provider"
>>> with Red5 internals (see the attached picture explaining our
>>> idea).
>>>
>>> Thanks,
>>>
>>> Victor
>>>
>>>
>>>
>>> Victor wrote:
>>>> Steven,
>>>>
>>>> thanks for your answer. Seems that we are talking about 2
>>>> different approaches in how to replicate video data between
>>>> servers:
>>>>
>>>> 1) implementing a Pipe which will work via network - this
>>>> is what you are implementing with Terracotta, correct?
>>>> 2) adding a special Consumer at server-1 and a special
>>>> Provider + BroadcastScope at server-2 (which will
>>>> communicate each other) - this is what we are thinking
>>>> about, please see the illustration attached.
>>>>
>>>> What do you think about our approach? Is it possible? Where
>>>> to start in Red5?
>>>> Yes, I see that there are many things to do - not only
>>>> sending video/audio messages between servers. We are going
>>>> to use JGroups to manage membership in cluster, JGroups can
>>>> also be used to send "control" messages between nodes (such
>>>> as "ReplicateBroadcastScope with name=xxx at node yyy",
>>>> etc). But video/audio packets will be sent directly using
>>>> UDP protocol.
>>>> What do you think about this?
>>>>
>>>> Victor
>>>>
>>>>
>>>> Steven Gong wrote:
>>>>> Hi Victor,
>>>>>
>>>>> On 8/30/07, *Victor* <dreamland_skib2 at mail333.com
>>>>> <mailto:dreamland_skib2 at mail333.com>
>>>>> <mailto:dreamland_skib2 at mail333.com>> wrote:
>>>>>
>>>>> Dear Red5 developers,
>>>>>
>>>>> we are investigating different cluster architectures
>>>>> using Red5. I
>>>>> read
>>>>> your emails about using Terracotta for clustering.
>>>>> But in our project, which uses live video/audio
>>>>> streaming via
>>>>> Red5, we
>>>>> are going to replicate video/audio packets between 2
>>>>> or more Red5
>>>>> instances located in one LAN. I would like to know
>>>>> your opinion - what
>>>>> is the best way to do the following things:
>>>>> 1) grab video/audio packets in Red5-instance1
>>>>> 2) put these packets into Red5-instance2
>>>>>
>>>>> That is I am talking about integration points with
>>>>> Red5. Network
>>>>> protocol between servers could be some UDP- or
>>>>> TCP-based protocol,
>>>>> this
>>>>> is not a problem.
>>>>>
>>>>> Should we implement a separate consumer at
>>>>> Red5-instance1?
>>>>> What about Red5-instance2? Do we need to create a
>>>>> separate
>>>>> BroadcastScope and a provider there which receives
>>>>> video/audio packets
>>>>> from Red5-instance1 via network and publishes them to
>>>>> its own
>>>>> consumers?
>>>>>
>>>>> Which classes in Red5 may be interesting for me? Where
>>>>> to start the
>>>>> investigation?
>>>>>
>>>>>
>>>>> The streaming framework is built from a messaging system
>>>>> and all the components are connected with pipes. So if you
>>>>> want to send packets from one component to another, I
>>>>> believe it's a good idea to implement a networking pipe
>>>>> instead of a "InMemory" one.
>>>>>
>>>>> The detail could be: originally the message is pushed from
>>>>> "BroadcastScope" (which is actually a Provider) from a
>>>>> "InMemory" pipe to a subscriber
>>>>> (PlaylistSubscriberStream). Now you make the pipe across
>>>>> the network so that the message will be pushed to the
>>>>> networking pipe (via the network) to the
>>>>> PlaylistSubscriberStream.
>>>>>
>>>>> The big problem for your solution is how to share the
>>>>> server states across the nodes. For example, when a
>>>>> broadcast scope is created in one node, how could another
>>>>> node know it. We use Terracotta to solve this.
>>>>>
>>>>> Any starting points would be great!
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Victor
>>>>>
>>>>> _______________________________________________
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org <mailto:Red5devs at osflash.org>
>>>>> <mailto:Red5devs at osflash.org>
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> Steven Gong
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>
>>>
>>> _______________________________________________
>>> Red5devs mailing list
>>> Red5devs at osflash.org <mailto:Red5devs at osflash.org>
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> Steven Gong
>>>
>>> InfraRed5 Red5 Consultant: http://www.infrared5.com,
>>> steven at infrared5.com <mailto:steven at infrared5.com>
>>>
>>> Red5 Developer: http://osflash.org/red5,
>>> http://jira.red5.org/confluence/display/~steven/Home
>>> <http://jira.red5.org/confluence/display/%7Esteven/Home>
>>>
>>> Modesty is an overrated quality in men of no great
>>> accomplishment. -- Ricky Jay
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Red5devs mailing list
>>> Red5devs at osflash.org <mailto:Red5devs at osflash.org>
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>
>
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org <mailto:Red5devs at osflash.org>
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
>
>
>
>
> --
> Best Regards
> Steven Gong
>
> InfraRed5 Red5 Consultant: http://www.infrared5.com,
> steven at infrared5.com <mailto:steven at infrared5.com>
>
> Red5 Developer: http://osflash.org/red5,
> http://jira.red5.org/confluence/display/~steven/Home
> <http://jira.red5.org/confluence/display/%7Esteven/Home>
>
> Modesty is an overrated quality in men of no great accomplishment. --
> Ricky Jay
> ------------------------------------------------------------------------
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: red5_cluster3.png
Type: image/png
Size: 37142 bytes
Desc: not available
Url : http://osflash.org/pipermail/red5devs_osflash.org/attachments/20070912/b953fd7c/attachment-0001.png
More information about the Red5devs
mailing list