[Red5devs] Root cause of CPU overload on VOD streaming (was Re: Red5 0.7 Sprint)
Dan Rossi
spam at electroteque.org
Wed Jan 30 01:19:59 PST 2008
I got your email steve from the archives, i cannot get emails from the
list for some reason today.
I updated trunk and ran a test, ive noticed the whole time a pattern
in the cpu usage. Its basically around 1% per request so 95-100
requests is 100%. It could be just the way this load tester flash is
loading the files im not sure ive been doing the request with 1 second
delay.
top - 20:21:28 up 1:42, 2 users, load average: 4.09, 2.66, 1.85
Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie
Cpu0 : 24.5%us, 26.8%sy, 0.0%ni, 48.3%id, 0.0%wa, 0.0%hi,
0.3%si, 0.0%st
Cpu1 : 31.7%us, 18.5%sy, 0.0%ni, 29.7%id, 0.0%wa, 11.6%hi,
8.6%si, 0.0%st
Mem: 1029420k total, 1019944k used, 9476k free, 16k buffers
Swap: 1959888k total, 20752k used, 1939136k free, 638992k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7034 root 18 0 504m 143m 10m S 122 14.3 4:02.09 java
at 100 request with 10 second delay
top - 20:19:16 up 1:40, 2 users, load average: 3.08, 2.53, 1.73
Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie
Cpu0 : 6.6%us, 15.0%sy, 0.0%ni, 78.1%id, 0.0%wa, 0.0%hi,
0.3%si, 0.0%st
Cpu1 : 8.3%us, 13.3%sy, 0.0%ni, 72.7%id, 0.0%wa, 3.0%hi,
2.7%si, 0.0%st
Mem: 1029420k total, 1018784k used, 10636k free, 16k buffers
Swap: 1959888k total, 20752k used, 1939136k free, 643964k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7034 root 18 0 476m 137m 10m S 48 13.7 3:03.85 java
On 29/01/2008, at 12:46 PM, Dan Rossi wrote:
> I have it back on the old copy and 48 requests gets about 22%. The
> problem is once it gets to 80 it also starts to max out. The new
> copies i keep trying to test max out much earlier than that. Its an
> issue because I cannot make a system using a custom webapp live
> until its somehow corrected because it uses alot of the logback
> changes for a custom access logging system for bandwidth billing of
> streams. I cannot seem to get the logback stuff to work in this
> version.
>
> On 29/01/2008, at 12:16 PM, Steven Gong wrote:
>
>>
>> On Jan 29, 2008 9:10 AM, Dan Rossi <spam at electroteque.org> wrote:
>> I have the latest svn on a server with flags set to false it seemed
>> to reach a higher cpu at 18 requests than with it set to true, im
>> seeing peaks at 90% with 34 requests, at 35 requests i got 100%.
>> This doesnt add up sorry. Maybe it has something to do with network
>> storage ?
>> Yes, maybe if you are getting the VOD via network. How do you read
>> the files?
>>
>>
>> On 29/01/2008, at 9:28 AM, Dan Rossi wrote:
>>
>>> With flag set to false and 100 requests
>>>
>>>
>>> top - 09:29:58 up 53 min, 2 users, load average: 9.25, 5.00, 4.40
>>> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
>>> Cpu0 : 40.3%us, 30.0%sy, 0.0%ni, 28.7%id, 0.0%wa, 0.0%hi,
>>> 1.0%si, 0.0%st
>>> Cpu1 : 17.3%us, 22.3%sy, 0.0%ni, 44.7%id, 0.0%wa, 8.7%hi,
>>> 7.0%si, 0.0%st
>>> Mem: 1029420k total, 999888k used, 29532k free, 372k
>>> buffers
>>> Swap: 1959888k total, 0k used, 1959888k free, 558692k
>>> cached
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 7521 root 17 0 505m 136m 10m S 126 13.6 1:31.58 java
>>>
>>> With flag set to true and 100 requests
>>>
>>> 90 users 100% on one core of the cpu 20% overall.
>>>
>>> top - 09:25:42 up 49 min, 2 users, load average: 5.74, 3.23, 3.91
>>> Tasks: 122 total, 1 running, 121 sleeping, 0 stopped, 0 zombie
>>> Cpu0 : 17.5%us, 31.1%sy, 0.0%ni, 50.7%id, 0.0%wa, 0.0%hi,
>>> 0.7%si, 0.0%st
>>> Cpu1 : 15.8%us, 22.8%sy, 0.0%ni, 47.2%id, 0.0%wa, 7.6%hi,
>>> 6.6%si, 0.0%st
>>> Mem: 1029420k total, 1010608k used, 18812k free, 372k
>>> buffers
>>> Swap: 1959888k total, 0k used, 1959888k free, 558836k
>>> cached
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 7189 root 17 0 502m 146m 10m S 101 14.6 4:02.54 java
>>>
>>>
>>> I don't see much of a difference :\
>>>
>>>
>>> On 29/01/2008, at 3:02 AM, Mondain wrote:
>>>
>>>> There no "fix" yet, but I think you may be able to set
>>>> generateMetadata to false in the interim. In red5-common.xml look
>>>> for this bean:
>>>> <bean id="flvFileService" class="org.red5.io.flv.impl.FLVService">
>>>> <property name="generateMetadata" value="true"/>
>>>> </bean>
>>>> Simply setting it to false may prevent your CPU spikes.
>>>>
>>>> Paul
>>>>
>>>> On Jan 28, 2008 12:42 AM, Dan Rossi <spam at electroteque.org> wrote:
>>>> Is the possible fixes committed at all ? I just tried on my
>>>> little ubuntu machine which is a 2.2G AMD X2 and reached 100% at
>>>> 100 users and was averaging at 85% the whole way through with the
>>>> loader tester app.
>>>>
>>>> On 26/01/2008, at 4:12 PM, Dan Rossi wrote:
>>>>
>>>>> I knew thats where it may have happened when the cache changed.
>>>>> The problem with this is has to identify files of same names
>>>>> inside different paths. I believe there was a clash there !
>>>>> Please let me know how you go, some frustrated bosses on my end
>>>>> wanting to bite the bullet and go with FMS and 1 and a bit years
>>>>> nearly wasted.
>>>>>
>>>>> Please let me know and I can test.
>>>>>
>>>>> On 26/01/2008, at 8:46 AM, Chris Allen wrote:
>>>>>
>>>>>> Guys,
>>>>>>
>>>>>> Let's get cranking on reverse engineering the H.264 stuff.
>>>>>>
>>>>>> http://www.flashcomguru.com/index.cfm/2008/1/25/fms3-released
>>>>>>
>>>>>> :-)
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>>
>>>>>> On Jan 23, 2008, at 8:09 PM, Steven Gong wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jan 24, 2008 12:24 AM, Chris Allen <mrchrisallen at gmail.com>
>>>>>>> wrote:
>>>>>>> Nice Steven! So, are you going to implement this fix?
>>>>>>> Yes, I will provide the fix before the release.
>>>>>>>
>>>>>>> -Chris
>>>>>>>
>>>>>>> On Jan 23, 2008, at 10:16 AM, Steven Gong wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> I found the root cause of big CPU load on VOD streaming. It
>>>>>>>> is caused by the loading of key frame metadata from metadata
>>>>>>>> cache. A simple solution to this is to enhance the metadata
>>>>>>>> cache implementation to cache the loaded metadata into memory
>>>>>>>> for FLV rather than load it again from the file system each
>>>>>>>> time.
>>>>>>>>
>>>>>>>> On Jan 22, 2008 11:58 PM, Mondain <mondain at gmail.com> wrote:
>>>>>>>> The load tester (src) is checked into the trunk btw..
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 21, 2008 10:34 PM, Luke Hubbard <luke at codegent.net >
>>>>>>>> wrote:
>>>>>>>> Hi Steven
>>>>>>>>
>>>>>>>> Yes please do.
>>>>>>>>
>>>>>>>> - Luke
>>>>>>>>
>>>>>>>>
>>>>>>>> On 1/22/08, Steven Gong < steven.gong at gmail.com> wrote:
>>>>>>>> Luke,
>>>>>>>> If you need Paul's load tester for the test, I can send a
>>>>>>>> copy to you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 22, 2008 7:34 AM, Dan Rossi < spam at electroteque.org>
>>>>>>>> wrote:
>>>>>>>> Paul's load tester flex app is a good one to use but i dont
>>>>>>>> think its accurate enough. I have tested the same app against
>>>>>>>> FMS and Wowza and there was only 5% cpu used maximum.
>>>>>>>>
>>>>>>>> On 21/01/2008, at 7:43 PM, Steven Gong wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jan 21, 2008 4:27 PM, Luke Hubbard < luke at codegent.net>
>>>>>>>>> wrote:
>>>>>>>>> I will try to test with a profiler see what is eating all
>>>>>>>>> that CPU.
>>>>>>>>> This is for VOD right, not live? Or both?
>>>>>>>>> This is the VOD streaming and can be reproduced by a simple
>>>>>>>>> load test VOD playback.
>>>>>>>>>
>>>>>>>>> - Luke
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 1/21/08, Dan Rossi < spam at electroteque.org> wrote:
>>>>>>>>> Less than that before its reaching 100% cpu. aparantly i go
>>>>>>>>> away on a break and everything goes down. It might be
>>>>>>>>> looking like we might have to jump onto the wowza bandwagon :\
>>>>>>>>>
>>>>>>>>> On 20/01/2008, at 9:55 AM, Dominick Accattato wrote:
>>>>>>>>>
>>>>>>>>>> Dan:
>>>>>>>>>>
>>>>>>>>>> are you saying that the core can only handle 50-80
>>>>>>>>>> concurrent connections?
>>>>>>>>>>
>>>>>>>>>> On Jan 19, 2008, at 6:28 AM, Dan Rossi wrote:
>>>>>>>>>>
>>>>>>>>>>> Great, i added the preliminary task in there but didnt get
>>>>>>>>>>> much responses about it at first :D Should we add back in
>>>>>>>>>>> the option to either compile javadoc in a full
>>>>>>>>>>> distribution task or not, that was added in as a
>>>>>>>>>>> suggestion from steve but was removed. It does reduce the
>>>>>>>>>>> recompile time alot.
>>>>>>>>>>>
>>>>>>>>>>> I believe the major one that i know of is the performance
>>>>>>>>>>> problems somewhere in the core, its maxing out at 50-80
>>>>>>>>>>> concurrents.
>>>>>>>>>>>
>>>>>>>>>>> On 19/01/2008, at 3:15 AM, Mondain wrote:
>>>>>>>>>>>
>>>>>>>>>>>> If we plan to inform users of the edge/origin features,
>>>>>>>>>>>> we will also need to get those parts of the build script
>>>>>>>>>>>> ironed-out before the release as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Paul
>>>>>>>>>>>>
>>>>>>>>>>>> On Jan 18, 2008 4:52 AM, Lenny Sorey < lrsorey at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hey Joachim,
>>>>>>>>>>>>
>>>>>>>>>>>> Welcome back!!
>>>>>>>>>>>>
>>>>>>>>>>>> Glad to see your return.
>>>>>>>>>>>>
>>>>>>>>>>>> Hope things are well with these days.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Lenny
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 1/17/08, Dominick Accattato < daccattato at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Team:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I want to give Joachim a warm welcome back. We missed
>>>>>>>>>>>> ya! Everyone has done a great job pushing Red5 forward
>>>>>>>>>>>> from the 0.6 release. In preparation of the release of
>>>>>>>>>>>> 0.7, I'd like to get a few more fixes in. If we can,
>>>>>>>>>>>> lets try to get this stuff done asap so that we can get
>>>>>>>>>>>> 0.7 out the door. Here is what I find to be of
>>>>>>>>>>>> importance for this release.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://jira.red5.org/browse/APPSERVER-184
>>>>>>>>>>>> amf3 synchronization issue with SharedObjects (I will
>>>>>>>>>>>> provide Joachim with a test app)
>>>>>>>>>>>> fix our tests since Paul noticed many of them are failing
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If anyone can think of any other important fixes, list
>>>>>>>>>>>> them so that we can look into their priority. Thanks team!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> It is difficult to free fools from the chains they
>>>>>>>>>>>> revere. - Voltaire
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Red5devs mailing list
>>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Red5devs mailing list
>>>>>>>>>> Red5devs at osflash.org
>>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Luke Hubbard
>>>>>>>>> codegent | coding for the people
>>>>>>>>> http://www.codegent.com
>>>>>>>>>
>>>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>>>> http://www.codegent.com/top100/
>>>>>>>>>
>>>>>>>>> want to know more?
>>>>>>>>> http://www.codegent.com/showreel/
>>>>>>>>>
>>>>>>>>> This e-mail may contain information which is privileged,
>>>>>>>>> confidential and protected from disclosure. If you are not
>>>>>>>>> the intended recipient of this e-mail, or any part of it,
>>>>>>>>> please delete this email and any attachments immediately on
>>>>>>>>> receipt. You should not disclose the contents to any other
>>>>>>>>> person or take copies. Any views expressed in this message
>>>>>>>>> are those of the individual sender, except where the sender
>>>>>>>>> specifically states them to be the views of codegent limited.
>>>>>>>>> _______________________________________________
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards
>>>>>>>>> Steven Gong
>>>>>>>>>
>>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>>
>>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>>> ~steven/Home
>>>>>>>>>
>>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>>> Red5devs mailing list
>>>>>>>>> Red5devs at osflash.org
>>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Luke Hubbard
>>>>>>>> codegent | coding for the people
>>>>>>>> http://www.codegent.com
>>>>>>>>
>>>>>>>> NMA Top 100 Interactive Agencies - Ones to watch!
>>>>>>>> http://www.codegent.com/top100/
>>>>>>>>
>>>>>>>> want to know more?
>>>>>>>> http://www.codegent.com/showreel/
>>>>>>>>
>>>>>>>> This e-mail may contain information which is privileged,
>>>>>>>> confidential and protected from disclosure. If you are not
>>>>>>>> the intended recipient of this e-mail, or any part of it,
>>>>>>>> please delete this email and any attachments immediately on
>>>>>>>> receipt. You should not disclose the contents to any other
>>>>>>>> person or take copies. Any views expressed in this message
>>>>>>>> are those of the individual sender, except where the sender
>>>>>>>> specifically states them to be the views of codegent limited.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> It is difficult to free fools from the chains they revere. -
>>>>>>>> Voltaire
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Steven Gong
>>>>>>>>
>>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com , steven at infrared5.com
>>>>>>>>
>>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>>> ~steven/Home
>>>>>>>>
>>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>>> accomplishment. -- Ricky Jay
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> Red5devs mailing list
>>>>>>>> Red5devs at osflash.org
>>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Steven Gong
>>>>>>>
>>>>>>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>>>>>>
>>>>>>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>>>>>>> ~steven/Home
>>>>>>>
>>>>>>> Modesty is an overrated quality in men of no great
>>>>>>> accomplishment. -- Ricky Jay
>>>>>>> _______________________________________________
>>>>>>> Red5devs mailing list
>>>>>>> Red5devs at osflash.org
>>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>>
>>>>>> _______________________________________________
>>>>>> Red5devs mailing list
>>>>>> Red5devs at osflash.org
>>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>>
>>>>> _______________________________________________
>>>>>
>>>>> Red5devs mailing list
>>>>> Red5devs at osflash.org
>>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> It is difficult to free fools from the chains they revere. -
>>>> Voltaire _______________________________________________
>>>> Red5devs mailing list
>>>> Red5devs at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>>
>>> _______________________________________________
>>>
>>> Red5devs mailing list
>>> Red5devs at osflash.org
>>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>> _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>>
>>
>>
>>
>> --
>> Best Regards
>> Steven Gong
>>
>> InfraRed5 Red5 Consultant: http://www.infrared5.com, steven at infrared5.com
>>
>> Red5 Developer: http://osflash.org/red5, http://jira.red5.org/confluence/display/
>> ~steven/Home
>>
>> Modesty is an overrated quality in men of no great accomplishment.
>> -- Ricky Jay _______________________________________________
>> Red5devs mailing list
>> Red5devs at osflash.org
>> http://osflash.org/mailman/listinfo/red5devs_osflash.org
>
> _______________________________________________
> Red5devs mailing list
> Red5devs at osflash.org
> http://osflash.org/mailman/listinfo/red5devs_osflash.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5devs_osflash.org/attachments/20080130/2ab8b535/attachment-0001.html
More information about the Red5devs
mailing list