[Red5] About the Fix of FLV and FLVReader in regards with OutOfMemoryError when playing large FLVs
Daniel Daley
dan at chameleoncode.jp
Tue Nov 21 20:27:34 EST 2006
I hacked together a quick version of FLVReader that eliminates the
memory mapping and pulls straight from the FileInputStream channel
and it so far has been able to handle much much more. The load on the
server is terribly high of course. Is this something that could be
implemented in the server as an option? It would probably be better
load wise if I created a buffer of a fixed size (couple hundred K or
something) and it would just keep that buffer filled with data from
the file as it needed it. I'll give that a shot if I get a chance but
having something like this in the actual Red5 distribution would be
wonderful, as loading the whole file to memory puts way too small of
a cap on how much we can serve.
Thanks,
--Dan--
On Nov 21, 2006, at 1:19 PM, Daniel Daley wrote:
> Looking at the code I guess it does load the file into memory
> completely.
>
> channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
>
> Wouldn't it be more efficient/scalable to have it seek around the
> FileInputStream and read the appropriate bytes for header data etc,
> and then have a configurable sized ByteBuffer for the reading of
> the main file data?
>
> Thanks,
>
> --Dan--
>
>
> On Nov 21, 2006, at 12:09 PM, Daniel Daley wrote:
>
>> What I think I'm running into here is the 2 gig per process memory
>> limit of 32 bit processors. Is there any way to reduce the amount
>> of memory required per connection? Right now it is still acting as
>> if it loads the entire file into memory as our files are on
>> average 20 megs and we can get up to around 100 connections before
>> the crash. Our server has 4 gigs of ram in it so we have plenty
>> left to use. I don't understand the java threading system very
>> well, but shouldn't the threading that red5 does create a separate
>> process with it's own limits per connection? Or do all connections
>> share the single master red5 process?
>>
>> Thanks,
>>
>> --Dan--
>>
>> On Nov 21, 2006, at 11:23 AM, Daniel Daley wrote:
>>
>>> I spoke too soon. Once the connections starting picking up today
>>> we went back to the previous errors. This one was at 100
>>> connected viewers:
>>>
>>> 2006-11-21 13:17:59.572534500 [ERROR] 194490 pool-1-thread-4:
>>> ( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader ::
>>> >
>>> 2006-11-21 13:17:59.572539500
>>> 2006-11-21 13:17:59.572541500 java.io.IOException: Cannot
>>> allocate memory
>>> 2006-11-21 13:17:59.572543500 at sun.nio.ch.FileChannelImpl.map0
>>> (Native Method)
>>> 2006-11-21 13:17:59.572545500 at sun.nio.ch.FileChannelImpl.map
>>> (FileChannelImpl.java:742)
>>> 2006-11-21 13:17:59.572548500 at
>>> org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
>>> 2006-11-21 13:17:59.572550500 at
>>> org.red5.io.flv.impl.FLV.getReader(FLV.java:172)
>>> 2006-11-21 13:17:59.572733500 at
>>> org.red5.server.stream.provider.FileProvider.init
>>> (FileProvider.java:179)
>>> 2006-11-21 13:17:59.572737500 at
>>> org.red5.server.stream.provider.FileProvider.pullMessage
>>> (FileProvider.java:87)
>>> 2006-11-21 13:17:59.572739500 at
>>> org.red5.server.messaging.InMemoryPullPullPipe.pullMessage
>>> (InMemoryPullPullPipe.java:72)
>>> 2006-11-21 13:17:59.572742500 at
>>> org.red5.server.stream.PlaylistSubscriberStream
>>> $PlayEngine.pullAndPush(PlaylistSubscriberStr
>>> eam.java:869)
>>> 2006-11-21 13:17:59.572789500 at
>>> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play
>>> (PlaylistSubscriberStream.jav
>>> a:703)
>>> 2006-11-21 13:17:59.572792500 at
>>> org.red5.server.stream.PlaylistSubscriberStream.play
>>> (PlaylistSubscriberStream.java:127)
>>> 2006-11-21 13:17:59.572795500 at
>>> org.red5.server.stream.StreamService.play(StreamService.java:175)
>>> 2006-11-21 13:17:59.572797500 at
>>> org.red5.server.stream.StreamService.play(StreamService.java:183)
>>> 2006-11-21 13:17:59.572821500 at
>>> sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
>>> 2006-11-21 13:17:59.572823500 at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke
>>> (DelegatingMethodAccessorImpl.java:25)
>>> 2006-11-21 13:17:59.572826500 at java.lang.reflect.Method.invoke
>>> (Method.java:585)
>>> 2006-11-21 13:17:59.572828500 at
>>> org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:
>>> 161)
>>> 2006-11-21 13:17:59.572830500 at
>>> org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:
>>> 287)
>>> 2006-11-21 13:17:59.572840500 at
>>> org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:469)
>>> 2006-11-21 13:17:59.572842500 at
>>> org.red5.server.net.rtmp.RTMPHandler.messageReceived
>>> (RTMPHandler.java:147)
>>> 2006-11-21 13:17:59.572844500 at
>>> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
>>> (RTMPMinaIoHandler.java:78)
>>> 2006-11-21 13:17:59.572847500 at
>>> org.apache.mina.common.support.AbstractIoFilterChain
>>> $2.messageReceived(AbstractIoFilterChain
>>> .java:189)
>>> 2006-11-21 13:17:59.572856500 at
>>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessage
>>> Received(AbstractIoFilte
>>> rChain.java:502)
>>> 2006-11-21 13:17:59.572858500 at
>>> org.apache.mina.common.support.AbstractIoFilterChain.access$1000
>>> (AbstractIoFilterChain.java:
>>> 52)
>>> 2006-11-21 13:17:59.572861500 at
>>> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
>>> $1.messageReceived(AbstractIoF
>>> ilterChain.java:777)
>>> 2006-11-21 13:17:59.572876500 at
>>> org.red5.io.filter.ExecutorFilter.processEvent
>>> (ExecutorFilter.java:231)
>>> 2006-11-21 13:17:59.572879500 at
>>> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
>>> (ExecutorFilter.java:279)
>>> 2006-11-21 13:17:59.572882500 at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>>> (ThreadPoolExecutor.java:650)
>>> 2006-11-21 13:17:59.572885500 at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run
>>> (ThreadPoolExecutor.java:675)
>>> 2006-11-21 13:17:59.572952500 at java.lang.Thread.run
>>> (Thread.java:595)
>>>
>>>
>>> On Nov 20, 2006, at 5:58 PM, Steven Gong wrote:
>>>
>>>>
>>>>
>>>> On 11/21/06, Daniel Daley <dan at chameleoncode.jp> wrote:
>>>> Looks like it actually just crashes when it reaches whatever amount
>>>> of ram the process is limited to. Being 32 bit if I leave all
>>>> limits
>>>> off once red5 allocates 2 gigs it crashes. Oddly though since this
>>>> patch the ram just seems to grow endlessly until it dies rather
>>>> than
>>>> fluctuating like it did before. Could there be something that is
>>>> not
>>>> releasing memory after each connection?
>>>>
>>>> It's quite strange that your result is different from that of
>>>> Bill's.
>>>>
>>>> Could you please try to disable the buffer by setting the type
>>>> as "none"? And use NoCacheImpl please.
>>>>
>>>> Thanks,
>>>>
>>>> --Dan--
>>>>
>>>>
>>>> On Nov 20, 2006, at 3:01 PM, Daniel Daley wrote:
>>>>
>>>> > Looks like mine is crashing much more frequently now with this
>>>> patch
>>>> > but with the same errors each time. What's odd is the server
>>>> does not
>>>> > in the least seem to be out of memory when it happens. Here's the
>>>> > output in my log:
>>>> >
>>>> > 2006-11-20 16:58:39.560730500 [ERROR] 437909 pool-1-thread-4:
>>>> > ( org.red5.io.flv.impl.FLVReader.error ) FLVReader ::
>>>> FLVReader ::>
>>>> > 2006-11-20 16:58:39.560735500
>>>> > 2006-11-20 16:58:39.560737500 java.io.IOException: Cannot
>>>> allocate
>>>> > memory
>>>> > 2006-11-20 16:58:39.560739500 at
>>>> sun.nio.ch.FileChannelImpl.map0
>>>> > (Native Method)
>>>> > 2006-11-20 16:58:39.560741500 at
>>>> sun.nio.ch.FileChannelImpl.map
>>>> > (FileChannelImpl.java:742)
>>>> > 2006-11-20 16:58:39.560744500 at
>>>> > org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
>>>> > 2006-11-20 16:58:39.560746500 at
>>>> org.red5.io.flv.impl.FLV.getReader
>>>> > (FLV.java:172)
>>>> > 2006-11-20 16:58:39.560917500 at
>>>> > org.red5.server.stream.provider.FileProvider.init
>>>> (FileProvider.java:
>>>> > 179)
>>>> > 2006-11-20 16:58: 39.560920500 at
>>>> > org.red5.server.stream.provider.FileProvider.pullMessage
>>>> > (FileProvider.java:87)
>>>> > 2006-11-20 16:58:39.560922500 at
>>>> > org.red5.server.messaging.InMemoryPullPullPipe.pullMessage
>>>> > (InMemoryPullPullPipe.java:72)
>>>> > 2006-11-20 16:58:39.560925500 at
>>>> > org.red5.server.stream.PlaylistSubscriberStream
>>>> $PlayEngine.pullAndPush
>>>> > (PlaylistSubscriberStream.java:869)
>>>> > 2006-11-20 16:58: 39.560937500 at
>>>> > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play
>>>> > (PlaylistSubscriberStream.java:703)
>>>> > 2006-11-20 16:58:39.560940500 at
>>>> > org.red5.server.stream.PlaylistSubscriberStream.play
>>>> > (PlaylistSubscriberStream.java:127)
>>>> > 2006-11-20 16:58:39.560943500 at
>>>> > org.red5.server.stream.StreamService.play(StreamService.java:175)
>>>> > 2006-11-20 16:58:39.560945500 at
>>>> > org.red5.server.stream.StreamService.play (StreamService.java:
>>>> 183)
>>>> > 2006-11-20 16:58:39.560960500 at
>>>> > sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
>>>> > 2006-11-20 16:58:39.560962500 at
>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke
>>>> > (DelegatingMethodAccessorImpl.java:25)
>>>> > 2006-11-20 16:58:39.560965500 at
>>>> java.lang.reflect.Method.invoke
>>>> > (Method.java:585)
>>>> > 2006-11-20 16:58:39.560967500 at
>>>> > org.red5.server.service.ServiceInvoker.invoke
>>>> (ServiceInvoker.java:161)
>>>> > 2006-11-20 16:58:39.560969500 at
>>>> > org.red5.server.net.rtmp.RTMPHandler.invokeCall
>>>> (RTMPHandler.java:287)
>>>> > 2006-11-20 16:58:39.560978500 at
>>>> > org.red5.server.net.rtmp.RTMPHandler.onInvoke
>>>> (RTMPHandler.java:469)
>>>> > 2006-11-20 16:58:39.560980500 at
>>>> > org.red5.server.net.rtmp.RTMPHandler.messageReceived
>>>> (RTMPHandler.java:
>>>> > 147)
>>>> > 2006-11-20 16:58:39.560982500 at
>>>> > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
>>>> > (RTMPMinaIoHandler.java:78)
>>>> > 2006-11-20 16:58:39.560985500 at
>>>> > org.apache.mina.common.support.AbstractIoFilterChain
>>>> $2.messageReceived
>>>> > (AbstractIoFilterChain.java:189)
>>>> > 2006-11-20 16:58: 39.560993500 at
>>>> >
>>>> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessag
>>>> eRe
>>>> > ce
>>>> > ived(AbstractIoFilterChain.java:502)
>>>> > 2006-11-20 16:58:39.560996500 at
>>>> > org.apache.mina.common.support.AbstractIoFilterChain.access$1000
>>>> > (AbstractIoFilterChain.java:52)
>>>> > 2006-11-20 16:58:39.560998500 at
>>>> > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
>>>> > $1.messageReceived(AbstractIoFilterChain.java:777)
>>>> > 2006-11-20 16:58: 39.561012500 at
>>>> > org.red5.io.filter.ExecutorFilter.processEvent
>>>> (ExecutorFilter.java:
>>>> > 231)
>>>> > 2006-11-20 16:58:39.561015500 at
>>>> org.red5.io.filter.ExecutorFilter
>>>> > $ProcessEventsRunnable.run(ExecutorFilter.java :279)
>>>> > 2006-11-20 16:58:39.561017500 at
>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>>>> > (ThreadPoolExecutor.java:650)
>>>> > 2006-11-20 16:58:39.561019500 at
>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run
>>>> > (ThreadPoolExecutor.java:675)
>>>> > 2006-11-20 16:58:39.561028500 at java.lang.Thread.run
>>>> (Thread.java:
>>>> > 595)
>>>> >
>>>> >
>>>> > On Nov 18, 2006, at 2:48 AM, Steven Gong wrote:
>>>> >
>>>> >> Hi Paul,
>>>> >> I have just committed some codes to fix the OutOfMemoryError.
>>>> The
>>>> >> related revision numbers are 1555, 1556 and 1557.
>>>> >>
>>>> >> The OutOfMemoryError is mainly due to the buffer in FLVReader
>>>> so I
>>>> >> added a new static oroperty 'maxBufferSize' to constrain the
>>>> buffer
>>>> >> size and also added a way to disable the buffer if none of
>>>> 'auto',
>>>> >> 'direct' and 'heap' buffer type is configured. The property can
>>>> >> also be configured in red5-common.xml.
>>>> >>
>>>> >> Another improvement is to let ICacheStore create ICacheable
>>>> >> instances so that in the case of NoCacheImpl, no instances
>>>> will be
>>>> >> created thus we won't create byte array in this case.
>>>> >>
>>>> >> Could you please have a review? Thanks. :-)
>>>> >>
>>>> >> As the OutOfMemoryError bug gives rise to many complaints of our
>>>> >> users, I also cc this mail to red5 list for notification
>>>> purpose.
>>>> >>
>>>> >> --
>>>> >> I cannot tell why this heart languishes in silence. It is for
>>>> small
>>>> >> needs it never asks, or knows or remembers. -- Tagore
>>>> >>
>>>> >> Best Regards
>>>> >> Steven Gong
>>>> >> _______________________________________________
>>>> >> Red5 mailing list
>>>> >> Red5 at osflash.org
>>>> >> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Red5 mailing list
>>>> > Red5 at osflash.org
>>>> > http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>>
>>>>
>>>>
>>>> --
>>>> I cannot tell why this heart languishes in silence. It is for
>>>> small needs it never asks, or knows or remembers. -- Tagore
>>>>
>>>> Best Regards
>>>> Steven Gong
>>>> _______________________________________________
>>>> Red5 mailing list
>>>> Red5 at osflash.org
>>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20061121/9ec7ad39/attachment-0001.htm
More information about the Red5
mailing list