[Red5] About the Fix of FLV and FLVReader in regards with OutOfMemoryError when playing large FLVs

[Daniel Daley]

I spoke too soon. Once the connections starting picking up today we  
went back to the previous errors. This one was at 100 connected viewers:

2006-11-21 13:17:59.572534500 [ERROR] 194490 pool-1-thread-4: 
( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader ::
 >
2006-11-21 13:17:59.572539500
2006-11-21 13:17:59.572541500 java.io.IOException: Cannot allocate  
memory
2006-11-21 13:17:59.572543500   at sun.nio.ch.FileChannelImpl.map0 
(Native Method)
2006-11-21 13:17:59.572545500   at sun.nio.ch.FileChannelImpl.map 
(FileChannelImpl.java:742)
2006-11-21 13:17:59.572548500   at  
org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
2006-11-21 13:17:59.572550500   at org.red5.io.flv.impl.FLV.getReader 
(FLV.java:172)
2006-11-21 13:17:59.572733500   at  
org.red5.server.stream.provider.FileProvider.init(FileProvider.java:179)
2006-11-21 13:17:59.572737500   at  
org.red5.server.stream.provider.FileProvider.pullMessage 
(FileProvider.java:87)
2006-11-21 13:17:59.572739500   at  
org.red5.server.messaging.InMemoryPullPullPipe.pullMessage 
(InMemoryPullPullPipe.java:72)
2006-11-21 13:17:59.572742500   at  
org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.pullAndPush 
(PlaylistSubscriberStr
eam.java:869)
2006-11-21 13:17:59.572789500   at  
org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play 
(PlaylistSubscriberStream.jav
a:703)
2006-11-21 13:17:59.572792500   at  
org.red5.server.stream.PlaylistSubscriberStream.play 
(PlaylistSubscriberStream.java:127)
2006-11-21 13:17:59.572795500   at  
org.red5.server.stream.StreamService.play(StreamService.java:175)
2006-11-21 13:17:59.572797500   at  
org.red5.server.stream.StreamService.play(StreamService.java:183)
2006-11-21 13:17:59.572821500   at  
sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
2006-11-21 13:17:59.572823500   at  
sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:25)
2006-11-21 13:17:59.572826500   at java.lang.reflect.Method.invoke 
(Method.java:585)
2006-11-21 13:17:59.572828500   at  
org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:161)
2006-11-21 13:17:59.572830500   at  
org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:287)
2006-11-21 13:17:59.572840500   at  
org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:469)
2006-11-21 13:17:59.572842500   at  
org.red5.server.net.rtmp.RTMPHandler.messageReceived(RTMPHandler.java: 
147)
2006-11-21 13:17:59.572844500   at  
org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived 
(RTMPMinaIoHandler.java:78)
2006-11-21 13:17:59.572847500   at  
org.apache.mina.common.support.AbstractIoFilterChain$2.messageReceived 
(AbstractIoFilterChain
.java:189)
2006-11-21 13:17:59.572856500   at  
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece 
ived(AbstractIoFilte
rChain.java:502)
2006-11-21 13:17:59.572858500   at  
org.apache.mina.common.support.AbstractIoFilterChain.access$1000 
(AbstractIoFilterChain.java:
52)
2006-11-21 13:17:59.572861500   at  
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl 
$1.messageReceived(AbstractIoF
ilterChain.java:777)
2006-11-21 13:17:59.572876500   at  
org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java:231)
2006-11-21 13:17:59.572879500   at org.red5.io.filter.ExecutorFilter 
$ProcessEventsRunnable.run(ExecutorFilter.java:279)
2006-11-21 13:17:59.572882500   at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask 
(ThreadPoolExecutor.java:650)
2006-11-21 13:17:59.572885500   at  
java.util.concurrent.ThreadPoolExecutor$Worker.run 
(ThreadPoolExecutor.java:675)
2006-11-21 13:17:59.572952500   at java.lang.Thread.run(Thread.java:595)


On Nov 20, 2006, at 5:58 PM, Steven Gong wrote:

>
>
> On 11/21/06, Daniel Daley <dan at chameleoncode.jp> wrote:
> Looks like it actually just crashes when it reaches whatever amount
> of ram the process is limited to. Being 32 bit if I leave all limits
> off once red5 allocates 2 gigs it crashes. Oddly though since this
> patch the ram just seems to grow endlessly until it dies rather than
> fluctuating like it did before. Could there be something that is not
> releasing memory after each connection?
>
> It's quite strange that your result is different from that of Bill's.
>
> Could you please try to disable the buffer by setting the type as  
> "none"? And use NoCacheImpl please.
>
> Thanks,
>
> --Dan--
>
>
> On Nov 20, 2006, at 3:01 PM, Daniel Daley wrote:
>
> > Looks like mine is crashing much more frequently now with this patch
> > but with the same errors each time. What's odd is the server does  
> not
> > in the least seem to be out of memory when it happens. Here's the
> > output in my log:
> >
> > 2006-11-20 16:58:39.560730500 [ERROR] 437909 pool-1-thread-4:
> > ( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader ::>
> > 2006-11-20 16:58:39.560735500
> > 2006-11-20 16:58:39.560737500 java.io.IOException: Cannot allocate
> > memory
> > 2006-11-20 16:58:39.560739500   at sun.nio.ch.FileChannelImpl.map0
> > (Native Method)
> > 2006-11-20 16:58:39.560741500    at sun.nio.ch.FileChannelImpl.map
> > (FileChannelImpl.java:742)
> > 2006-11-20 16:58:39.560744500   at
> > org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
> > 2006-11-20 16:58:39.560746500    at  
> org.red5.io.flv.impl.FLV.getReader
> > (FLV.java:172)
> > 2006-11-20 16:58:39.560917500   at
> > org.red5.server.stream.provider.FileProvider.init(FileProvider.java:
> > 179)
> > 2006-11-20 16:58: 39.560920500   at
> > org.red5.server.stream.provider.FileProvider.pullMessage
> > (FileProvider.java:87)
> > 2006-11-20 16:58:39.560922500   at
> > org.red5.server.messaging.InMemoryPullPullPipe.pullMessage
> > (InMemoryPullPullPipe.java:72)
> > 2006-11-20 16:58:39.560925500   at
> > org.red5.server.stream.PlaylistSubscriberStream 
> $PlayEngine.pullAndPush
> > (PlaylistSubscriberStream.java:869)
> > 2006-11-20 16:58: 39.560937500   at
> > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play
> > (PlaylistSubscriberStream.java:703)
> > 2006-11-20 16:58:39.560940500   at
> > org.red5.server.stream.PlaylistSubscriberStream.play
> > (PlaylistSubscriberStream.java:127)
> > 2006-11-20 16:58:39.560943500   at
> > org.red5.server.stream.StreamService.play(StreamService.java:175)
> > 2006-11-20 16:58:39.560945500   at
> > org.red5.server.stream.StreamService.play (StreamService.java:183)
> > 2006-11-20 16:58:39.560960500   at
> > sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> > 2006-11-20 16:58:39.560962500   at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke
> > (DelegatingMethodAccessorImpl.java:25)
> > 2006-11-20 16:58:39.560965500   at java.lang.reflect.Method.invoke
> > (Method.java:585)
> > 2006-11-20 16:58:39.560967500   at
> > org.red5.server.service.ServiceInvoker.invoke  
> (ServiceInvoker.java:161)
> > 2006-11-20 16:58:39.560969500   at
> > org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java: 
> 287)
> > 2006-11-20 16:58:39.560978500   at
> > org.red5.server.net.rtmp.RTMPHandler.onInvoke (RTMPHandler.java:469)
> > 2006-11-20 16:58:39.560980500   at
> > org.red5.server.net.rtmp.RTMPHandler.messageReceived 
> (RTMPHandler.java:
> > 147)
> > 2006-11-20 16:58:39.560982500   at
> > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
> > (RTMPMinaIoHandler.java:78)
> > 2006-11-20 16:58:39.560985500   at
> > org.apache.mina.common.support.AbstractIoFilterChain 
> $2.messageReceived
> > (AbstractIoFilterChain.java:189)
> > 2006-11-20 16:58: 39.560993500   at
> >  
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRe
> > ce
> > ived(AbstractIoFilterChain.java:502)
> > 2006-11-20 16:58:39.560996500   at
> > org.apache.mina.common.support.AbstractIoFilterChain.access$1000
> > (AbstractIoFilterChain.java:52)
> > 2006-11-20 16:58:39.560998500   at
> > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > $1.messageReceived(AbstractIoFilterChain.java:777)
> > 2006-11-20 16:58: 39.561012500   at
> > org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java:
> > 231)
> > 2006-11-20 16:58:39.561015500   at org.red5.io.filter.ExecutorFilter
> > $ProcessEventsRunnable.run(ExecutorFilter.java :279)
> > 2006-11-20 16:58:39.561017500   at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > (ThreadPoolExecutor.java:650)
> > 2006-11-20 16:58:39.561019500   at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run
> > (ThreadPoolExecutor.java:675)
> > 2006-11-20 16:58:39.561028500   at java.lang.Thread.run(Thread.java:
> > 595)
> >
> >
> > On Nov 18, 2006, at 2:48 AM, Steven Gong wrote:
> >
> >> Hi Paul,
> >> I have just committed some codes to fix the OutOfMemoryError. The
> >> related revision numbers are 1555, 1556 and 1557.
> >>
> >> The OutOfMemoryError is mainly due to the buffer in FLVReader so I
> >> added a new static oroperty 'maxBufferSize' to constrain the buffer
> >> size and also added a way to disable the buffer if none of 'auto',
> >> 'direct' and 'heap' buffer type is configured. The property can
> >> also be configured in red5-common.xml.
> >>
> >> Another improvement is to let ICacheStore create ICacheable
> >> instances so that in the case of NoCacheImpl, no instances will be
> >> created thus we won't create byte array in this case.
> >>
> >> Could you please have a review? Thanks. :-)
> >>
> >> As the OutOfMemoryError bug gives rise to many complaints of our
> >> users, I also cc this mail to red5 list for notification purpose.
> >>
> >> --
> >> I cannot tell why this heart languishes in silence. It is for small
> >> needs it never asks, or knows or remembers.  -- Tagore
> >>
> >> Best Regards
> >> Steven Gong
> >> _______________________________________________
> >> Red5 mailing list
> >> Red5 at osflash.org
> >> http://osflash.org/mailman/listinfo/red5_osflash.org
> >
> >
> > _______________________________________________
> > Red5 mailing list
> > Red5 at osflash.org
> > http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
>
> -- 
> I cannot tell why this heart languishes in silence. It is for small  
> needs it never asks, or knows or remembers.  -- Tagore
>
> Best Regards
> Steven Gong
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20061121/8305b069/attachment.htm