[Red5] crash with Red5 when trying to play 100mb mp3

Tue Aug 12 07:30:26 PDT 2008

Hi,

was wondering if the error below is familiar to someone.

I'm running Red5 trunk v2814 on a small server (only 384 mb memory) and while trying to playback a 100 Mb mp3 file (2+ hours 192 kbit) R5 crashes after requesting the specific mp3 file.

Here are the exact sizes:

-rwxrw-r-- 1 red5 red5 106063872 Feb  2  2008 armin1.mp3
-rw-r--r-- 1 root root   9423403 Aug 12 15:55 armin1.mp3.meta

The metadata-file contains 169.179 lines.

I have a feeling that Red5 tries to load the meta-file (only 10 mb) and chokes on the sheer amount of keyframes in it.

When I'm requesting the file the memory usage goes from 44mb to 104mb (60 mb increase) and cpu usages skyrockets to > 90% for at least 20 seconds.

Then the following crash-log is generated.

After the logging of the event the server continues running, playing other (small) video-FLVs work fine, cpu-usage drops to normal levels (near zero when nothing is being requested).

After playing that normal video I requested the 100mb mp3 again and the exact behaviour repeated , though the memory usage did stay at the same level (~ 104mb)

To see if it was the OS that was being unfriendly about the heap-space problem I added these commandline-options to Red5:

-Xms32M -Xmx32M -Xss128K

and force Red5 to use less memory.

In this scenario the memory usage would not rise to around ~104 Mb but to ~ 72 Mb .. However the same crash occurs (and the cpu spike for like 5-10 seconds).

My concern is this:

If I would serve large mp3-files (think of 2 hour recordings of live-events) could a few users take down a server with 2+ Gb installed ? If each large mp3 eats 50 or more megabyte due to the very large meta-file then only 40 different long mp3s would consume possible 2 Gb of memory. Needless to say that would take a few hours to happen on a live server.

You can find a syntax-highlighted version of the error-log on this link: http://pastebin.com/m48b0df3

-----

2008-08-12 16:11:28,586 [pool-3-thread-1] ERROR o.r.s.m.InMemoryPullPullPipe - exception when pulling message from provider
java.lang.OutOfMemoryError: Java heap space
        at com.sun.org.apache.xml.internal.utils.NodeVector.addElement(NodeVector.java:128)
        at com.sun.org.apache.xpath.internal.axes.NodeSequence.nextNode(NodeSequence.java:295)
        at com.sun.org.apache.xpath.internal.axes.NodeSequence.runTo(NodeSequence.java:442)
        at com.sun.org.apache.xml.internal.dtm.ref.DTMNodeList.<init>(DTMNodeList.java:79)
        at com.sun.org.apache.xpath.internal.objects.XNodeSet.nodelist(XNodeSet.java:339)
        at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.getResultAsType(XPathExpressionImpl.java:353)
        at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.eval(XPathExpressionImpl.java:99)
        at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:180)
        at org.red5.io.FileKeyFrameMetaCache.loadKeyFrameMeta(FileKeyFrameMetaCache.java:111)
        at org.red5.io.CachingFileKeyFrameMetaCache.loadKeyFrameMeta(CachingFileKeyFrameMetaCache.java:35)
        at org.red5.io.mp3.impl.MP3Reader.analyzeKeyFrames(MP3Reader.java:440)
        at org.red5.io.mp3.impl.MP3Reader.<init>(MP3Reader.java:140)
        at org.red5.io.mp3.impl.MP3.getReader(MP3.java:48)
        at org.red5.server.stream.provider.FileProvider.init(FileProvider.java:225)
        at org.red5.server.stream.provider.FileProvider.pullMessage(FileProvider.java:124)
        at org.red5.server.messaging.InMemoryPullPullPipe.pullMessage(InMemoryPullPullPipe.java:74)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1492)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access$400(PlaylistSubscriberStream.java:827)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine$PullAndPushRunnable.run(PlaylistSubscriberStream.java:2126)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
2008-08-12 16:11:28,588 [pool-3-thread-1] ERROR o.r.s.s.PlaylistSubscriberStream - error notify streamPlaylistItemStop
java.lang.NullPointerException
        at org.red5.server.adapter.MultiThreadedApplicationAdapter.streamPlaylistItemStop(MultiThreadedApplicationAdapter.java:1102)
        at org.red5.server.stream.PlaylistSubscriberStream.notifyItemStop(PlaylistSubscriberStream.java:723)
        at org.red5.server.stream.PlaylistSubscriberStream.access$1700(PlaylistSubscriberStream.java:81)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.stop(PlaylistSubscriberStream.java:1347)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1495)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access$400(PlaylistSubscriberStream.java:827)
        at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine$PullAndPushRunnable.run(PlaylistSubscriberStream.java:2126)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20080812/5e335ff3/attachment-0001.html 