[Red5] server hanging
Steven Gong
steven.gong at gmail.com
Sun Apr 29 01:14:30 EDT 2007
Don't know if it helps. Try to add a line:
if (message == null) return false;
at the beginning of okayToSendMessage() in PlaylistSubscriberStream.
On 4/29/07, Dan Rossi <spam at electroteque.org> wrote:
>
> OK, which file / line ?
>
> Steven Gong wrote:
> > It's easy to make a workaround for this in okayToSendMessage() to
> > ignore the null pointer but I think we'd better find the root cause
> > this issue as this should be happen in normal cases.
> >
> > On 4/29/07, *Dan Rossi* <spam at electroteque.org
> > <mailto:spam at electroteque.org>> wrote:
> >
> > OK it did it again, hanged without warnings at all this time until i
> > tried to stop the service and then displayed JVM dumps, logged via
> the
> > java service wrapper not red5. Ive put red5 logging into fatal to
> > see if
> > that helps ?
> >
> > Steven Gong wrote:
> > > Will this bug hang the entire server? I guess this should be a
> > > contention that the pengingMessage has been released before it's
> > > pushed to the consumer. So the body of pendingMessage is null and
> > > causes that exception.
> > >
> > > Dan, if possible, please record this as a tracker so that this
> won't
> > > be forgotten. Thanks.
> > >
> > > On 4/29/07, *Dan Rossi* < spam at electroteque.org
> > <mailto:spam at electroteque.org>
> > > <mailto:spam at electroteque.org <mailto:spam at electroteque.org>>>
> > wrote:
> > >
> > > I already have for the exception errors should i duplicate the
> > > report ?
> > >
> > > Thijs Triemstra | Collab wrote:
> > > > Please create a ticket on JIRA..
> > > >
> > > > cheers,
> > > >
> > > > Thijs
> > > >
> > > > On Apr 29, 2007, at 2:43 AM, Dan Rossi wrote:
> > > >
> > > >
> > > >> Hi this is a bizarre one but after quite a while having
> > the server
> > > >> up it
> > > >> will hang without any outofmemory error blocking
> > connections it
> > > could
> > > >> because of these kinds of exceptions
> > > >>
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | [ERROR] 745875
> > > >> pool-3-thread-2:
> > > >> ( org.red5.server.stream.PlaylistSubscriberStream.error
> > > >> ) Error while pulling message.
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 |
> > > >> java.lang.NullPointerException
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream
> > > >>
> > $PlayEngine.okayToSendMessage(PlaylistSubscriberStream.java:1241)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream
> > > >> $PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1301)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access
> > > >> $100(PlaylistSubscriberStream.java:682)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream.written
> > > >> (PlaylistSubscriberStream.java :535)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageSent
> > > >> (BaseRTMPHandler.java:205)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageSent
> > > >> (RTMPMinaIoHandler.java:169)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.apache.mina.common.support.AbstractIoFilterChain
> > > >> $TailFilter.messageSent( AbstractIoFilterChain.java:716)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > >
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSe
> > > >> nt( AbstractIoFilterChain.java:391)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain.access$1300
> > > >> (AbstractIoFilterChain.java :54)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > > >> $1.messageSent(AbstractIoFilterChain.java :806)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.io.filter.ExecutorFilter.processEvent
> > > (ExecutorFilter.java:
> > > >> 358)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > > >> (ExecutorFilter.java:416)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > (Unknown
> > > Source)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> java.lang.Thread.run(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | [ERROR] 745875
> > > >> pool-3-thread-2:
> > > >> ( org.red5.server.stream.PlaylistSubscriberStream.error
> > > >> ) Error while pulling message.
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 |
> > > java.lang.RuntimeException:
> > > >> expected IStreamData but got class
> > > >> org.red5.server.net.rtmp.event.Unknown (type 91)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream
> > > >> $PlayEngine.okayToSendMessage(
> > PlaylistSubscriberStream.java:1203)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream
> > > >> $PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1301)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access
> > > >> $100(PlaylistSubscriberStream.java:682)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.stream.PlaylistSubscriberStream.written
> > > >> (PlaylistSubscriberStream.java:535)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageSent
> > > >> (BaseRTMPHandler.java:205)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageSent
> > > >> (RTMPMinaIoHandler.java:169)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.apache.mina.common.support.AbstractIoFilterChain
> > > >> $TailFilter.messageSent( AbstractIoFilterChain.java:716)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > >
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSe
> > >
> > > >> nt( AbstractIoFilterChain.java:391)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain.access$1300
> > > >> (AbstractIoFilterChain.java :54)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > > >> $1.messageSent(AbstractIoFilterChain.java :806)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java:
> > > >> 358)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > > >> (ExecutorFilter.java:416)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> > (Unknown
> > > Source)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >>
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > > >> java.lang.Thread.run(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:01:52 | [ERROR] 1062437
> > > >> DefaultQuartzScheduler_Worker-8:(
> > > >> org.red5.server.scheduling.QuartzSchedulingService.error
> > ) Job
> > > >> org.red5.server.net.rtmp.RTMPConnection$KeepAliveJob at d0570e
> > > execution
> > > >> failed: null
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | [ERROR] 1083125
> > > >> pool-3-thread-6:(
> > org.red5.server.service.ServiceInvoker.error
> > > ) Error
> > > >> executing call: Service: null Method: closeStream Num
> > Params: 0
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | [ERROR] 1083141
> > > >> pool-3-thread-6:(
> > org.red5.server.service.ServiceInvoker.error )
> > > >> Service
> > > >> invocation error
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 |
> > > >> java.lang.reflect.InvocationTargetException
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> sun.reflect.GeneratedMethodAccessor21.invoke(Unknown
> Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> java.lang.reflect.Method.invoke(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > >
> > org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java
> :181)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > >
> > org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:174)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> org.red5.server.net.rtmp.RTMPHandler.onInvoke
> > (RTMPHandler.java:355)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived
> > > >> (BaseRTMPHandler.java:126)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
> > > >> (RTMPMinaIoHandler.java:116)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> org.apache.mina.common.support.AbstractIoFilterChain
> > > >> $TailFilter.messageReceived(AbstractIoFilterChain.java:703)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > >
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRe
> > >
> > > >> ceived(AbstractIoFilterChain.java:362)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain.access$1200
> > > >> (AbstractIoFilterChain.java :54)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > > >> $1.messageReceived(AbstractIoFilterChain.java:800)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java:
> > > >> 355)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > > >> (ExecutorFilter.java:416)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> > > Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >>
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > > >> java.lang.Thread.run(Unknown Source)
> > > >>
> > > >>
> > > >>
> > > >> and when i try and stop the windows service I get errors
> > like these
> > > >>
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 |
> > "pool-2-thread-99" prio=6
> > > >> tid=0x6e6fe400 nid=0xe3c waiting on condition
> > > [0x7077f000..0x7077fa98]
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25
> > > | java.lang.Thread.State:
> > > >> WAITING (parking)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >> sun.misc.Unsafe.park(Native Method)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | - parking
> > to wait for
> > > >> <0x09de5318> (a
> > > java.util.concurrent.SynchronousQueue$TransferStack)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >> java.util.concurrent.locks.LockSupport.park (Unknown
> Source)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >>
> > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > > >> (Unknown
> > > >> Source)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > > >> (Unknown Source)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >> java.util.concurrent.SynchronousQueue.take(Unknown Source)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >> java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
> > Source)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > > >>
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > > >>
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | JNI global
> > > references: 640
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | Heap
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | def new
> generation
> > > total
> > > >> 72768K, used 49874K [0x02690000, 0x07580000, 0x09cb0000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | eden space
> > 64704K, 65%
> > > >> used
> > > >> [0x02690000, 0x04ff8020, 0x065c0000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | from space
> > 8064K, 92%
> > > >> used
> > > >> [0x06da0000, 0x074ec890, 0x07580000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | to space
> > 8064K, 0%
> > > >> used
> > > >> [0x065c0000, 0x065c0000, 0x06da0000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | tenured
> > generation
> > > total
> > > >> 967936K, used 29501K [0x09cb0000, 0x44df0000, 0x62690000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | the space
> > > 967936K, 3%
> > > >> used [0x09cb0000, 0x0b97f798, 0x0b97f800, 0x44df0000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | compacting perm
> > > gen total
> > > >> 12800K, used 12747K [0x62690000, 0x63310000, 0x66690000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | the space
> > > 12800K, 99%
> > > >> used
> > > >> [0x62690000, 0x63302f20, 0x63303000, 0x63310000)
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 | No shared spaces
> > > configured.
> > > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > > >> ERROR | wrapper | 2007/04/28 20:30:29 | JVM did not exit
> on
> > > request,
> > > >> terminated
> > > >>
> > > >>
> > > >> and another server gave us this but only when trying to
> > stop the
> > > >> service
> > > >>
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | Server daemon
> > died!
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 |
> > > java.lang.OutOfMemoryError:
> > > >> unable to create new native thread
> > > >> STATUS | wrapper | 2007/04/28 20:38:50 | Filter trigger
> > matched.
> > > >> Restarting JVM.
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >> java.lang.Thread.start0(Native Method)
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >> java.lang.Thread.start(Unknown Source)
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >>
> org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner
> > > >> ( WrapperManager.java:3152)
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >> org.tanukisoftware.wrapper.WrapperManager.handleSocket
> > > >> (WrapperManager.java:3797)
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >> org.tanukisoftware.wrapper.WrapperManager.run
> > (WrapperManager.java:
> > > >> 4084)
> > > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > > >> java.lang.Thread.run(Unknown Source)
> > > >> ERROR | wrapper | 2007/04/28 20:39:27 | Shutdown failed:
> > > Timed out
> > > >> waiting for signal from JVM.
> > > >> STATUS | wrapper | 2007/04/28 20:39:27 | Dumping JVM
> state.
> > > >> ERROR | wrapper | 2007/04/28 20:39:31 | JVM did not exit
> on
> > > request,
> > > >> terminated
> > > >>
> > > >> _______________________________________________
> > > >> Red5 mailing list
> > > >> Red5 at osflash.org <mailto:Red5 at osflash.org>
> > <mailto:Red5 at osflash.org <mailto:Red5 at osflash.org>>
> > > >> http://osflash.org/mailman/listinfo/red5_osflash.org
> > > >>
> > > >
> > > >
> > > > _______________________________________________
> > > > Red5 mailing list
> > > > Red5 at osflash.org <mailto:Red5 at osflash.org>
> > <mailto:Red5 at osflash.org <mailto:Red5 at osflash.org>>
> > > > http://osflash.org/mailman/listinfo/red5_osflash.org
> > <http://osflash.org/mailman/listinfo/red5_osflash.org>
> > > >
> > > >
> > >
> > >
> > > _______________________________________________
> > > Red5 mailing list
> > > Red5 at osflash.org <mailto:Red5 at osflash.org> <mailto:
> > Red5 at osflash.org <mailto:Red5 at osflash.org>>
> > > http://osflash.org/mailman/listinfo/red5_osflash.org
> > >
> > >
> > >
> > >
> > > --
> > > I cannot tell why this heart languishes in silence. It is for
> small
> > > needs it never asks, or knows or remembers. -- Tagore
> > >
> > > Best Regards
> > > Steven Gong
> > >
> >
> ------------------------------------------------------------------------
> >
> > >
> > > _______________________________________________
> > > Red5 mailing list
> > > Red5 at osflash.org <mailto:Red5 at osflash.org>
> > > http://osflash.org/mailman/listinfo/red5_osflash.org
> > >
> >
> >
> > _______________________________________________
> > Red5 mailing list
> > Red5 at osflash.org <mailto:Red5 at osflash.org>
> > http://osflash.org/mailman/listinfo/red5_osflash.org
> >
> >
> >
> >
> > --
> > I cannot tell why this heart languishes in silence. It is for small
> > needs it never asks, or knows or remembers. -- Tagore
> >
> > Best Regards
> > Steven Gong
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Red5 mailing list
> > Red5 at osflash.org
> > http://osflash.org/mailman/listinfo/red5_osflash.org
> >
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
--
I cannot tell why this heart languishes in silence. It is for small needs it
never asks, or knows or remembers. -- Tagore
Best Regards
Steven Gong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20070429/29b3c087/attachment-0001.htm
More information about the Red5
mailing list