[Red5] server hanging
Steven Gong
steven.gong at gmail.com
Sun Apr 29 00:34:59 EDT 2007
It's easy to make a workaround for this in okayToSendMessage() to ignore the
null pointer but I think we'd better find the root cause this issue as this
should be happen in normal cases.
On 4/29/07, Dan Rossi <spam at electroteque.org> wrote:
>
> OK it did it again, hanged without warnings at all this time until i
> tried to stop the service and then displayed JVM dumps, logged via the
> java service wrapper not red5. Ive put red5 logging into fatal to see if
> that helps ?
>
> Steven Gong wrote:
> > Will this bug hang the entire server? I guess this should be a
> > contention that the pengingMessage has been released before it's
> > pushed to the consumer. So the body of pendingMessage is null and
> > causes that exception.
> >
> > Dan, if possible, please record this as a tracker so that this won't
> > be forgotten. Thanks.
> >
> > On 4/29/07, *Dan Rossi* < spam at electroteque.org
> > <mailto:spam at electroteque.org>> wrote:
> >
> > I already have for the exception errors should i duplicate the
> > report ?
> >
> > Thijs Triemstra | Collab wrote:
> > > Please create a ticket on JIRA..
> > >
> > > cheers,
> > >
> > > Thijs
> > >
> > > On Apr 29, 2007, at 2:43 AM, Dan Rossi wrote:
> > >
> > >
> > >> Hi this is a bizarre one but after quite a while having the
> server
> > >> up it
> > >> will hang without any outofmemory error blocking connections it
> > could
> > >> because of these kinds of exceptions
> > >>
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | [ERROR] 745875
> > >> pool-3-thread-2:
> > >> ( org.red5.server.stream.PlaylistSubscriberStream.error
> > >> ) Error while pulling message.
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 |
> > >> java.lang.NullPointerException
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream
> > >> $PlayEngine.okayToSendMessage(PlaylistSubscriberStream.java:1241)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream
> > >> $PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1301)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access
> > >> $100(PlaylistSubscriberStream.java:682)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream.written
> > >> (PlaylistSubscriberStream.java :535)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageSent
> > >> (BaseRTMPHandler.java:205)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageSent
> > >> (RTMPMinaIoHandler.java:169)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain
> > >> $TailFilter.messageSent(AbstractIoFilterChain.java:716)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >>
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSe
> > >> nt( AbstractIoFilterChain.java:391)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain.access$1300
> > >> (AbstractIoFilterChain.java:54)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > >> $1.messageSent(AbstractIoFilterChain.java:806)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.io.filter.ExecutorFilter.processEvent
> > (ExecutorFilter.java:
> > >> 358)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > >> (ExecutorFilter.java:416)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> > Source)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.lang.Thread.run(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | [ERROR] 745875
> > >> pool-3-thread-2:
> > >> ( org.red5.server.stream.PlaylistSubscriberStream.error
> > >> ) Error while pulling message.
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 |
> > java.lang.RuntimeException:
> > >> expected IStreamData but got class
> > >> org.red5.server.net.rtmp.event.Unknown (type 91)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream
> > >> $PlayEngine.okayToSendMessage( PlaylistSubscriberStream.java
> :1203)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream
> > >> $PlayEngine.pullAndPush(PlaylistSubscriberStream.java :1301)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.access
> > >> $100(PlaylistSubscriberStream.java:682)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.stream.PlaylistSubscriberStream.written
> > >> (PlaylistSubscriberStream.java:535)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageSent
> > >> (BaseRTMPHandler.java:205)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageSent
> > >> (RTMPMinaIoHandler.java:169)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain
> > >> $TailFilter.messageSent(AbstractIoFilterChain.java:716)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >>
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSe
> >
> > >> nt(AbstractIoFilterChain.java:391)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain.access$1300
> > >> (AbstractIoFilterChain.java :54)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > >> $1.messageSent(AbstractIoFilterChain.java:806)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.io.filter.ExecutorFilter.processEvent(
> ExecutorFilter.java:
> > >> 358)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > >> (ExecutorFilter.java:416)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> > Source)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >> INFO | jvm 1 | 2007/04/28 13:56:36 | at
> > >> java.lang.Thread.run(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 14:01:52 | [ERROR] 1062437
> > >> DefaultQuartzScheduler_Worker-8:(
> > >> org.red5.server.scheduling.QuartzSchedulingService.error ) Job
> > >> org.red5.server.net.rtmp.RTMPConnection$KeepAliveJob at d0570e
> > execution
> > >> failed: null
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | [ERROR] 1083125
> > >> pool-3-thread-6:( org.red5.server.service.ServiceInvoker.error
> > ) Error
> > >> executing call: Service: null Method: closeStream Num Params: 0
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | [ERROR] 1083141
> > >> pool-3-thread-6:( org.red5.server.service.ServiceInvoker.error )
> > >> Service
> > >> invocation error
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 |
> > >> java.lang.reflect.InvocationTargetException
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> java.lang.reflect.Method.invoke(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >>
> > org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java
> :181)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >>
> > org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:174)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java
> :355)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived
> > >> (BaseRTMPHandler.java:126)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
> > >> (RTMPMinaIoHandler.java:116)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain
> > >> $TailFilter.messageReceived(AbstractIoFilterChain.java:703)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >>
> >
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRe
> >
> > >> ceived(AbstractIoFilterChain.java:362)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain.access$1200
> > >> (AbstractIoFilterChain.java :54)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> > >> $1.messageReceived(AbstractIoFilterChain.java:800)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.red5.io.filter.ExecutorFilter.processEvent(
> ExecutorFilter.java:
> > >> 355)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run
> > >> (ExecutorFilter.java:416)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> > Source)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >> INFO | jvm 1 | 2007/04/28 14:02:13 | at
> > >> java.lang.Thread.run(Unknown Source)
> > >>
> > >>
> > >>
> > >> and when i try and stop the windows service I get errors like
> these
> > >>
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | "pool-2-thread-99"
> prio=6
> > >> tid=0x6e6fe400 nid=0xe3c waiting on condition
> > [0x7077f000..0x7077fa98]
> > >> INFO | jvm 3 | 2007/04/28 20:30:25
> > | java.lang.Thread.State:
> > >> WAITING (parking)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> sun.misc.Unsafe.park(Native Method)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | - parking to wait
> for
> > >> <0x09de5318> (a
> > java.util.concurrent.SynchronousQueue$TransferStack)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill
> > >> (Unknown
> > >> Source)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.SynchronousQueue$TransferStack.transfer
> > >> (Unknown Source)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.SynchronousQueue.take(Unknown Source)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> > >> INFO | jvm 3 | 2007/04/28 20:30:25 | at
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >>
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | JNI global
> > references: 640
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | Heap
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | def new generation
> > total
> > >> 72768K, used 49874K [0x02690000, 0x07580000, 0x09cb0000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | eden space
> 64704K, 65%
> > >> used
> > >> [0x02690000, 0x04ff8020, 0x065c0000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | from space
> 8064K, 92%
> > >> used
> > >> [0x06da0000, 0x074ec890, 0x07580000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | to space 8064K,
> 0%
> > >> used
> > >> [0x065c0000, 0x065c0000, 0x06da0000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | tenured generation
> > total
> > >> 967936K, used 29501K [0x09cb0000, 0x44df0000, 0x62690000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | the space
> > 967936K, 3%
> > >> used [0x09cb0000, 0x0b97f798, 0x0b97f800, 0x44df0000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | compacting perm
> > gen total
> > >> 12800K, used 12747K [0x62690000, 0x63310000, 0x66690000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | the space
> > 12800K, 99%
> > >> used
> > >> [0x62690000, 0x63302f20, 0x63303000, 0x63310000)
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 | No shared spaces
> > configured.
> > >> INFO | jvm 3 | 2007/04/28 20:30:26 |
> > >> ERROR | wrapper | 2007/04/28 20:30:29 | JVM did not exit on
> > request,
> > >> terminated
> > >>
> > >>
> > >> and another server gave us this but only when trying to stop the
> > >> service
> > >>
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | Server daemon died!
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 |
> > java.lang.OutOfMemoryError:
> > >> unable to create new native thread
> > >> STATUS | wrapper | 2007/04/28 20:38:50 | Filter trigger matched.
> > >> Restarting JVM.
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> java.lang.Thread.start0(Native Method)
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> java.lang.Thread.start(Unknown Source)
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner
> > >> (WrapperManager.java:3152)
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> org.tanukisoftware.wrapper.WrapperManager.handleSocket
> > >> (WrapperManager.java:3797)
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> org.tanukisoftware.wrapper.WrapperManager.run (
> WrapperManager.java:
> > >> 4084)
> > >> INFO | jvm 1 | 2007/04/28 20:38:50 | at
> > >> java.lang.Thread.run(Unknown Source)
> > >> ERROR | wrapper | 2007/04/28 20:39:27 | Shutdown failed:
> > Timed out
> > >> waiting for signal from JVM.
> > >> STATUS | wrapper | 2007/04/28 20:39:27 | Dumping JVM state.
> > >> ERROR | wrapper | 2007/04/28 20:39:31 | JVM did not exit on
> > request,
> > >> terminated
> > >>
> > >> _______________________________________________
> > >> Red5 mailing list
> > >> Red5 at osflash.org <mailto:Red5 at osflash.org>
> > >> http://osflash.org/mailman/listinfo/red5_osflash.org
> > >>
> > >
> > >
> > > _______________________________________________
> > > Red5 mailing list
> > > Red5 at osflash.org <mailto:Red5 at osflash.org>
> > > http://osflash.org/mailman/listinfo/red5_osflash.org
> > >
> > >
> >
> >
> > _______________________________________________
> > Red5 mailing list
> > Red5 at osflash.org <mailto:Red5 at osflash.org>
> > http://osflash.org/mailman/listinfo/red5_osflash.org
> >
> >
> >
> >
> > --
> > I cannot tell why this heart languishes in silence. It is for small
> > needs it never asks, or knows or remembers. -- Tagore
> >
> > Best Regards
> > Steven Gong
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Red5 mailing list
> > Red5 at osflash.org
> > http://osflash.org/mailman/listinfo/red5_osflash.org
> >
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
>
--
I cannot tell why this heart languishes in silence. It is for small needs it
never asks, or knows or remembers. -- Tagore
Best Regards
Steven Gong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://osflash.org/pipermail/red5_osflash.org/attachments/20070429/dd2b2dfb/attachment-0001.htm
More information about the Red5
mailing list