[Red5] Please help --Error:Unexpected exceptionopening backend socket:java.net.SocketException:No bufferspace available (maximumconnections reached?):

Sat Feb 28 11:42:45 PST 2009

Well, that specific connection come from inside some Sony network, from inside a corporate environment.  I've also seen the same type of stuff from other locations... anyway, hopefully adjusting the KeepAliveTime will fix it.  Now I'm seeing some other new error in the log, which I'll start a new thread about. Stay tuned!  :)

Thanks,

Costin

----------------------------------------
> From: walter at waltertak.com
> To: red5 at osflash.org
> Date: Sat, 28 Feb 2009 00:21:50 +0100
> Subject: Re: [Red5] Please help --Error:Unexpected exceptionopening backend socket:java.net.SocketException:No bufferspace available (maximumconnections reached?):
>
> I'd block the IP in the firewall.
>
> Maybe someone is actively harrassing you or their proxy has gone beserk. You
> can spend hours or days trying to figure out what's happening or just block
> it and monitor the healthy clients.
>
> Btw I tried a traceroute to that IP and the route 'crashed' from 3 different
> locations (different data-centres) overhere,
> dunno if it's related but it's sure is odd.
>
>>
>> Hi Walter,
>>
>> Funny thing that you mention that article, because I did go ahead and set
>> the KeepAliveTime setting to 5 minutes yesterday. I might go as low as 1
>> minute in the future if I need to.
>>
>> As far as these rogue Flash clients, perhaps a firewall would help but
>> this is what the client is sending (I used a packet analyzer):
>>
>> HTTP: POST Request from Client
>> HTTP: Request Method =POST
>> HTTP: Uniform Resource Identifier =/idle/17185380/12124
>> HTTP: Protocol Version =HTTP/1.1
>> HTTP: Host =XXXXXXXXXX
>> HTTP: Connection =keep-alive
>> HTTP: Content-Length =1
>> HTTP: Content-Type =application/x-fcs
>> HTTP: User-Agent =Shockwave Flash
>> HTTP: Pragma =no-cache
>> HTTP: X-Forwarded-For = 43.135.131.97
>> HTTP: Via =1.0 NetCache1-sd (NetCache NetApp/6.0.7)
>>
>> Notice its a Post from a Flash client with an Idle, and then the server
>> just responds:
>>
>> HTTP: Response to Client; HTTP/1.1; Status Code = 200 - OK
>> HTTP: Protocol Version =HTTP/1.1
>> HTTP: Status Code = OK
>> HTTP: Reason =OK
>> HTTP: Server =Apache-Coyote/1.1
>> HTTP: Connection =Keep-Alive
>> HTTP: Cache-Control =no-cache
>> HTTP: Content-Type =application/x-fcs
>> HTTP: Content-Length =1
>> HTTP: Date =Fri, 27 Feb 2009 18:26:55 GMT
>>
>> However, that request from the client generates this error in the log
>> usually:
>>
>> INFO | jvm 1 | 2009/02/27 10:29:38 | [WARN] [http-80-exec-6]
>> org.red5.server.net.rtmpt.RTMPTServlet - Null connection for clientId:
>> 9325846
>> INFO | jvm 1 | 2009/02/27 10:29:39 | [WARN] [http-80-exec-1]
>> org.red5.server.net.rtmpt.RTMPTServlet - Null connection for clientId:
>> 9325846
>>
>> And I've modified the RTMPTServlet code to like so:
>>
>> protected void handleBadRequest(String message, HttpServletResponse resp)
>> throws IOException {
>>
>> /* THIS TELLS THE CLIENT TO KEEP SENDING MESSAGES
>> resp.setStatus(HttpServletResponse.SC_BAD_REQUEST);
>> resp.setContentType("text/plain");
>> resp.setContentLength(message.length());
>> resp.getWriter().write(message);
>> resp.flushBuffer();
>> */
>>
>>
>> // THIS SHOULD THE CLIENT TO GO AWAY AND CLOSE THE CONNECTION
>> resp.setStatus(HttpServletResponse.SC_BAD_REQUEST);
>> resp.setHeader("Connection", "close");
>> resp.setHeader("Cache-Control", "no-cache");
>> resp.setContentType(CONTENT_TYPE);
>> resp.setContentLength(1);
>> resp.getWriter().write((byte) 0);
>> resp.flushBuffer();
>> log.warn("disconnecting connection: " + message);
>> //returnMessage((byte) 0, resp);
>>
>> }
>>
>> BUT I'm seeing THOUSANDS of connections from that rogue Flash player, yet
>> only a handful of those "Null connection for clientId: 9325846" errors in
>> the log. And also, in the packet analyzer, I haven't seen my response
>> with the SC_BAD_REQUEST and Connection close command. So it seems like a
>> bunch of those requests from the rogue flash are handled by the
>> Apache/TomCat server before it ever even gets to the Red5 logic. Yet, it
>> still ties up the connections on the server because I can see thousands of
>> them with the "netstat -n -a" command.
>>
>> I hope this makes some sense and that we could ultimately solve this issue
>> of the rogue Flash player. One interesting aspect is this: HTTP: Via
>> =1.0 NetCache1-sd (NetCache NetApp/6.0.7) -- perhaps this is just being
>> forwarded by a proxy or something and there's something else to think
>> about there.
>>
>> Anyway, thats so much for your help!
>>
>> Costin
>>
>>
>>
>>
>> ----------------------------------------
>>> From: walter at waltertak.com
>>> To: red5 at osflash.org
>>> Date: Fri, 27 Feb 2009 12:59:30 +0100
>>> Subject: Re: [Red5] Please help -- Error:Unexpected exceptionopening
>>> backend socket:java.net.SocketException: No bufferspace available
>>> (maximumconnections reached?):
>>>
>>> Well if you stop Red5 and thus stop Java and you still have a lot of open
>>> connections then it's an OS problem for sure.
>>>
>>> That problem might be started by Red5 or Java but it really should be
>>> solved
>>> on OS level.
>>>
>>> I quickly found this link: http://support.microsoft.com/kb/Q324270
>>>
>>> In the 2nd half of the text:
>>>
>>> Value name: KeepAliveTime
>>> Key: Tcpip\Parameters
>>> Value Type: REG_DWORD-Time in milliseconds
>>> Valid Range: 1-0xFFFFFFFF
>>> Default: 7,200,000 (two hours)
>>>
>>> This value controls how frequently TCP tries to verify that an idle
>>> connection is still intact by sending a keep-alive packet. If the remote
>>> computer is still reachable, it acknowledges the keep-alive packet.
>>> Keep-alive packets are not sent by default. You can use a program to
>>> configure this value on a connection. The recommended value setting is
>>> 300,000 (5 minutes).
>>>
>>> Basically it's possible that a rogue client can occupy one port for a
>>> maximum of 2 hours, on a win 2003 server (see bottom for applicable
>>> servers)
>>> ; MS recommends to lower it to 5 minutes, you might even consider
>>> lowering
>>> it to say 2 minutes ; if a client cannot send a tcp/ip packet for 2
>>> minutes
>>> then it will be disconnected by Red5 anyway (Red5 has a simple 1 minute
>>> timeout system, unless you changed that in the config
>>> [red5.properties] ->
>>> lines:
>>>
>>> rtmp.ping_interval=5000
>>> rtmp.max_inactivity=60000
>>>
>>> Anyway ; lower the KeepAliveTime to say 3 minutes and have a look at a
>>> decent firewall that can limit the amount of concurrent connections from
>>> a
>>> single IP.
>>>
>>> You might want to limit the amount of maximum concurrent connections from
>>> one IP to your server anyway, as the current situation
>>> is actually a DoS (denial of service) at your end.
>>>
>>>>
>>>> Hi Walter,
>>>>
>>>> Here's more details on my predicament, following your format:
>>>>
>>>> The problem -- The server started generating exceptions and would not
>>>> accept any new connections. Even after restarting Red5 several times, it
>>>> was a no go. The OS had to be rebooted. Most likely all available
>>>> sockets were taken up.
>>>>
>>>> The version: 0.8 RC 2 - From the Installer
>>>>
>>>> Java Version 6 Update 11 (build 1.6.0_11-b03)
>>>>
>>>> OS - Windows 2003 R2 SP2 Standard Edition
>>>>
>>>> The problems seem to be in the server code -- sockets don't get
>>>> released.
>>>> After resetting the server, I haven't seen the issue, but another very
>>>> strange thing is happening when using RTMPT (which I MUST use).
>>>>
>>>> When calling "netstat -a -n -o" I keep seeing connections, THOUSANDS of
>>>> them, either in TIME_WAIT or ESTABLISHED all on port 80, all from the
>>>> same
>>>> IP address, but a different port. Everytime I restart the server,
>>>> there's
>>>> thousands more within a few minutes, from the same few addresses. When I
>>>> set the log to DEBUG, I see lots of these:
>>>>
>>>> INFO | jvm 1 | 2009/02/26 23:12:06 | [WARN] [http-80-exec-4]
>>>> org.red5.server.net.rtmpt.RTMPTServlet - Null connection for clientId:
>>>> 2954177
>>>> INFO | jvm 1 | 2009/02/26 23:12:07 | [WARN] [http-80-exec-2]
>>>> org.red5.server.net.rtmpt.RTMPTServlet - Null connection for clientId:
>>>> 2954177
>>>>
>>>> It seems that certain RTMPT clients (flash player) keep connecting to
>>>> the
>>>> server, and the server doesn't somehow respond in a way that makes the
>>>> client stop. Either that, or I'm getting attacked! I doubt the latter,
>>>> so for some reason, I keep getting hit. I believe that at some point
>>>> that
>>>> was a valid connection, and then I restarted the server, and the
>>>> clientId
>>>> became invalid, and now there's no stopping that client from just
>>>> sending
>>>> the IDLE command via RTMPT.
>>>>
>>>> Like I said, this is on a production server, so any help in the short
>>>> term
>>>> would be greatly appreciated. Should I try to upgrade to the latest
>>>> builds? Has any work been done on the RTMPT stuff?
>>>>
>>>> Thanks so much for your help,
>>>>
>>>> Costin
>>>>
>>>>
>>>>
>>>> From: walter at waltertak.com
>>>> To: red5 at osflash.org
>>>> Date: Fri, 27 Feb 2009 02:34:25 +0100
>>>> Subject: Re: [Red5] Please help -- Error: Unexpected exceptionopening
>>>> backend socket: java.net.SocketException: No bufferspace available
>>>> (maximum connections reached?):
>>>>
>>>>
>>>>
>>>>
>>>> I don't have a final answer to your question (would be a bit too easy
>>>> right) but found this post :
>>>> http://www.nabble.com/The-java.net.SocketException:-No-buffer-space-available-(maximum-connections-reached-):-listen-failed-Exception-td19269806.html
>>>>
>>>> Quote from the bottom:
>>>>
>>>> Re: The java.net.SocketException: No buffer space available (maximum
>>>> connections reached?): listen failed Exception
>>>>
>>>> by chitraa Sep 22, 2008; 08:46am :: Rate this Message: - Use ratings to
>>>> moderate (?)
>>>> Reply | Reply to Author | Print | View Threaded | Show Only this Message
>>>>
>>>> I don't seem to be getting this error anymore. From what I can gather
>>>> (for
>>>> the reference of others who have/are currently encountering this
>>>> problem),
>>>> there were two reasons why the 'No buffer space available' exception was
>>>> being thrown:
>>>>
>>>> 1. As explained by Daniel in his post, if an exception occurs while a
>>>> binded socket tries to make a connection to a machine, the socket does
>>>> not
>>>> get unbounded and closed. Try adding a java try catch block to ensure
>>>> all
>>>> sockets get unbounded and closed if an exception is thrown while either
>>>> i)
>>>> trying to establish a connection to the machine or ii) performing file
>>>> transfer.
>>>>
>>>> 2. A memory or file handle leak can also be the source of the problem.
>>>> When a program opens up too many handles rapidly, it exhausts the system
>>>> resources - hence throwing the "No Buffer Space" error. In my case, I
>>>> found that an external application was opening up too many handles. When
>>>> I
>>>> closed that process, the error did not appear. To check and see how many
>>>> handles the processes on your computer has opened up, look at your Task
>>>> Manager and click View -> Select Columns -> Handle Count.
>>>>
>>>> Thanks to everyone who took the time to reply to this post and give me
>>>> valuable suggestions. :)
>>>>
>>>> Regards,
>>>> Chitra
>>>>
>>>>
>>>>
>>>> What version of Red5 are you using ?
>>>> Are you able to upgrade to JRE 1.6.0_12 from your current 1.6.0_1
>>>> version
>>>> ?
>>>> Also see the posting "[Red5] Tips for posting requests for help on the
>>>> Red5mailinglist" of 26/2 for what you could provide us to provide the
>>>> best
>>>> remote help.
>>>>
>>>> Are you opening database-connections from your Application and if so ;
>>>> are
>>>> you sure you close them and are you sure you don't open one connection
>>>> per
>>>> connected user , for example ?
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: Costin Tuculescu
>>>> To: red5 at osflash.org
>>>> Sent: Friday, 27 February 2009 01:01
>>>> Subject: Re: [Red5] Please help -- Error: Unexpected exceptionopening
>>>> backend socket: java.net.SocketException: No bufferspace available
>>>> (maximum connections reached?):
>>>>
>>>> Hi Paul and Walter,
>>>>
>>>> I'm looking into the server networking settings (per
>>>> http://smallvoid.com/article/winnt-tcpip-max-limit.html), to release
>>>> TIME_WAITs faster, and to perhaps increase the max number of connections
>>>> available to the server. I also do see about 40 127.0.0.1 connections
>>>> that are owned by the Red5 (java) process, so its not a zillion, but
>>>> they
>>>> are there.
>>>>
>>>> Paul, since I'm a Java noob, would you mind just elaborating a bit on
>>>> how
>>>> to "try the non-NIO connector"? Where would I find this setting and what
>>>> are my options?
>>>>
>>>> Per Walter's suggestions earlier, as far as the help I need for this:
>>>> this
>>>> is a production box, and would be willing to pay or donate to have any
>>>> of
>>>> your Red5 experts work with me directly to solve this issue. Thank you
>>>> very much.
>>>>
>>>> -Costin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Date: Thu, 26 Feb 2009 14:16:11 -0800
>>>> From: mondain at gmail.com
>>>> To: red5 at osflash.org
>>>> Subject: Re: [Red5] Please help -- Error: Unexpected exception opening
>>>> backend socket: java.net.SocketException: No buffer space available
>>>> (maximum connections reached?):
>>>>
>>>> Try the non-NIO connector and also look into setting socket parameters
>>>> in
>>>> windows server, such as timeout and maximum connections.
>>>>
>>>>
>>>> Paul
>>>>
>>>>
>>>> On Thu, Feb 26, 2009 at 1:43 PM, Costin Tuculescu wrote:
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> Had a very bad error start showing up in my logs today. It started like
>>>> so:
>>>>
>>>> INFO | jvm 1 | 2009/02/26 11:53:28 | WrapperManager Error: Unexpected
>>>> exception opening backend socket: java.net.SocketException: No buffer
>>>> space available (maximum connections reached?): connect
>>>> ERROR | wrapperp | 2009/02/26 11:53:28 | server socket listen failed.
>>>> (10055)
>>>>
>>>> And I had several of those in a row, and then I started getting this:
>>>>
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | [WARN] [SocketAcceptor-0]
>>>> org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected
>>>> exception.
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | java.io.IOException: Unable to
>>>> establish loopback connection
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.PipeImpl$Initializer.run(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> java.security.AccessController.doPrivileged(Native Method) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at sun.nio.ch.PipeImpl.(Unknown
>>>> Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> java.nio.channels.Pipe.open(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.WindowsSelectorImpl.(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown Source)
>>>> [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> java.nio.channels.Selector.open(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker(SocketIoProcessor.java:93)
>>>> [mina-core-1.1.7.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew(SocketIoProcessor.java:82)
>>>> [mina-core-1.1.7.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSessions(SocketAcceptor.java:293)
>>>> [mina-core-1.1.7.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run(SocketAcceptor.java:225)
>>>> [mina-core-1.1.7.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
>>>> [mina-core-1.1.7.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>>>> [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>> [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at java.lang.Thread.run(Unknown
>>>> Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | Caused by:
>>>> java.net.SocketException: No buffer space available (maximum connections
>>>> reached?): listen
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.listen(Native Method) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | ... 16 common frames omitted
>>>> INFO | jvm 1 | 2009/02/26 11:53:36 | WrapperManager Error: Unexpected
>>>> exception opening backend socket: java.net.SocketException: No buffer
>>>> space available (maximum connections reached?): connect
>>>> ERROR | wrapperp | 2009/02/26 11:53:36 | server socket listen failed.
>>>> (10055)
>>>> ERROR | wrapperp | 2009/02/26 11:53:36 | server socket listen failed.
>>>> (10055)
>>>>
>>>> So then I stopped the Red5 service and tried it again and I only kept
>>>> getting those types of errors. It seems like the OS had reached the max
>>>> number of connections and could not create anymore sockets. Here are the
>>>> errors after the restart:
>>>>
>>>> ERROR | wrapperp | 2009/02/26 12:13:07 | server socket listen failed.
>>>> (10055)
>>>> ERROR | wrapperp | 2009/02/26 12:13:07 | server socket listen failed.
>>>> (10055)
>>>> ERROR | wrapperp | 2009/02/26 12:13:07 | server socket listen failed.
>>>> (10055)
>>>> ERROR | wrapperp | 2009/02/26 12:13:07 | server socket listen failed.
>>>> (10055)
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | [ERROR] [http-80-Acceptor-6]
>>>> org.apache.tomcat.util.net.NioEndpoint - Socket accept failed
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | java.io.IOException: An
>>>> operation on a socket could not be performed because the system lacked
>>>> sufficient buffer space or because a queue was full
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> [tomcat-coyote-6.0.18.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at java.lang.Thread.run(Unknown
>>>> Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | [ERROR] [http-80-Acceptor-7]
>>>> org.apache.tomcat.util.net.NioEndpoint - Socket accept failed
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | java.io.IOException: An
>>>> operation on a socket could not be performed because the system lacked
>>>> sufficient buffer space or because a queue was full
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source) [na:1.6.0_01]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> [tomcat-coyote-6.0.18.jar:na]
>>>> INFO | jvm 1 | 2009/02/26 12:13:07 | at java.lang.Thread.run(Unknown
>>>> Source) [na:1.6.0_01]
>>>>
>>>> After a few restarts of the service, I kept getting the same error.
>>>> Ultimately, I had to reboot the machine, which is a production
>>>> service --
>>>> not good at all.
>>>>
>>>> According to my log, I had about 10000 connections and disconnections
>>>> within a 12 hour period.
>>>>
>>>> I'm running Windows Server 2003 R2 SP2 with 1024MB for the Heap Size.
>>>> Please let me know what other information I can provide to help diagnose
>>>> and solve this problem. Thank you very much for the help.
>>>>
>>>> Best Regards,
>>>>
>>>> Costin
>
>
> _______________________________________________
> Red5 mailing list
> Red5 at osflash.org
> http://osflash.org/mailman/listinfo/red5_osflash.org
_________________________________________________________________
Windows Live™ Contacts: Organize your contact list. 
http://windowslive.com/connect/post/marcusatmicrosoft.spaces.live.com-Blog-cns!503D1D86EBB2B53C!2285.entry?ocid=TXT_TAGLM_WL_UGC_Contacts_032009