[Red5] Memory Leak - Test Case
Tyler Kocheran
rfkrocktk at gmail.com
Thu Jun 11 13:55:53 PDT 2009
Locating the conf/ehcache.xml file turns out to be a real nightmare. This is
the latest patch, which tries three different ways of getting access to the
CacheManager.
1. An instance of CacheManager, loading the "conf/ehcache.xml" as a
configuration source.
2. The singleton CacheManager, loading the "conf/ehcache.xml"
3. A default singleton CacheManager.
Take a look at the code, and if you have a better way of doing it, let me
know. Attached is the latest patch.
(The problem in getting the CacheManager is in the different ways that the
server can be started. If started regularly from a script, Red5's root is
the actual root of the distribution's directory. If started from Ant using
'ant server', the root is set to the root of the project: red5_server/ )
On Thu, Jun 11, 2009 at 1:27 PM, Tyler Kocheran <rfkrocktk at gmail.com> wrote:
> Yeah, I've already done that, It's passing fine :)
> I'm determined to get a non-singleton CacheManager, I'm going to see if
> there's any way I can make it happen.
>
> - TK
>
>
> On Thu, Jun 11, 2009 at 1:24 PM, Art Clarke <aclarke at xuggle.com> wrote:
>
>> Tyler, can you run the Red5 test suite on it. That will actually test
>> AMF0 for you.
>>
>> - Art
>>
>> On Thu, Jun 11, 2009 at 10:44 AM, Tyler Kocheran <rfkrocktk at gmail.com>wrote:
>>
>>> Since JIRA's down, I'll have to post this here. Paul, maybe you could
>>> help out a little here: do you think that refactoring the string cache (as
>>> well as the other caches in the Output classes) to use an EhCache Cache
>>> instance rather than a ConcurrentMap is a good idea? If you think it's a
>>> good move, I could definitely implement it.
>>>
>>> However, the cache would have to be moved from being a static member to
>>> an instance member, so it could be injected. I don't know how/if the Output
>>> classes are injected ever or not, but if you have any suggestions, Paul,
>>> they'd be welcomed!
>>>
>>> Thomas, yes, you can modify your app to get rid of caching for the moment
>>> until we get this fixed. For the org.red5.io.amf.Output class, comment out
>>> line 505, and replace line 500 with "byte[] encoded = null;" Then, modify
>>> org.red5.io.amf3.Output, changing line 145 to "byte[] encoded = null;" and
>>> commenting out line 150.
>>>
>>> On Thu, Jun 11, 2009 at 9:43 AM, Tyler Kocheran <rfkrocktk at gmail.com>wrote:
>>>
>>>> Well, what appears to be happening is this.
>>>> You're using AMF0, since the bug seems to occur in the
>>>> org.red5.io.amf.Output<http://code.google.com/p/red5/source/browse/java/server/trunk/src/org/red5/io/amf/Output.java>class and not the
>>>> org.red5.io.amf3.Output<http://code.google.com/p/red5/source/browse/java/server/trunk/src/org/red5/io/amf3/Output.java>class. However, on closer look, this bug may be affecting AMF3 strings as
>>>> well, since the AMF3 Output extends the AMF0 Output. It seems that for a
>>>> performance gain, a static final ConcurrentMap<String, byte[]> is held in
>>>> memory, caching all Strings and their byte[] representations to make future
>>>> acesses faster.
>>>>
>>>> Unfortunately, this cache appears to be unbounded, so if you've got a
>>>> long-running server (which is something we all hope for ;] ), you've got a
>>>> huge problem if you're using any remoting in your application, since all
>>>> Strings and their byte[] representations are held *in memory
>>>> indefinitely*. Just as in your testing example, it would be very easy
>>>> for a rogue client to attack your server's RAM by using something as simple
>>>> as the echo service, just passing random strings to RMI calls on an infinite
>>>> loop. This is fixable, of course, we could probably fix it using EhCache
>>>> which is already included in the trunk. The problem with Maps is that
>>>> they're not so easy to set bounds for.
>>>>
>>>> EhCache, on the other hand, is extremely configurable, even clusterable,
>>>> and can even overflow to disk and optionally create an eternal (persistent
>>>> between server restarts) cache. I'm going to look into implementing it into
>>>> the Output classes.
>>>>
>>>> On Thu, Jun 11, 2009 at 7:47 AM, Thomas Auge <auge at virtues.net> wrote:
>>>>
>>>>> Waiting a little longer after a little over 26000 loops, grep -c
>>>>> reports 26032 string snippets in the live heap.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> And do this, knowing the time, that now it is high time to awake out of
>>>> sleep;
>>>> for now our salvation is nearer than when we first believed.
>>>>
>>>
>>>
>>>
>>> --
>>> And do this, knowing the time, that now it is high time to awake out of
>>> sleep;
>>> for now our salvation is nearer than when we first believed.
>>>
>>> _______________________________________________
>>> Red5 mailing list
>>> Red5 at osflash.org
>>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>>
>>>
>>
>>
>> --
>> http://www.xuggle.com/
>> xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and
>> video.
>>
>> Use Xuggle to get the power of FFMPEG in Java.
>>
>> _______________________________________________
>> Red5 mailing list
>> Red5 at osflash.org
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>>
>>
>
>
> --
> And do this, knowing the time, that now it is high time to awake out of
> sleep;
> for now our salvation is nearer than when we first believed.
>
--
And do this, knowing the time, that now it is high time to awake out of
sleep;
for now our salvation is nearer than when we first believed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://osflash.org/pipermail/red5_osflash.org/attachments/20090611/0f301606/attachment-0001.html>
-------------- next part --------------
### Eclipse Workspace Patch 1.0
#P red5_server
Index: src/org/red5/io/amf/Output.java
===================================================================
--- src/org/red5/io/amf/Output.java (revision 3662)
+++ src/org/red5/io/amf/Output.java (working copy)
@@ -20,6 +20,7 @@
*/
+import java.io.File;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
@@ -30,9 +31,12 @@
import java.util.Map;
import java.util.Set;
import java.util.TimeZone;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
+import net.sf.ehcache.Cache;
+import net.sf.ehcache.CacheException;
+import net.sf.ehcache.CacheManager;
+import net.sf.ehcache.Element;
+
import org.apache.commons.beanutils.BeanMap;
import org.apache.mina.core.buffer.IoBuffer;
import org.red5.annotations.Anonymous;
@@ -58,12 +62,37 @@
protected static Logger log = LoggerFactory.getLogger(Output.class);
/**
- * Cache encoded strings.
+ * Cache encoded strings... the TK way...
*/
- protected static final ConcurrentMap<String, byte[]> stringCache = new ConcurrentHashMap<String, byte[]>();
- protected static final Map<Class<?>, Map<String, Boolean>> serializeCache = new HashMap<Class<?>, Map<String, Boolean>>();
- protected static final Map<Class<?>, Map<String, Field>> fieldCache = new HashMap<Class<?>, Map<String, Field>>();
- protected static final Map<Class<?>, Map<String, Method>> getterCache = new HashMap<Class<?>, Map<String, Method>>();
+ private static Cache stringCache;
+
+ private static Cache serializeCache;
+
+ private static Cache fieldCache;
+
+ private static Cache getterCache;
+
+ private static CacheManager cacheManager;
+
+ private static CacheManager getCacheManager() {
+ if (cacheManager == null) {
+ try {
+ cacheManager = new CacheManager("conf" + File.separator + "ehcache.xml");
+ } catch (CacheException ce) {
+ log.warn("Could not create a CacheManager instance: " + ce.getMessage());
+ try {
+ CacheManager.create("conf" + File.separator + "ehcache.xml");
+ cacheManager = CacheManager.getInstance();
+ } catch (CacheException ce2) {
+ log.warn("Could not create the CacheManager singleton, failing over: " + ce2.getMessage());
+ CacheManager.create();
+ cacheManager = CacheManager.getInstance();
+ }
+ }
+ }
+
+ return cacheManager;
+ }
/**
* Output buffer
@@ -322,15 +351,19 @@
buf.put(AMF.TYPE_END_OF_OBJECT);
}
+ @SuppressWarnings("unchecked")
protected boolean serializeField(Serializer serializer, Class<?> objectClass, String keyName, Field field, Method getter) {
- Map<String, Boolean> serializeMap = serializeCache.get(objectClass);
+// to prevent, NullPointerExceptions, get the element first and check if it's null.
+ Element element = getSerializeCache().get(objectClass);
+ Map<String, Boolean> serializeMap = (element == null ? null : (Map<String,Boolean>)element.getObjectValue());
+
if (serializeMap == null) {
serializeMap = new HashMap<String, Boolean>();
- serializeCache.put(objectClass, serializeMap);
+ getSerializeCache().put(new Element(objectClass, serializeMap));
}
Boolean serialize;
- if (serializeCache.containsKey(keyName)) {
+ if (getSerializeCache().isKeyInCache(keyName)) {
serialize = serializeMap.get(keyName);
} else {
serialize = serializer.serializeField(keyName, field, getter);
@@ -339,12 +372,15 @@
return serialize;
}
-
+
+ @SuppressWarnings("unchecked")
protected Field getField(Class<?> objectClass, String keyName) {
- Map<String, Field> fieldMap = fieldCache.get(objectClass);
+// again, to prevent null pointers, check if the element exists first.
+ Element element = getFieldCache().get(objectClass);
+ Map<String, Field> fieldMap = (element == null ? null : (Map<String,Field>)element.getObjectValue());
if (fieldMap == null) {
fieldMap = new HashMap<String, Field>();
- fieldCache.put(objectClass, fieldMap);
+ getFieldCache().put(new Element(objectClass, fieldMap));
}
Field field = null;
@@ -366,13 +402,15 @@
return field;
}
-
+
+ @SuppressWarnings("unchecked")
protected Method getGetter(Class<?> objectClass, String keyName) {
-
- Map<String, Method> getterMap = getterCache.get(objectClass);
+// check element to prevent null pointer
+ Element element = getGetterCache().get(objectClass);
+ Map<String, Method> getterMap = (element == null ? null : (Map<String, Method>)element.getObjectValue());
if (getterMap == null) {
getterMap = new HashMap<String, Method>();
- getterCache.put(objectClass, getterMap);
+ getGetterCache().put(new Element(objectClass, getterMap));
}
Method getter = null;
@@ -497,12 +535,13 @@
* @return encoded string
*/
protected static byte[] encodeString(String string) {
- byte[] encoded = stringCache.get(string);
+ Element element = getStringCache().get(string);
+ byte[] encoded = (element == null ? null : (byte[])element.getObjectValue());
if (encoded == null) {
ByteBuffer buf = AMF.CHARSET.encode(string);
encoded = new byte[buf.limit()];
buf.get(encoded);
- stringCache.put(string, encoded);
+ getStringCache().put(new Element(string, encoded));
}
return encoded;
}
@@ -552,4 +591,31 @@
clearReferences();
}
+ protected static Cache getStringCache() {
+ if (stringCache == null)
+ stringCache = getCacheManager().getCache("org.red5.io.amf.Output.stringCache");
+
+ return stringCache;
+ }
+
+ protected static Cache getSerializeCache() {
+ if (serializeCache == null)
+ serializeCache = getCacheManager().getCache("org.red5.io.amf.Output.serializeCache");
+
+ return serializeCache;
+ }
+
+ protected static Cache getFieldCache() {
+ if (fieldCache == null)
+ fieldCache = getCacheManager().getCache("org.red5.io.amf.Output.fieldCache");
+
+ return fieldCache;
+ }
+
+ protected static Cache getGetterCache() {
+ if (getterCache == null)
+ getterCache = getCacheManager().getCache("org.red5.io.amf.Output.getterCache");
+
+ return getterCache;
+ }
}
\ No newline at end of file
Index: src/conf/ehcache.xml
===================================================================
--- src/conf/ehcache.xml (revision 3606)
+++ src/conf/ehcache.xml (working copy)
@@ -12,7 +12,7 @@
Subdirectories can be specified below the property e.g. java.io.tmpdir/one
-->
- <diskStore path="java.io.tmpdir"/>
+ <diskStore path="java.io.tmpdir/red5"/>
<!--
Specifies a CacheManagerEventListenerFactory, be used to create a CacheManagerPeerProvider,
@@ -285,4 +285,35 @@
diskExpiryThreadIntervalSeconds="120"
memoryStoreEvictionPolicy="LFU"
/>
+
+ <!-- org.red5.io.amf.Output.stringCache
+ Caches strings for output via AMF serialization. Keeps up to 1000 elements in memory,
+ does not overflow to disk, is not eternal, and expires elements after 20 minutes of
+ inactivity. -->
+ <cache name="org.red5.io.amf.Output.stringCache"
+ maxElementsInMemory="1000"
+ eternal="false"
+ timeToIdleSeconds="1200"
+ overflowToDisk="false"/>
+
+ <!-- org.red5.io.amf.Output.serializeCache
+ -->
+ <cache name="org.red5.io.amf.Output.serializeCache"
+ maxElementsInMemory="200"
+ eternal="false"
+ timeToIdleSeconds="1200"
+ overflowToDisk="false"/>
+
+ <cache name="org.red5.io.amf.Output.fieldCache"
+ maxElementsInMemory="200"
+ eternal="false"
+ timeToIdleSeconds="1200"
+ overflowToDisk="false"/>
+
+ <cache name="org.red5.io.amf.Output.getterCache"
+ maxElementsInMemory="200"
+ eternal="false"
+ timeToIdleSeconds="1200"
+ overflowToDisk="false"/>
+
</ehcache>
Index: src/org/red5/io/amf3/Output.java
===================================================================
--- src/org/red5/io/amf3/Output.java (revision 3662)
+++ src/org/red5/io/amf3/Output.java (working copy)
@@ -29,6 +29,8 @@
import java.util.Map;
import java.util.Set;
+import net.sf.ehcache.Element;
+
import org.apache.commons.beanutils.BeanMap;
import org.apache.mina.core.buffer.IoBuffer;
import org.red5.annotations.Anonymous;
@@ -142,12 +144,13 @@
}
protected static byte[] encodeString(String string) {
- byte[] encoded = stringCache.get(string);
+ Element element = getStringCache().get(string);
+ byte[] encoded = (element == null ? null : (byte[])element.getObjectValue());
if (encoded == null) {
ByteBuffer buf = AMF3.CHARSET.encode(string);
encoded = new byte[buf.limit()];
buf.get(encoded);
- stringCache.put(string, encoded);
+ getStringCache().put(new Element(string, encoded));
}
return encoded;
}
More information about the Red5
mailing list