[Red5devs] Offer to the Red5 Community: Continuous building

[Art Clarke]

IF YOU'RE ON THE RED5 TEAM PLEASE READ THIS.  IT'S AN OFFER TO HELP,
NOT A REQUEST FOR HELP.

Hi folks,

First off, thanks so much for all the great work on Red5; It's amazing!

And now for the suggestion: I think it'd be great if the following
existed, and we at Vlideshow would be HAPPY to set it up if folks
wanted us to (with the exception of option F which is beyond our
control).

Please let me know by saying which options you'd like if any (for
example, "I'd like A, B, C and E", or "I'd like none, thank you very
much").

Options
Option A) I'd like a continuous Red5 build server that rebuilds the
server whenever a check-in occurs and e-mails the person who checked
in last if the build fails.
Option B) I'd like someone to go through the "ant run-tests" tests and
make them all run successfully
Option C) I'd like to have the continuous Red5 build server also run
the Java tests and let folks know if they succeed or fail
Option D) I'd like a set of standalone flash-based (not RTMP-Client)
unit tests that test connecting to the server, playing back video
files, publishing files, and calling a method on objects, as a way to
do minimal end-to-end validation of a check-in.
Option E) I'd like the continuous build server to execute those
flash-based tests in a headless (i.e. no windows) environment
Option F) I'd like world peace.
Option G) I'd like the continuous build server to auto build of the
javadoc for the server

If we end up decided to do any of these, all I ask for right now is a
sandbox area I can check code into (asked for username: aclarke).  If
you want grimy details on what I'm proposing, please ready on.

<h1>Background</h1>

At Vlideshow (still in early early alpha) we use Red5 as our server.
Every few weeks we integrate to tip of tree to stay current, but
otherwise we stay on a stable revision number.  However every time we
integrate to tip (like yesterday) we uncover new issues that take
valuable time to fix.  These issues are to be expected (heck, it's
active code), but might be reducable if Red5 was continuously
integrated.

The first time we ran into this was integrating from 0.6.x to 0.7.0 --
a lot of changes which took us over a week to work through.  So at
that time, we built a series of tests for our own code that
auto-builds red5, and that then auto-runs a server, and fires up a set
of headless (yes) flash tests.

13 days ago we were able to see that the server build was succeeding,
but that old custom applications would fail to connect if they had the
same .jar file in their lib directories that red5 had.  We couldn't
drop stuff to fix it then, but we've fixed that now in our custom
application, and today are trying to figure out why some of our tests
still fail -- specifically invoking remote methods.  We'll track that
down next.

The thing that occurred to us last night when we debugging this was it
might be nice if the folks on the Red5 team had access to this test
framework as well.  We were in the midst of user-testing last week so
couldn't look into the problems 13 days ago when they started, but
I'll bet that if someone was checking in code on those days and got a
hint that something about legacy apps might break, they might
appreciate it.

<h1>technical proposal</h1>

<h2>Option A:</H2>

Our continuous server is based on Hudson
(https://hudson.dev.java.net/), and already builds and e-mails us if
the red5 build breaks.  What I'm suggesting is that someone let me
know the e-mails to add to the distribution list, and I'll make it
send e-mails there as well when builds break.  I can also provide
login-accounts for people to browse the hudson server, and will try
initially setting it up so anonymous users can see the results of
builds (but may need to curtail that if load is too high:  Right now
this would be hosted off our dev-server (which is a Fedora 8 box
sitting in my apartment) but if folks have a Linux environment that
they wanted to volunteer I'd be happy to talk offline about making a
more "hardy" version).

<h2>Option B</h2>
This will require some work but I think it's valuable.  Right now
(after fixing the build file to compile the tests) 64% of unit tests
pass.  My proposal is that we at Vlideshow go through the unit tests
and get us to 100% by either (a) fixing the test (preferred) or (b)
removing the test if it's beyond our ability to fix right now (I'll
e-mail any such cases to the list).

<h2>Option C</h2>
This depends on Option B, but once the "ant run-tests" works it's
trivial to add this in (we do it for all of our other projects), and
if we get Option A to allow folks to browse, it's really easy to see
the results and why they failed.

<h2>Option D</h2>
We have already build a set of tests in ActionScript using the
http://asunit.org/ framework.  Today we run the following tests:
1) Connect to a server and disconnect once.
2) Connect to a server and disconnect over 100 times.
3) Connect to a server and play a video file.
4) Connect to a server and publish the contents of the camera, pausing
every 2-seconds.
5) Connect to a server and playback a recorded file.
6) Connect to a server and call a method on the server and check the
return value.
(there are a couple of other tests, but they are testing the "state
machines" we use to manage connecting and playing.  did we mention
we'd also donate those state machines, which are what we use for
managing connections in our application and seem to work well,
abstracting away all the NetConnection and NetStream random messages).

To use you just bring up a standalone flash player and run the
compiled swf file.  The first time you run them you should set it to
"remember" that camera access is allowed and from then on they can run
unattended.  If you set your permissions correctly it will also exit
the flash player if all tests pass and return a code of 0 so you can
use it in automated builds (with some caveats).

This option is a little tricky as asunit is GPL licensed.  What I'm
proposing is that we make a top-level source directory
svn/test/flash/trunk/autotest/ and in there we'd put our source code
as well as a reference to ASUnit.  This part of the tree would be
GPL-infected, but by organizing it in this way we can assure that
people only get GPL-infected if they decided to go into this area.

If you like this option, people can add their own unit tests there as
well over time.

Unfortunately while there may be other flash auto-testing frameworks
that are not GPL licensed, <b>we are not offering to use them</b>.
AsUnit works well for us internally and was a large investment of
resources to get working.

<h2>Option E</h2>
We have written some scripts that will attempt to use Option D and
auto-run it.  It's about 90% reliable at this point but we continue to
work on it (the biggest problem is getting the flash-player to
auto-exit if a failure occurs), but I'd be happy to add people to the
e-mail distribution when failures occurr.  Let me know.

<h2>Option F</h2>
No comment.

<h2>Option G</h2>
This is also trivial assuming we selection Option A.

Let me know your thoughts,

- Art & Robert (also known as Vlideshow)