Hosted by: Authors of the largest study of Learning Management Systems, providing a comparative analysis of 70+ LMS products. Click Here for more information.

There's lots of discussion in the industry about the "scalability" of learning management systems (LMS). However, until now, much of that discussion has been focused on how an LMS can scale to meet the needs of an organization with thousands and even tens-of-thousands of learners. With the extreme focus on large-scale deployments, some LMS vendors will now only engage in a project if there is a minimum of 2000 or 2500 learners. This shootout gave viewers an opportunity to see systems that can also be scaled downward to meet the needs of a relatively small number of learners (under 2000). See the results and a description of the Shootout below:

September 23, 24, and 25, 2002

Anaheim, CA

Results – Top Finishers by Audience Vote (composite score - 35 points possible)

NOTE: See more detailed scoring by category below.

Scoring Description

Shootout Description

Teams were invited to participate in the shootout based on their ability to meet all the requirements of a scenario of a a made-up company with the following characteristics...

Scenario:

• The company has 250 employees, spread out in 3 different countries (hence the need for e-learning delivery). They would like to be able to run their LMS and e-learning courses from one central server (at the home office in the U.S.), but are open to suggestions on the best approach.
• They want to create about 80% of their own content. They aren’t necessarily looking for content authoring in the LMS, but would at least like you to make a recommendation on what authoring products to use. They have some anxiety from reading articles about the difficulties of integrating e-learning content with LMS…they’ve heard that “standards compliant” doesn’t always mean plug-and-play. They are looking to you for guidance in helping them make good choices.
• They want their internal SME’s to create much of the content.
• They would like to entertain the idea of buying some 3^rd party courses (sales and customer service training) to round out their curriculum. They would like to know how open your system is to launching 3^rd party courses, especially ones that are standards (AICC/SCORM) compliant. They figure about 20% of their content will be off-the-shelf. If your system isn’t standards compliant, how would you recommend that they proceed?
• They want to make sure they have full access to the performance data. They’d like to do ad hoc reporting, if possible. How can they do this?
• They also want their employees to be able to communicate with others who are enrolled in the same course. They are not looking for live, synchronous (virtual classroom) capability right now, but they would like to use threaded discussions (topic based) and/or learner-to-learner, instructor-to-learner email, or something like that. If you don’t have this capability, how would you recommend integrating it?
• They also want some basic classroom management. They offer about 25 courses a year and want to post these courses through their learning site when they become available. They also want the system to do some basic enrollment and wait listing if possible. Other areas that would get them excited are automatic notification via email for those who enroll and they would also like to have managers grant approval before enrollment occurs.
• They haven’t figured out yet whether they want to have you host the LMS for them or install it on their local intranet. They are looking to you for guidance in helping them figure out which would be more cost effective and which will work best for their situation. Make your case either way….or both ways for them to choose.
• Bottom line: How much will all this cost? Can you give them some ballpark figures based on what they are trying to do today?
• Finally, they think they might open up their training to end-users in the next two years. Their estimate is that about 2000 end users (in addition to internal employees) might benefit from the training. How does the system (and more importantly the pricing model) scale to meet their needs?

The format of the shootout was to literally have each team, in a round-robin format, address the needs of our made up company. Each team was given a list of tasks to complete in a 3:00 debrief. Here is a copy of the actual checklist each team was given and the time constraints for each task.

Click Here to see a copy of the actual checklist that was given to each team.

The Shootout area was set up with 4 large screens allowing the audience to view the competitors side by side (See the pictures below).

RESULTS

All Participants (listed alphabetically by product name)

Top 3 Finishers - Learner Interface Adaptation  (audience score - 5 points possible)

This category required the LMS vendors to go to the website www.entnet.org/index2.cfm and match their look and feel in creating an empty learning portal. We had all the vendors bring up the interface they created at the same time, then we asked the audience to vote on who had done the best job.

Notice: each of the teams in this picture is displaying the learner interface the created from the AAO website listed above.

Top 3 Finishers - e-Learning Management  (audience score - 5 points possible)

This category required each team to (1) attach an AICC-compliant or SCORM-conformant course to their course catalog, (2) Add a new user, (3) assign the new user to the course, (4) launch the course as the learner, (5) show how performance data flowed from the course into the LMS, (6) show how a learner would see his/her own score, and (7) show how an instructor or administrator would also see this score.

Top 3 Finishers - Performance Reporting  (audience score - 5 points possible)

This category required each team to (1) show how a learner sees his/her own score, (2) show how an admin user would see the score, and (3) discuss how reports might be customized.

Top 3 Finishers - Classroom Management  (audience score - 5 points possible)

This category required each team to (1) Add a classroom event to the curriculum, (2) register the learner for the class, (3) show how enrollment and notification work, and (4) show how an instructor could add performance data for the classroom event.

Top 3 Finishers - Collaborative Learning  (audience score - 5 points possible)

This category required each team to (1) show how threaded discussion might be added, (2) show internal, learner-to-learner and learner-to-instructor email, (3) show how a virtual classroom session might be added, and (4) demonstrate any other unique collaborative capabilities.

Top 3 Finishers - Price-to-Value Ratio  (audience score - 5 points possible)

This category required each team to (1) tell how long it would take to implement their solution, given the scenario above, (2) tell how much their solution would cost for both a hosted and a locally installed solution, and (3) discuss what upgrade paths look like if our made-up company moved from 250 learners to 2000 learners. NOTE: In general, the audience gave the highest scores to those with the lowest cost.

Top 3 Finishers - Overall Impression  (audience score - 5 points possible)

Audience preference based on overall impression.

Links to Actual "Learning Portals" created during the shootout 

Watch for more links as they become available. 

Shootout Hosts

brandon-hall.com & Online Learning Magazine jointly hosted this shootout event

Authors of the largest study of Learning Management Systems, providing a comparative analysis of 70+ LMS products. Click Here for more information.

More Pictures of the Shootout

Another view showing that LMS vendors matched the interface from the website www.entnet.org/index2.cfm

The audience played a critical role, serving as judges for the competition, providing evaluations in each of the major categories listed above.

Bryan Chapman, e-learning analyst at brandon-hall.com, served as referee for the Shootout event with assistance from Amy Sitze, publications manager at brandon-hall.com.

Oracle giving a 3 minute debrief, following the task checklist.

In order to be completely fair and provide a level playing field, the referees were sticklers about time limits. As soon as the count down clock ended, the team had to immediately complete their presentation on each topic. The whole shootout was conducted in a round-robin format.

A view showing the placement of the 4 screens (4 teams a day, over a 3-day time period), as well as the countdown clock. The team from Websoft is walking through the scenario in this piciture.