Introduction

SWAMI is a framework for running collaborative filtering algorithms and evaluating the effectiveness of those algorithms. It uses the EachMovie dataset, generously provided by Compaq.

SWAMI was a class project in the Large Datasets course, Fall 1999. The class paper we wrote up is also available.

The EachMovie Dataset

The EachMovie movie preference dataset was collected by DEC (now Compaq) research. It consists of 72916 users, 1628 movies, and 2811983 movie votes. More information, including the original Eachmovie data and terms of usage, is available.

Our Thoughts on the EachMovie Dataset

Some of the users can be eliminated outright, because of low number of votes.

Interestingly, each of the movie votes has a weight and a score, where the weight denotes whether or not the user actually saw the movie, and where the score denotes the user's movie rating. (For our algorithms, we ignored the weight, setting it to 1 for all users.)

Just FYI, here are some interesting stats on the EachMovie dataset:

   Some "macro-statistics" about the dataset as a whole.
   These stats were taken only over the users who actually
   voted, and movies for which votes were cast.

     number of users who actually voted: 61263
     number of movies with at least 1 vote: 1623

     maximum number of votes by a user: 1455
     min : 1
     mean number of votes by a user: 45.9
     median : 26

     maximum number of votes for a movie: 32864
     min : 1
     mean votes for a movie: 1732.4
     median : 379

   Roughly speaking, a user cast a mean vote of about 3.5 with a
   variation of about 1.25.  This suggests that the Microsoft
   paper, which showed a number of algorithms that predicted
   votes to within 1 vote, didn't really do much better than what
   one can do by just predicting the user's mean vote on every
   movie.

Here are some interesting graphs of the EachMovie dataset:

num_votes_user.gif shows how many users cast at least a certain number of votes
num_votes_movie.gif shows how many movies received at least a certain number of votes
dist_user_vote_avg.gif shows the mean vote of user votes
dist_user_vote_stddev.gif shows the stddev of user votes
dist_movie_vote_avg.gif shows the mean vote on movies
dist_movie_vote_stddev.gif shows the stddev of votes on movies
Vote distribution on Like Water for Chocolate
Vote Distribution on Nell
Vote Distribution on Natural Born Killers
corrdist.gif shows the distribution of the Pearson correlation coefficient. Randomly selected 1415 users at a time, computed all pairs of correlation coeffs between pairs. Repeated this 100 times, that's the plot. (center line is mean, dashed lines are min/max for each coeff. note logarithmic scale.) Punchline: strong correlations are rare? (according to the figure, approx. 1e-6 chance that a given *pair* of users will be strongly correlated).
uservote_overlap.gif is a graph of the distribution of percent overlap. Random pairs with at least 26 votes (median number of votes) were drawn, and intersection of votes computed. The "percent overlap" is the number of intersecting votes divided by the smallest number of votes cast by either member of the pair.

Just FYI, here are some interesting attempts at visualizing the EachMovie dataset:

Principal Component Analysis (PCA)
In these graphs, we performed a Principal Component Analysis on the movies, extracting the top 6 eigenvectors. These 6 eignevectors are mapped, in order, to x, y, z, r, g, b. Not much to see here, though. The following are four images of the data at different angles.
Principal Component Analysis and Clustering
Here is the same PCA data using only the top 3 eigenvectors for x, y, z. The colors are derived from clustering the movies using k-means with 200 clusters. Same color means same cluster, but the color for a cluster is arbitrary.
- pca-movie-xyzcluster-km200m-1.gif
- pca-movie-xyzcluster-km200m-2.gif

Here are the MatLab files used to generate these graphs.

Haven't received them yet...

Overview of SWAMI Architecture

SWAMI can be roughly partitioned into four parts: Pre-indexed data, filters, predictors, and evaluation. Each of the four parts is, for the most part, loosely coupled and interchangeable. A system block-diagram is depicted below.

The Pre-indexed data is generated from the EachMovie dataset, which organizes the votes to make them fast to retrieve by user or by movie. We create a BTree-like index on top of the file after sorting it. More information on the indexing process is available.

In some cases, we use a subset of the data, for performance purposes. The subset is comprised of the original data with certain members aggregated together using Pearson clustering, or it could just be a random subset of the data.

The filters let you filter out votes based on user, movie, or list of votes. The filters come in handy when doing evaluation (explained in more detail below).

The predictors predict what a user's vote will be on a movie. Currently, we have a few simple predictors and a few more complex ones. The predictors are the heart of a collaborative filtering system, and are what we were most interested in evaluating in SWAMI.

The evaluation is a suite of scripts and Java classes for seeing how well a predictor works. There are a range of attributes for ranking predictors, for example elapsed time and accuracy of prediction. We were most interested in the accuracy of predictions. The approach we used was to take a user, filter out some votes that this user actually had, and then use a predictor to predict a score. This predicted score was compared to the actual score.

Downloading SWAMI and Running it on Your System

System Requirements

Java Development Kit (JDK 1.2.2 or above)
Java Web Server (JWSDK 1.0.1+) and Java Server Pages (JSP 1.0.1+) (optional)
Perl 5 or above (optional)
Around 120 Megs of disk space (after the above is installed)
A good-sized dosage of your favorite caffeinated beverages.

Here are the files for download

SWAMI snapshot
- swami-classes.zip (200K - required)
- swami-data.zip (20Megs - required) (More info on indexing)
- swami-javadoc.zip (300K - optional)
- swami-webpages.zip (20K - optional)
Check out the README file for info on running the SWAMI snapshot.
The Berkeley Software License, under which everything above (except for the data) is distributed. There is a separate terms of usage license for the EachMovie dataset.
Feel free to browse our Javadoc documentation online.

Functionality and Non-Functionality

We have the capability of reading in gzipped files. However, this functionality isn't used, since we didn't try to see if we could index gzipped files so that votes could be read in quickly.
We have the capability of reading in files over HTTP, but it suffers from the same problem as above.
Currently, we do not have the capability to permanently add a user. Users can be added in memory, but not to disk. For the most part, our evaluations run only on the existing data set.
Performance is decent for the most part, but we did not profile the code. In other words, SWAMI runs pretty well even though we didn't do many optimizations, but there's definitely room for improvements in performance.

Current Status and Contacting Us

SWAMI was developed in 1999 as a class project. As you can probably guess, the current status of the software is "as is". As graduate students, we don't really have too much spare time to revise the software. However, if you want to submit contributions and make them available to the research community, we'd be glad to host them on our web pages.