NWN Modding Statistical Analysis: Part 1

A while back, I was perusing the NWN2 section of the Bioware Social Forums when a discussion of the download and voting rates of the current top new mods caught my eye for a reason that will soon become clear. The OP of that thread was disappointed over how few votes he was getting compared to his downloads and was wondering if he’d hit 10 votes by the time his module was off the "Newly Released" list on the right side of the Vault main page.

I've long thought it was just a generally accepted fact that only about one to two percent of downloaders will vote. However, I’ve been thinking over that conversation for a few weeks and I decided to do a little digging. So I took the current list of top 15 new mods from the Vault sidebar (as of February 24, 2011). Note that these are only the ones that haven’t yet achieved the Vault Hall of Fame section. The raw data for these 15 mods is given below.


Now to be clear, some of these download numbers appear to be such that the module should graduate to the HoF (5000 downloads). In these cases, I’ve included downloads of all different forms of the same module. For example, some modules have a self-extracting download and then a manual install download. Neither of these are individually above 5000, but they are above 5000 when combined. I've added the two download numbers up because I think it's reasonable to conclude that these two groups constitute different players. On the other hand, several of these have multiple modules attached to the entry (such as TMGS), all of which are required to play. Therefore, these do not represent different players and so I only took the largest number. For TMGS, for example, the download count is that for Module 1, which is the individual module with the most downloads.

Now it should be clear why this conversation particularly caught my eye. Whereas most of the modules have vote percentages in the one to two percent range, one module stands out as a significant outlier: mine. In an effort to explain what was going on, I looked at the downloads per month for each of the 15 modules. While TMGS is at the upper end of the group, it certainly isn't the highest. "Path of Evil" is more than doubling TMGS' pace, although it has only been released two months and one would expect the biggest surge immediately after release. On the other hand, "Planescape: the Shaper of Dreams" has been out seven months longer than TMGS and has almost 150 more downloads per month... and yet the vote percentage is still under one percent.

I was interested to see how these numbers compared with some of the "classic" modules from NWN2's past, so I looked at the top 50 modules overall and pulled out some of the notable ones. The only criteria used to select this group over the others was that I remembered them being big news when released. The expanded table is given below.


So again, even the older modules have the same roughly one to two percent voting rate, so I'm at a loss to explain why TMGS seems to be almost tripling the voting percentage of most of the other mods out there.

However, I was also interested to see how download rates have changed over time. It is obvious that the player base is smaller, so download rates must have diminished, but by how much? So I put the data into a handy little chart shown below.


A few points. First, the x-axis is the months since release, meaning further out along the x-axis represents longer ago. For reference, I've put vertical lines where the change in years occurred. Two months ago was the change to 2011, 12 months before that was 2010, and so forth. I also added in the release points for both NWN2 expansion packs and a couple other fantasy-themed RPGs to see if that might shed some light. Dragon Age released in November 2009 and SoZ was in November 2008. MotB and The Witcher both released in October 2007.

The first thing that stands out to me is the tremendous scattering of the data, although the obvious trend is still clear. The downloads per month is generally going down. The linear "best fit" trendline as calculated by Excel and its equation are also shown on the graph. According to this, a module released today (x = 0) should expect a download rate of about 232 per month.

However, I looked at the list of 15 top-rated new modules and noticed that several aren't traditional adventure mods. I don't wish to debate the merits of including such modules in a list of modules here, but I did wonder if removing these from the data would tighten up the scatter a bit. So I removed "The Heist at the Neverwinter Lights Casino", "NWN2 OC Makeover", "SOZ Holiday Expansion Project", and "Tanithiel." I also had to remove "Halloween" from the legacy group. The culled-data graph is given below.


What is interesting is that all of the five removed modules were below the line in the first graph, which means they were all being downloaded at a rate below what would be expected (perhaps an indication of their niche nature). As expected, this moved the line up generally, but especially on the right, meaning the slope increased. I refrained from doing the rigorous math because even by eye it is obvious the scattering decreased a bit. However, the prognosis for a module released right now was basically confirmed. One could expect about 228 downloads per month.

From examining the release points for the expansion packs and other games, it looks like Dragon Age did a decent job of damaging NWN2. "Trinity", "Misery Stone", and "Planescape: Shaper of Dreams" were all released within a month or two of Dragon Age, when several players were presumably giving NWN2 a last hurrah while DA bugs were found and fixed, and these maintained a fairly healthy download rate of above 400 per month. And yes, other modules were getting considerably less than this, but after that point, no module except "Path of Evil" is coming remotely close to that rate, and that module is still too new for me to be believe that rate will continue. For the most part, the top downloaded modules now are pulling in what the bottom downloaded (but still highly-rated) modules were doing even fourteen months ago.

Another observation. Using the non-culled data of the first chart, the trend line will cross the x-axis at -36.93 months, which is March of 2014. Using the culled data with a steeper slope, it will be -25.59 months, or April of 2013. What does this mean? Well, that’s the point when, theoretically, a newly-released mod will have a download rate of zero per month. In other words, it is the functional end of NWN2's life unless something happens to arrest this curve.

Also, based off the first line, TMGS projects to end with 5993 downloads in March of 2014. Based off the second line, it will end with 4633 downloads in April of 2013. What this means is that there's the very real possibility - even the probability - that TMGS will never make the HoF.

Now I know there are problems with this over-simplistic analysis. First, the data set is comparatively small. There are 154 NWN2 modules on the Vault, although using the bottom half to project the success of future highly-rated mods would be useless. Still, using all of the top 50 instead of only 25 of them would be better. Second, the download rate will never truly go to zero, so some type of true curve with an asymptote at the x-axis would be more accurate. Finally, all modules get downloaded more in their first couple months than at other times. However, I have no download by month data, so I don't have any way of knowing how many of "Harp and Chrysanthemum’s" 27 thousand downloads were in year one, year two, and so forth. It is almost certainly not getting 700 downloads per month now while it must have gotten fifteen hundred per month or more at its height. Finally, any true analysis should factor in promotional efforts by the author. For all I know, it might be possible to greatly exceed these numbers with an unrelenting ad campaign.

For the record, I've graphed the data for votes per download for all the above modules (full and culled). These graphs are below, and they show what we already knew from the raw data. The vote percentage has remained pretty flat over time, indicating the time of release is pretty irrelevant in this case. In the first graph, the slope is -0.0002, and it is roughly -0.0003 in the graph. The negative slopes actually show that it is slightly better for newer modules over all, but not very significantly. A second point for these graphs? There's TMGS in the top left of both as a major outlier. And so we've come full circle with no more answers than when I started. For some reason, TMGS seems to compel a greater percentage of people who play it to come back and vote.




I have two more analysis I want to do with this series. My next post will look at voting over time. I say this because the current top 15 new mods are all in the top 33 mods of all-time. Half of the top 10 all-time is within the current new top 15. That seems statistically unlikely unless vote inflation is occurring. My final piece will take a look at the same kinds of stats for NWN1. Perhaps that will shed further light on the expected lifespan of NWN2.