NWN Modding Statistical Analysis: Part 1

A while back, I was perusing the NWN2 section of the Bioware Social Forums when a discussion of the download and voting rates of the current top new mods caught my eye for a reason that will soon become clear. The OP of that thread was disappointed over how few votes he was getting compared to his downloads and was wondering if he’d hit 10 votes by the time his module was off the "Newly Released" list on the right side of the Vault main page.

I've long thought it was just a generally accepted fact that only about one to two percent of downloaders will vote. However, I’ve been thinking over that conversation for a few weeks and I decided to do a little digging. So I took the current list of top 15 new mods from the Vault sidebar (as of February 24, 2011). Note that these are only the ones that haven’t yet achieved the Vault Hall of Fame section. The raw data for these 15 mods is given below.


Now to be clear, some of these download numbers appear to be such that the module should graduate to the HoF (5000 downloads). In these cases, I’ve included downloads of all different forms of the same module. For example, some modules have a self-extracting download and then a manual install download. Neither of these are individually above 5000, but they are above 5000 when combined. I've added the two download numbers up because I think it's reasonable to conclude that these two groups constitute different players. On the other hand, several of these have multiple modules attached to the entry (such as TMGS), all of which are required to play. Therefore, these do not represent different players and so I only took the largest number. For TMGS, for example, the download count is that for Module 1, which is the individual module with the most downloads.

Now it should be clear why this conversation particularly caught my eye. Whereas most of the modules have vote percentages in the one to two percent range, one module stands out as a significant outlier: mine. In an effort to explain what was going on, I looked at the downloads per month for each of the 15 modules. While TMGS is at the upper end of the group, it certainly isn't the highest. "Path of Evil" is more than doubling TMGS' pace, although it has only been released two months and one would expect the biggest surge immediately after release. On the other hand, "Planescape: the Shaper of Dreams" has been out seven months longer than TMGS and has almost 150 more downloads per month... and yet the vote percentage is still under one percent.

I was interested to see how these numbers compared with some of the "classic" modules from NWN2's past, so I looked at the top 50 modules overall and pulled out some of the notable ones. The only criteria used to select this group over the others was that I remembered them being big news when released. The expanded table is given below.


So again, even the older modules have the same roughly one to two percent voting rate, so I'm at a loss to explain why TMGS seems to be almost tripling the voting percentage of most of the other mods out there.

However, I was also interested to see how download rates have changed over time. It is obvious that the player base is smaller, so download rates must have diminished, but by how much? So I put the data into a handy little chart shown below.


A few points. First, the x-axis is the months since release, meaning further out along the x-axis represents longer ago. For reference, I've put vertical lines where the change in years occurred. Two months ago was the change to 2011, 12 months before that was 2010, and so forth. I also added in the release points for both NWN2 expansion packs and a couple other fantasy-themed RPGs to see if that might shed some light. Dragon Age released in November 2009 and SoZ was in November 2008. MotB and The Witcher both released in October 2007.

The first thing that stands out to me is the tremendous scattering of the data, although the obvious trend is still clear. The downloads per month is generally going down. The linear "best fit" trendline as calculated by Excel and its equation are also shown on the graph. According to this, a module released today (x = 0) should expect a download rate of about 232 per month.

However, I looked at the list of 15 top-rated new modules and noticed that several aren't traditional adventure mods. I don't wish to debate the merits of including such modules in a list of modules here, but I did wonder if removing these from the data would tighten up the scatter a bit. So I removed "The Heist at the Neverwinter Lights Casino", "NWN2 OC Makeover", "SOZ Holiday Expansion Project", and "Tanithiel." I also had to remove "Halloween" from the legacy group. The culled-data graph is given below.


What is interesting is that all of the five removed modules were below the line in the first graph, which means they were all being downloaded at a rate below what would be expected (perhaps an indication of their niche nature). As expected, this moved the line up generally, but especially on the right, meaning the slope increased. I refrained from doing the rigorous math because even by eye it is obvious the scattering decreased a bit. However, the prognosis for a module released right now was basically confirmed. One could expect about 228 downloads per month.

From examining the release points for the expansion packs and other games, it looks like Dragon Age did a decent job of damaging NWN2. "Trinity", "Misery Stone", and "Planescape: Shaper of Dreams" were all released within a month or two of Dragon Age, when several players were presumably giving NWN2 a last hurrah while DA bugs were found and fixed, and these maintained a fairly healthy download rate of above 400 per month. And yes, other modules were getting considerably less than this, but after that point, no module except "Path of Evil" is coming remotely close to that rate, and that module is still too new for me to be believe that rate will continue. For the most part, the top downloaded modules now are pulling in what the bottom downloaded (but still highly-rated) modules were doing even fourteen months ago.

Another observation. Using the non-culled data of the first chart, the trend line will cross the x-axis at -36.93 months, which is March of 2014. Using the culled data with a steeper slope, it will be -25.59 months, or April of 2013. What does this mean? Well, that’s the point when, theoretically, a newly-released mod will have a download rate of zero per month. In other words, it is the functional end of NWN2's life unless something happens to arrest this curve.

Also, based off the first line, TMGS projects to end with 5993 downloads in March of 2014. Based off the second line, it will end with 4633 downloads in April of 2013. What this means is that there's the very real possibility - even the probability - that TMGS will never make the HoF.

Now I know there are problems with this over-simplistic analysis. First, the data set is comparatively small. There are 154 NWN2 modules on the Vault, although using the bottom half to project the success of future highly-rated mods would be useless. Still, using all of the top 50 instead of only 25 of them would be better. Second, the download rate will never truly go to zero, so some type of true curve with an asymptote at the x-axis would be more accurate. Finally, all modules get downloaded more in their first couple months than at other times. However, I have no download by month data, so I don't have any way of knowing how many of "Harp and Chrysanthemum’s" 27 thousand downloads were in year one, year two, and so forth. It is almost certainly not getting 700 downloads per month now while it must have gotten fifteen hundred per month or more at its height. Finally, any true analysis should factor in promotional efforts by the author. For all I know, it might be possible to greatly exceed these numbers with an unrelenting ad campaign.

For the record, I've graphed the data for votes per download for all the above modules (full and culled). These graphs are below, and they show what we already knew from the raw data. The vote percentage has remained pretty flat over time, indicating the time of release is pretty irrelevant in this case. In the first graph, the slope is -0.0002, and it is roughly -0.0003 in the graph. The negative slopes actually show that it is slightly better for newer modules over all, but not very significantly. A second point for these graphs? There's TMGS in the top left of both as a major outlier. And so we've come full circle with no more answers than when I started. For some reason, TMGS seems to compel a greater percentage of people who play it to come back and vote.




I have two more analysis I want to do with this series. My next post will look at voting over time. I say this because the current top 15 new mods are all in the top 33 mods of all-time. Half of the top 10 all-time is within the current new top 15. That seems statistically unlikely unless vote inflation is occurring. My final piece will take a look at the same kinds of stats for NWN1. Perhaps that will shed further light on the expected lifespan of NWN2.

The Holy Grail

Long time readers of this blog may suppose – given my interest in medieval history – that I’m about to write about the Knights Templars or some DaVinci Code crap, but that isn’t the case. Rather, I’m talking about the Holy Grail of CRPGs...

First a bit of history. Back in the 80s, I became enthralled in D&D P&P sessions, and I had a small group that would occasionally play games when we could. By about 1990 or so, that phase had ended for me. My college days focused more on games that could be easily played over the dorm LAN such as Warcraft II, Starcraft, or Fortress Quake, but after I graduated in 1999, I started looking at CRPGs. By 2000 my eye fell on Baldur’s Gate II, and I was instantly captivated by how amazing RPGs had become.

Over the next couple years, I bought BG1, the Icewind Dale series, Planescape: Torment, the NWN series, and even Temple of Elemental Evil, all of which I loved to one extent or another, but Baldur’s Gate has always been the gold standard to me.

I also have always been a D&D homer. This probably stems from my childhood memories, but I can honestly say that my interest in playing Mass Effect, Knights of the Old Republic, and World of Warcraft is zero, and my interest in Dragon Age is just barely north of zero. On the other hand, if a D&D-themed game came out tomorrow, I’d be there to buy it unless the reviews were terrible. And I mean REALLY terrible. I bought ToEE after all.

Needless to say, this makes the current situation between WotC and Atari quite annoying.

So a couple weeks ago, I reloaded BG1 onto my computer, and I must say that while some of it still holds up, it is certainly showing its age. I don’t necessarily think the graphics are all that horrible, although the Infinity Engine is certainly archaic. Rather, the simplicity of the dialogs and quests along with the horrible pathing and blatantly bad AI all make the game frustrating to those accustomed to more recent improvements.

That got me thinking about BG III. I remember a couple years ago – give or take several months – that I was talking to an Ossian-mate about potential new projects when I called BG III the "Holy Grail." For the record, he disagreed and thought something else was... but that’s a story for another time.

But my point was that the BG series is unique in the history of CRPGs, and not just for me. I would imagine that the great majority of CRPG fans / modders would have BG III very high on the list if they were told to make a wish about any project they would want to be green-lighted for. However, with the pending litigation over D&D rights, not only is BG III not a realistic idea for the near future, I don’t even think any serious D&D-themed RPG is. And, no, I don’t count Cryptic’s MMORPG as a serious RPG. Actually, by the time the dust is settled, I doubt a BG III ever comes out. It’s already 11 years since BG II, and I can’t think of many games that had 15 or more years between sequels... although Starcraft comes close.

So then that got me thinking. What if a group of modders just decided to make BG III in, say, the NWN2 toolset? I’m thinking about a major effort along the lines of Misery Stone or Purgatorio. It would certainly be possible, although it would take a lot of dedication from many people with little or no recognition for years. That's tough. After all, Misery Stone, though a great game as is, was admittedly hurried out the door towards the end. I assume the group realized continued interest by the development team was flagging and needed to just get it out. And Purgatorio is, well, in purgatory.

But if it could be done - so long as the game was offered for free - it would be perfectly legal. And if enough word of mouth could be generated, might it even become a somewhat "official" version in the absence of anything else?

But what would BG III look like? What would be the "must haves" without which one couldn’t lay claim to the title? My (probably incomplete) list would look something like:

* Single player campaign
* 80 – 100 hours of gameplay
* A minimum of 10 – 12 NPCs that can be substituted in and out from a party of 6.
* NPC-based quests
* Romances
* A story-line somehow tied in to the Bhaal-spawn legend
* Most of the game taking place in a city environment, although large portions can be outside the city

My gut is that, as the Bhaal-spawn is now either a god or has turned down the Throne of Bhaal and is merely an incredibly powerful mortal, I would think the third chapter would start over with a new protagonist starting at level 1 and perhaps going up to level 10 or so. However, I wouldn’t put this as a "must have" to be a legitimate successor.

I think all of this is more than possible with a two-year development time, so long as the team didn’t have to develop an engine (which it wouldn’t in this scenario). MoW took roughly six months to build once all the design documents were done and another couple months of testing. MoW clocked in at around 20-25 hours and had 3 NPCs. Expanding the six months to two years would lead to 80-100 hours and 12 NPCs. As each of those companions already had an associated quest, this work is already included in the 3 NPCs in 6 months metric. The only additional effort would be to include romances... a topic for a different post. Anyway, yes, the analysis is simple, but it gives a rough order of magnitude. I do think it could be done...

But it won't. All of this is obviously just a bit of day-dreaming on my part. Truthfully, I’ve finally accepted that the BG series is dead and now relegated to the warm, fuzzy, halcyon days of yesteryear... right alongside my memories of the red box set.

But what a pleasant dream it is.