Monday, November 9, 2015

FiMFiction vs. Fanfiction.net: an Absolutely, 100% Definitive Comparison of Authorial Quality

Recently, I've seen a few separate instances of discussion on the relative quality of stories on FiMFiction versus fanfiction.net--notably here, and in a few smaller comments and discussions as well.  It goes without saying that the latter's UI is... not optimal, but what about the user base itself?

Well, my experiences with FF.net have all been pretty negative, but that's purely anecdotal (even if the person anecdoting is me).  What if we tried looking at a random cross-section of stories from FiMFiction, and from various fandoms on FF.net?  What would we find?

I thought I'd find out.  So, head down below the break to see what I did, and what I found out.



To start with, my goals: I want to see how the quality of writing compares when looking at FiMFic vs. a given sub-fandom on FF.net.  I'm not interested in comparing how good the story ideas are, because that's both more subjective, and harder to do at a glance.  I'm also not interested in how easy it is to find stories, interact with the author, or anything else of the sort; those differences are obvious enough without any poking around.  So, how to do that?

I decided that what I wanted to do was to check a random cross-section of fics, and get a rough idea of how many well written/passably written/technically "written" stories I came across.  To that end, here's what I did:

1) I checked how many stories there were in the fandom/site I was searching (as of Saturday morning, US central).

2) I plugged that in as the maximum number on a random number generator, then generated a random digit.

3) I went to that story, using FiMFic's story ID and re-generating the number if it didn't return a story on FiMFic, and by counting back from a list of all stories in that fandom on FF.net and re-generating when I came up with a story that wasn't in English (so technically, a low randomly generated number got me an old story on FiMFic and a new one on FF.net, but the point is that they were selected randomly).  Then, I read the description and first few paragraphs, to get an idea of the general quality of the writing.  Nothing too detailed, and I didn't worry about the plot at all; all I was looking for was a general impression of the construction, nothing more.

4) I did this ten times for each fandom, then wrote up a brief summary of what I found in each story, after which I summarized the site's representative fics into four categories: PROMISING, COMPETENT, TROUBLING, and BARELY LEGIBLE--again, all based on writing quality only--to give us some numbers to compare.






And, here are the results!  First, let's look at FiMFiction:

16595: Termina's Final Twilight

A Zelda crossover that was abandoned after four chapters.  The writing is tell-y and bland, but the editing and spelling are decent, and the description's clear enough (to someone familiar with Zelda, anyway).  COMPETENT

73897: A Princess Save A Crystal Empire

All the story's chapters have been deleted, but if the summary's any indication, this lacked any punctuation, coherence, or other basic elements of the English language.  BARELY LEGIBLE

1117: A New Generation

Another one with no chapters.  The description here shows a questionable understanding of punctuation, but is otherwise a comprehensible (if not promising) description of a far-future OC researching the main six.  COMPETENT

67434: Welcome to Equine University

This story doesn't look like something I'd ever want to read--it's about a high schooler going to the first human/pony joint university to open since Earth and Equestria suddenly merged, and his "five crazy roommates," and on top of that it's got the sex tag--but from a writing perspective, it's not terrible.  The language used it a bit simple and prone to redundancy, which is especially and issue considering the conceit, but the writing quality here definitely isn't the deal-breaker.  COMPETENT

33321: The Nightmares of Yesterday

The description is full of inconsistent capitalization, there are a number of formatting issues, and the start of the story is needlessly confusing regardless of one's familiarity with "dead space 2."  TROUBLING

32520: Derpy Plays Minecraft

There's a lot of repetitive phrasing here (the story uses the word "XBox" four times in the first paragraph alone), and the pacing suggests a summary rather than an actual story, but for all that, it's well-edited and perfectly clear.  COMPETENT

24530: Frozen Heart

Although the spelling and punctuation are there, this story is so perfunctory that it makes Derpy Plays Minecraft look fleshed-out by comparison.  Also, the dialogue is painfully wooden and unnatural, even from a quick glance.  TROUBLING

60719: Dance

Oh, it's nice to see some decent writing after a jaunt through a truly random cross-section of FiMFic.  PROMISING

66589: Fall Of Equestria

No, it's not that FoE; totally different story with the same title, thank goodness.  The story itself doesn't look promising (Luna's long-lost Demon Pony lover asks Twilight to help reunite them), and the writing's not much better.  Ellipses everywhere, a few noticeable editing errors, and dialogue punctuation issues.  TROUBLING

16379: My First Jump

The story is almost impossible to parce right away, as the author elects to go with Pinkie and the narrator breaking the fourth wall for "humor" from the get-go.  The spelling's at least okay, but the author shows only the barest understanding of what a sentence is, and the editing's an absolute mess.  BARELY LEGIBLE

So, final score for FiMFic:

PROMISING: 1
COMPETENT: 4
TROUBLING: 3
BARELY LEGIBLE: 2







Now, let's compare that with a few FF.net fandoms!  I'm going to start with LotR, because that's the one I've got the most personal familiarity with, and I'm curious to see how my prejudices hold up to a random sampling.

49181: Foul Rebirth

The formatting on this is a mess: extra spaces, missing spaces, and the like.  Also, this story uses some bizarre hybrid of script and narrative format with its dialogue, which results in dialogue and character actions wedged together in the same sentence without differentiation.  There are a lot of issues beyond that here; it's just something that particularly bothered me.  BARELY LEGIBLE

23520: Eowyn

An aside: I wanted to copy-paste a sentence from this story into this comment as an example... and then I remembered that FF.net specifically prevents you from doing that, presumably because they want to make sure authors don't get any reviews.  It's kind of impressive that FF.net's UI has actually gotten worse since I was last active on it... anyway, this story: it's a reasonably well-formatted diary-story, and while it doesn't seem to add much to canon at a glance (it's basically Eowyn recounting canon events), it does a reasonable job of capturing its character's attitude and speaking style.  PROMISING

5264: Orëndil

I just have to say that I ended up reading this entire (albeit short) story while trying to figure out what its deal was, and it makes no freaking sense whatsoever; it's basically "the scene from The Prancing Pony, if Legolas had been around and maybe in a relationship with Aragorn, but everything went exactly the same anyway."  Ugh, I'm having flashbacks... but that's not why I'm here.  As far as the writing goes, there's a fair bit of gratuitous elvish, and the use of parentheses in the narration is questionable at best, but beyond that it's reasonably well-constructed, technically.  COMPETENT


Decent spelling and grammar on what's basically a scenefic, but word choice is an issue (I particularly noted the phrase "utilizing his elf-eyes...").  Despite that, it's still easy enough to follow the action, though.  COMPETENT


Another aside: I'd almost forgotten how prevalent those "disclaimer" statements are, outside of FiMFic.  Doesn't really bother me--I did it, too--but once you get used to not seeing them, it's a change... Anyway, the prose is a bit on the purple side in this fic, but considering the source material, I don't consider that a flaw.  In fact, the first few paragraphs here are very solid, and set the tone well for a contemplative after-the-war tale.  PROMISING


I'm not convinced I want to read a story about a Steve Erwin-inspired Legolas, but I can't fault the technical construction here.  The dialogue is very awkward, however, and the narration tends to repeat itself.  COMPETENT


Look, half of this story is the lyrics from a song on the Shrek soundtrack, okay?  I'm glad FiMFic bans this kind of thing, at least--stories in which characters recite (non-original) song lyrics as the fic's raison d'etre are a thing I've never felt needed to exist.  The actual writing is comprehensible, so I can't give this the lowest score I'm using, but sentences like "Passerby's gaped second glances for it couldn't be" (I remembered the copy-paste workaround!) don't inspire a lot of confidence, even when the subject matter is ignored.  TROUBLING


Another thing that I've gotten used to not seeing on FiMFic: author's notes preceding the story.  Sure, you occasionally see them in pony stories, but over on FF.net, it's almost expected.  Those I'm glad not to have; sometimes, a bit of context can help a reader appreciate a story, but in a case like this ("I'm not very good at writing humor fics, but here's one anyway"), I feel the author isn't doing himself any favors.  This story's not well edited--in addition to a few formatting issues, there's a lot of missing punctuation--and it's essentially a list of actions, rather than anything resembling a narrative.  TROUBLING


Ah, slashfiction.  I'm trying to think of what the pony equivalent of Ecthelion/Erestor shipping is.  I guess shipping Smart Cookie and Beauty Brass, since 1) they're both semi-obscure characters, and 2) they lived a thousand or more years apart (technically, Erestor and (the most famous) Ecthelion could have been alive at the same time, but the fic is set in the Fourth Age, and none of the three Ecthelions Tolkien mentions were alive then.  Yes, I looked it up just to make sure; I mean, I assume this fic is about the Ecthelion who slew Gothmog, but it wouldn't do to find out it could've been about some other Ecthelion, would it?).  Glaring historical inconsistency aside, the writing here is quite nice, but is marred by regular editing errors ("Water had proofed to be his doom").  COMPETENT


Here's the first sentence of the story: "Legolas watched her, from across the great hall in Minas Tirth, a deep frown etched upon his face as his eyes stared upon her bright form; she stood proud and regal and looking ahead, not noticing him - not even deigning to give him a slight nod of the head in acknowledgement and he could feel his hands ball in to tight fists at the very sight of her smiling and talking with the other Elvish Lords and not him, wasn't he of enough import?"  Yeah, that entire thing's a single sentence.  It's not even close to being the longest sentence, either, and that's before getting into the dialogue issues.  TROUBLING

PROMISING: 2
COMPETENT: 4
TROUBLING: 3
BARELY LEGIBLE: 1





So, the numbers are pretty close to MLP's, in terms of writing quality, with LotR averaging a titch higher.  Perhaps a different 10 fics would've returned different numbers, but as a (relatively) quick test, it seems to suggest that there's a similar breakdown of good/okay/bad writing between FiMFic and FF.net's LotR community.

But what about other fandoms?  LotR was convenient because I'm familiar with the source material and it has a lot of stories to sort through (about 55k, compared to FiMFic's 90k-ish), but comparing the fandom of a classic piece of literature/popular movie trilogy to that of a show nominally targeted at pre-pubescent girls is a bit apples-to-oranges.  So, I looked for a cartoon series aimed at young girls, with a large adult male fanbase, to see how those results would stack up.







And wouldn't you know, I found one!  Time to look at some Sailor Moon fanfiction!  Target audience: allegedly females age 8-14, though it seems to skew older.  Amount of writing: a bit over 40k fics on FF.net.  Here we go!

9215: 5 Women

Oh, hey, another story that's an excuse to recite song lyrics at me!  Really don't miss that... I'm pretty sure I would hate this story if I had more than the barest of familiarity with the source material (in addition to the fact that it's a lyric-reciting fic, I present without commentary this line from the A/N: "All characters are very OOC, but I did try to use who I thought would fit best"), but just in terms of writing, it's "only" underwhelming.  Poor punctuation, a tendency to use shorthand/textspeak, but nothing actually difficult to parse.  TROUBLING

14023: Sailor Moon Monthly Fanfiction January 09

There's a title that inspires confidence... incidentally, I think I'm getting snarkier as I continue this experiment, but I'm doing my best to keep the ratings consistent.  For science, doncha know.  Anyway, I'm linking to all the stories, so if you're suspicious of my assessments, you can go check them for yourself.  Okay, this fic: the writing's competent in terms of style, but it's got a worrying number of spellcheck errors and run-ons to begin.  TROUBLING

19527: Deal Or No Deal: Anime Edition (!!!!)

If you ever wonder why "normal people" don't take fanfiction seriously, it's because when you say "I write fanfiction," they either think you mean that you write porn, or stuff like this.  BARELY LEGIBLE

30987: Second Glance

The issues with this one are mostly formatting-related: a few paragraphs have multiple speakers, for example.  But the writing's clear, if mercenary in quality.  COMPETENT

4195: Darkness Within

Gah, I hate it when an author tells me their age; it's hard be grumpy about the quality of someone's writing when they're only ten.  I mean, I don't know what my writing looked like when I was ten offhand, but I'll bet dollars to doughnuts it was awful... unfortunately, this devolves pretty quickly into a punctuation-less morass of uncertain tense and undifferentiated blocks of narration/dialogue/sound effects.  BARELY LEGIBLE.

Great, now I feel bad.  Let's move on.

11114: Past Transgressions

With a title like that, I was really hoping for Sailor Nyx, but no such luck.  Okay, serious question: I read all under-200 words of this, and I have no idea what qualifies it as a "story."  Can someone who's more familiar with Sailor Moon explain what context I'm missing, here?  Because... well, whatever.  The writing's got plenty of missing punctuation and mis-wording, but it's not actually hard to follow.  TROUBLING

6900: Escape

Does it say something that the story with the best writing--by far--that I've found so far in this fandom is a BDSM fic?  Probably nothing more interesting than "you've only looked at seven stories so far."  Anyway, the only issue with the writing here is an occasional run-on, and it's not even like we're talking twelve-clause monstrosities or anything.  PROMISING

32653: Star

The description ("uranus main caraters, pluto in it, all new scoutmine Discontinue") suggests a story gunning for a "barely legible."  And boy, does this fic deliver.  BARELY LEGIBLE


A (T-rated!) "questionable consent" fic, thankfully abandoned before it goes anywhere.  My, how edgy; we're really hitting the seamier side of the Sailor Moon fandom, it seems.  Anyway, I'm not judging that (I'm totally judging that right now): I'm just looking at the writing, which is actually pretty solid.  A bit inconsistent in its formality, and with a few unfortunate editing mistakes, but the writing is definitely not the questionable part of this fic.  COMPETENT


Okay, well, it's not a "half the fic is song lyrics" fic... it's a "the characters talk about the author's favorite music" fic!  That's... that's a better thing to be, I guess.  Despite barely knowing a thing about Sailor Moon, I'm pretty sure the voicing of these characters doesn't hold up at all, but the construction isn't bad.  A few run-ons, but not much worse than that.  COMPETENT.

PROMISING: 1
COMPETENT: 3
TROUBLING: 3
BARELY LEGIBLE: 3




So, Sailor Moon's fanfiction didn't hold up, writing-wise, to either of the other fandoms I looked at.  At least, on average, based on ten stories; this isn't exactly a definitive sample size for any of these categories.  Even so, there were a few stories with writing good enough not to immediately scare me off (even if, in many cases, the plot would've).  If we accept the premise that Sailor Moon is an appropriate fandom/source material to compare to FiM, then that would seem to lend credence to the idea that the writers on FiMFic are "better" than should be expected.

Or maybe Sailor Moon and FiM's fanbases aren't comparable, either.  I took my best guess at a "comparable fandom," but in the end it's just that: a guess.  

So, I think it's safe to say there are some technically competent writers on both FiMFic and FF.net; if, as some have contested, there are more on FiMFic, I at least doubt it's by an overwhelming amount, percentage-of-the-fandom-wise.  It's harder (and more subjective) to evaluate plot, but I suspect that writing quality is a reasonable stand-in for evaluating that as well (on a fandom level, not for individual fics); that is to say, I suspect that if there is, in fact, a greater concentration of great ideas in the stories on FiMFic, it's by an at-most marginal amount.

The big difference, of course, is that it's pretty easy for someone who's new to FiMFic to filter out bad stories, and with a little experience, one can find good ones on a regular basis.  On FF.net, neither of those things are true.  And of course, that's without getting into things like access to reviews and writing assistance, ease of author-reader communication, and the like.

It'd be interesting to look at a few more fandoms--or to pull and compare a larger number of stories the three I selected.  But frankly, I've sunk multiple hours into this already, and I think I'm at my limit of random story skimming.  If you want to pull up a few random fics and double-check my results, though, be my guest, and be sure to let me know what you find!

21 comments:

  1. This post is actually quite helpful, because now I can add, "Writing is, on average, better than 10 Sailor Moon stories selected randomly from FF.net," to my resume without feeling like I'm lying.

    ReplyDelete
  2. And there goes my best chance of being 'reviewed' on Chris's blog.

    That's not an "I can't do it" whine, but now I would actually have to earn it, dammit!

    ReplyDelete
  3. I've read a couple fanfiction on FF.net, and God Almighty I don't recommend the experience. Looking for good stuff is almost impossible, you can't highlight shit in case you need to focus your eyes on a particular line (something useful when the author writes walls of text like that) and if you want to leave feedback, then good luck.

    If I have to read stuff outside of FiMFic, it's either AO3 or... It's just AO3, really. And even then it's much, MUCH easier to find good stuff on FiMFic, thanks to the upvotes/downvotes thingy.

    FF.net is a monster. Although my latest interactions with it have been about people stealing my fics and posting them in there under their name, so who knows, maybe I'm biased.

    (The Sailor Moon fandom, by the way, is absolutely fucking bonkers. When I was a little kid I tried to read fanfics about it, and the first story that I found that wasn't completely illegible was about two characters dry-humping to Avril Lavinge for over 3,000 words and then going all "TEEEH HEEH HEEH I THINK YOU LIIIIIKE MEEEE". Well, girl, he went at you like a monkey goes at a banana, so I think we're safe to assume that).

    ReplyDelete
  4. Woo hoo! As soon as I saw the line about wanting to find a more comparable fandom than LotR, I guessed Sailor Moon, and then I scrolled down to find all my dreams had come true!

    I actually did enjoy watching Sailor Moon, though I never saw all of it. I thought it had pretty good character work, and while the day-to-day plot was formulaic and repetitive, the overarching one could be surprisingly elaborate and inventive. I never ventured into the fan community, though, limiting my involvement to simply watching the show. I started out watching it to make fun of it, but it really ended up appealing to me.

    Now I just need to catch up on the episodes I missed (I never saw any of season 5, which is the only one watchcartoononline doesn't have!), and I need to start watching the new series.

    ReplyDelete
    Replies
    1. Kinda surprised he didn't go with Powerpuff Girls, honestly, but I suppose Sailor Moon works too. I was never part of that fandom, but I adored that show as a kid (Lita's best Scout!). It was one of my first Japanese cartoons, after Maya the Honey Bee and possibly Speed Racer. Tried re-watching it awhile back, but it didn't hold up. Had better luck with the comics, though

      Delete
  5. NEED MORE DATA!

    This sounds like an excellent way to produce content while shirking your other, more important duties, and I am all for it. :D

    ReplyDelete
    Replies
    1. It would be an excellent way of shirking my other, more important duties... except that this post took longer to pull together than any review post, and probably still beats out most of the reviews of shorter fics even if we include reading time!

      Delete
  6. One thing I wonder about is what ff.net's MLP fanfic is like in quality terms. Is the fiction that's there and not on Fimfiction mostly stuff that couldn't pass moderation on Fimfiction? I can't see why someone would post on ff.net for preference in this fandom, but I don't know for sure.

    ReplyDelete
    Replies
    1. Without having gone into it too deeply, my impression is that much, perhaps the majority, of the MLP material on FF.net is also on FiMFiction (much as is/was the case with the Pony Fiction Archive). As for the portion that's only available on that site? I couldn't say, but it'd be interesting to know.

      Delete
    2. There are a few older fanfics that were written before FimFic really rose to prominence that still haven't made the transition. They were usually about average quality for a 2011 fanfic, but that's just speaking for the ones that I've actually read, which bear in mind, I was directed to mostly by recommendation, so they probably don't actually represent the average.

      Delete
  7. So, sorry to say this, but we need to bring in some math to this discussion.

    With a population of ~60k, and only 10 samples taken, you have a margin of error of 30% (you can do some calculations here: http://www.surveysystem.com/sscalc.htm). This means that, if had selected 10 different stories, there is a 30% chance you would've gotten different results

    This is while at the same time, if we assigned 1-4 scores on your 4 categories, we get averages of:

    FiMFic: 2.4
    LoTR: 2.7
    SM: 2.2

    These values are within 19% of each other, with FiMFic and SM being within 9% of each other (the difference being only that one story got BARELY LEGIBLE in SM, and another got COMPETENT in FiMFic)

    As much as I was entertained by reading your post, you should recognize that it has no statistical significance whatsoever, and no conclusions can or should be drawn from it. I think it would be valiant to state that clearly in the post body itself.

    If you wanted to do anything close to significant evaluation, you'd need to go up to 50 fics of each fandom for 15% margin of error. And, when selecting a story for FF, you should subtract the random number you're getting from the total number of fics, so that your selection is not confounded on different fic ages for the same random distribution.

    ReplyDelete
  8. And the wet blanket has been deployed!

    ReplyDelete
    Replies
    1. I am sorry if it read that way :(

      The post itself is very entertaining. I love reading Chris's opinions on things, and it was fascinating to get impressions on fics from other fandoms.

      I just wanted to highlight that we shouldn't be making judgment calls on quality based on that. It's important to know how close we are to actual significant conclusions. My analysis was also very broad and simple, in reality there are even more confounding variables, and I'd wager that the actual margin of error is probably larger than 30%.

      Speculation is interesting, but we should take it just as that - speculation. It seemed as if some people were taking it as evidence for objective difference in quality of writing, which it just can't be.

      ... OK, yeah, I see what you mean by wet blanket.

      Delete
    2. As a statistician, I have to agree with Crafty Shadow. I don't like it when my discipline is inappropriately used to make a point, even for the most trivial of things. I also don't like the fact that the MLP sample included a few stories that had their chapters removed (they should be thrown out).

      Delete
    3. I'm not a statistician, but I still agree with you (though I think Pascoite was just joking). The fics with missing chapters bugged me too

      Does anyone have the free-time and knowledge to make a more serious attempt at this? I love the concept

      Congratulations, Chris, you're the new Dr. Kinsey!

      Delete
    4. Chris is just having a bit of fun here. I don't think he actually purports to find any objective conclusions. If he had found something drastic, like every single FF.net story was barely legible, then maybe. All this tells me is that there isn't clear evidence of a significant difference, which is probably about what I would have expected anyway.

      Delete
    5. As Pasco says, I was just having a bit of fun with this one; I'm well aware that ten samples is more or less the opposite of "significant" in this context. I'd hoped the title would make that clear, but sarcasm doesn't always travel well over the internet. Anyway, lesson learned: don't make light of statistics, lest you incur the wrath of their dark practitioners!

      And Bugs, Crafty: if either of you decide to do an actually statistically significant analysis, let me know. Although I had fun doing this, I'd be interested to see some more reputable facts on the matter!

      Delete
  9. If anyone would happen to know how to directly download a fic from ff.net, then I could generate some kind of automatic metric for this involving, for instance, the ratio of simple spellcheck errors. It would require some dictionary work to include fandom-specific terms, however.

    ReplyDelete
    Replies
    1. I'm not aware of more specialized software, but I use Calibre ebook reader and its FanFicFare addon to download fics from most sites; it mostly works around the EPUB format, but it sure works with ff.net.

      Delete
  10. So does this mean you will start reviewing mlp stories from FF.net?

    ReplyDelete
  11. Thank you for doing this so I don't have to!

    I was hoping you'd compare ponyfics on fanfiction.net, though. To compare the websites, not the fandoms.

    ReplyDelete