Author Income Surveys Are Misleading and Flawed—And Focus on the Wrong Message for Writers

From Jane Friedman:

Reliably, every year or so, you’ll see headlines about new research that claims author incomes are on the decline. The most recent is from the Authors’ Licensing and Collecting Society (ALCS), a British nonprofit run by writers for writers.

. . . .

Before I continue with this epic-length post, here’s the short version: I don’t trust these surveys’ results and I question their usefulness in improving the fortunes of writers. Too often it feels like promotion of a self-interested narrative from writers’ organizations, with the outcome boring and predictable: There is media coverage that claims writers’ incomes are plummeting, a few big-name authors come out and try to shame publishers or even society for not valuing writers properly, debate ensues, then everyone gets back to work—until a new study emerges.

This latest gnashing of teeth has motivated me to finally write a comprehensive post about why these reports are so frustrating, in the hopes more people will ask critical questions and notice their flaws. In the long run, I hope organizations will either reassess how these studies get done, or focus on more useful support of professional authors. However, a brief side note for industry insiders: For the purposes of this article, I’m setting aside the fact this research may be done mainly to support arguments and legislation for strengthening and protecting of authors’ copyright and thus (presumably) their earnings potential. Frankly, I don’t think weak copyright law is the problem, and I believe such efforts have little effect on the average author. My thinking on copyright aligns with what you might hear from Cory Doctorow—but that’s a post for another day.

When considering author-income research, my concerns fall into these areas.

  1. How reliable is the research?
  2. What other data points do we have?
  3. If the data is directionally correct, then why are incomes declining?
  4. Does the research tell us something meaningful or useful about how writers earn a living?

Link to the rest at Jane Friedman

Back in the dawn of the modern era, PG graduated from college. His favorite professor told him he needed to find a real job and sent him to the university placement center. Lo and behold, a couple of weeks later, he had a real job. For more than minimum wage.

That job was as an economic and marketing research analyst for a large financial services company. Suffice to say, PG had a lot to learn as his formal education lacked any mention of economic and marketing research.

One of the things PG learned is that most surveys of this or that group are garbage. The reasons are simple. If you wish to learn something real about a large group of people, you have two choices:

  1. Query every single person in the group in a way that will result in an accurate answer.
  2. Carefully select a truly random sample that accurately represents the group and query them in a way that will result in an accurate answer.

Both approaches are expensive and require a lot of work – by people who know how to create questions that will result in accurate answers. Real random samples assume you understand the constituent qualities of the entire group very well and are able to obtain answers from respondents that include appropriate numbers of those with each of the constituent qualities. (If your group contains 50% women and 50% men, you may be in trouble if 80% of responses are from women. Ditto if your group contains 50% native English speakers and 50% non-native English speakers or if your group contains 50% who earn less than $10,000 per year and 50% who earn more than $10,000 per year.)

The surveying process can go wrong in many different ways. These ways include:

  1. Obtaining answers from people who aren’t part of the target group.
  2. Failing to obtain significant numbers of answers from people who are part of the group.
  3. Obtaining answers to ambiguous questions.
  4. Asking questions that are likely to generate answers that are guesses.
  5. Using the cheapest method of obtaining some sort of responses instead of the best method of obtaining accurate answers.

Quite often, it is useful to conduct some preparatory research (usually by talking to people) to understand how they view the subject of the research and to help map the types of responses people in the target demographic are likely to give when asked about the subject generally or specific sub-parts of the subject. If you create a multiple-choice question that doesn’t include all the meaningful alternative answers for the target audience, that’s one way the survey can fail and result in inaccurate results.

Here are a couple of example of bad multiple choice questions:

You indicated that you eat at Burger King less than once every month. Why don’t you eat at Burger King more often? (choose one answer):

  •  There are no Burger King restaurants near my house
  •  I became sick after eating at a Burger King
  •  I’ve never eaten at Burger King

What is your overall opinion of the Authors’ Licensing and Collecting Society?

  •  Pretty good
  •  Great
  •  Fantastic
  •  Incredible
  •  The Best Ever

The OP discusses other problems with 99.999% of surveys of author income. In short, poor surveys of author income are probably less useful than collecting anecdotal evidence of author income (in part because people are likely to be suspicious of anectodal evidence).

22 thoughts on “Author Income Surveys Are Misleading and Flawed—And Focus on the Wrong Message for Writers”

  1. A very basic question is, “What is a writer?” How do we define the target group from which a random sample will be drawn? What characteristics make one a member of the target set?

  2. I’ll add a problem to the methodology — assuming that A has any causation rather than mere correlation with B, for many reasons. I’ll go sideways to financial products, which are analysed out the wazoo by millions of companies around the world.

    First, there is a reason why every regulated financial product says “past results are not a prediction / guarantee of future performance”. Snapshots are occasionally useful but they don’t necessarily reveal a trend.

    Second, if there is some other reason — like the stock market boomed — then the management of that mutual fund may have had very little to do with the outcome.

    Third, how was the analysis conducted? Was it “bias-neutral” or was it “leading/narrowing”?

    My favorite example is internet usage and whether people will “cut it” during recessions…the questions that are often done are of the form:

    Are you concerned about how much money you have?
    Are you concerned about the recent downturn in the economy?
    Are you considering any reductions in luxury goods?
    Are you considering scaling back any unnecessary expenses?
    Would you consider scaling back your entertainment expenses?
    WOuld you consider cutting back on your internet costs?

    Conclusion — 98% of people surveyed said they might cut back on internet costs.

    Do you think children need to be prepared for the the skills of the future?
    Do you think digital skills are important for work and school?
    Do you think understanding IT and the internet is a necessity for people to succeed in the new economy?
    Do you think kids with internet at home have an advantage over kids who don’t?
    Do you try to help your children succeed?
    Would you consider internet a luxury or a necessity in today’s world?
    Would you cut back on your internet at home or would you cut back on other entertainment expenses?

    Conclusion: 98% of people would cut entertainment before cutting the internet

    My brain isn’t quite as honed today, as the second case is done roughly, but some of the surveys are that badly worded.

    To say “author incomes are declining”, you have to define author, income, are and declining!

    a. AUTHOR: Who’s in / who’s out of the sample — Aunt Edna who threw up a book of cat poems? Or only writers where writing is their primary occupation? Rapidly expanding newbies or someone in the biz longer than a week? And who are you choosing to ask/answer?

    b. INCOMES: What constitutes income — income from publishers, income from self-publishing, income from speaking gigs related to writing, income from web-freelance, income from secondary jobs (cuz if you work PT and lost that job, your overall income would go down, just not your publishing income)

    c. ARE: What timeframe — short-tail, long-tail? Over three-year rolling totals or just within the third week of April?

    d. DECLINING: While that sounds easy, the real question is how would you know? To answer it, the person has to tell you their income before and after a certain point. A survey will almost never do that accurately unless the surveyor is accessing your income tax return. Self-reported income in surveys has very little semblance to actual income on income tax forms. I work in an industry where we need to know the impact of our activities on income, and we have basically eliminated all sources of income data except government data. It’s the only near-universal source that is reliable, and even it has problems with self-reported incomes.

    In my day job, part of my task is to analyse crap surveys like this and see if there is any value anywhere in it. About the only thing it tells us is perception of the respondents and whether that nudges their economic behaviour right or left…Or, is it like law school? Students are going to enrol even when they know there are no jobs, and writers are going to write, whether they think they can make a living or not, because “I’m a population of one, not a random sample from a larger pool”.


    • +10,000 (Since PG has no voting, I can do it as many times as I feel like!)

      Although your best is last:

      “I’m a population of one, not a random sample from a larger pool”


  3. I have to mention one of the most frequently seen problems with even ‘professionally’ done surveys from organisations such as YouGov, which is questions which are designed to elicit the response that the pollster (or its employer) wants, ofttimes by the order of the questioning. It is rife particularly in political polling.

    • We see this in reading/publishing surveys all the time.

      Especially the ones that try to show digital as a fad because younger readers “prefer” print for their *textbooks*. (Of course, the fact that textbook readers, who have no choice of format in the k-12 space, are in the sample is buried deep in the reporting. Or not addressed at all.)

  4. I think our American friends misunderstand these surveys somewhat, which, incidentally, I broadly support, albeit with the usual reservations about all surveys. (Although I agree with J R above about political surveys.) The ALCS is an efficient organisation and the surveys do allow for a breadth of opinion. I don’t think they are using them to make copyright more draconian, although they are certainly using them to obtain some kind of general picture about professional writers and to defend the rights of full time writers to be paid fairly for what they do. Just as a recent example: a popular women’s magazine over here has changed hands and started to (a) claim worldwide rights in the stories they buy and (b) pay even less for the rights grab involved. The Society of Authors over here, which conducts some of these surveys, with a professional membership, is doing its best to defend writers. The fact remains that professional author earnings in the UK, both anecdotally and according to surveys, are diminishing drastically. Most of us do many other things to earn a living. Given my time over, I would make damn sure those other things were my main job and paid me well, according to the excellent qualifications I have. But the undoubted fact remains that everyone in the publishing and media companies earns considerably more than the writers without whom there would be nothing. And yet even the best of them pretend it isn’t so. The late great Harlan Ellison had it right. For universities and colleges to even begin to suggest that there is money to be made by writing – with the courses taught by writers who patently can’t make any money from writing!- is to conduct a mass deception of epic proportions. If these surveys can deter even one person from believing that they can make any kind of living from writing in the current climate, they will have done their job. And that includes self publishing here in the UK. Buying a weekly lottery ticket would be a much better bet. And yes, I know that you’re more likely to die on your way to buy the ticket than you are to win. I suspect the odds of making any kind of living from writing these days here in the UK, are similar. I can’t speak for the US.

    • The lot of writers in the UK or US tells us nothing about the statistical rigor of the surveys. It doesn’t matter if the survey is for writers or hedge fund managers, a poorly designed and executed survey is unreliable. Nor do good intentions of the surveyors or beneficial outcomes from the survey tell us anything about the reliability of the survey.

      • Or the damage that those meaningless surveys produce.
        If those surveys are the best they can do they’d be more helpful doing nothing.

        Bad information is way worse than no information.
        Or as they say: “…paved with good intentions.”

    • I think our friends across the pond are too used to the government solving all their problems for them (not that I think they ever actually have). Tradpub is never going to pay you a living wage. Stop waiting for a law that fixes that and start writing for yourself.

      • The government *pretending* to solve problems.

        Still, in this case what they are asking for is something modern governments do all the time: use the state’s monopoly on the legal use of force to take from one person/group to give to another.

        With it happening so often, surely they are as entitled as anybody to ask why not them, right?

        Of course, if the UK politicians are anything like the ones on this side, we know the answer to that question: they’ve been outbid in the influence peddling game.

        Surely there is no harm in asking, and asking and asking and asking and if they keep begging maybe one of these days Oliver Twist will get a bit more. Or maybe not.

        As you say, modern corporate publishing is a hopeless game so the best answer is not to play at all. Just move on to other, newer channels and leave the legacy business to legacy authors.

  5. Let me throw this out.

    Maybe the reason writers aren’t making a living is because the stuff they produce is poorly written, badly edited, pushes messages, and most importantly, isn’t entertaining.

    • …or aimed at the wrong audience through the wrong distribution channels.

      In the movie business we are seeing a revival in romcoms, “small dramas”, and other “midlist” movies that hollywood and the theater chains won’t touch because the audiences stopped going there.

      The revival is coming via Netflix, Prime, Hulu…

      If John Hughes were working today, he’d be working though the streamers, not Hollywood. After all, his movies’ (adjusted for inflation) average gross was “only” $118M ler movie.

      Times change and audiences change.
      Just ask B&N.

  6. If I may add another concern, based on a recent survey that I failed to complete…

    Some surveys are just too long, too nosy and too time consuming. My suspicion is that these are more likely to be completed by individuals who either love taking surveys (who?) or have fairly simple financial pictures. Thus, skewed results.

    This one said it would take less than an hour to complete, then asked for five years of information not only on my writing income but also on my other income and my family (read my husband’s plus mine) income. Really? Digging out old income tax forms, figuring out what constitutes income (do I subtract expenses?) and working out percentages–which were requested–would take me hours of frustration. Not to mention that my family income has nothing to do with my writing income and is nobody’s business except ours and the tax agency’s (arguably).

    The publishing world is changing by the hour, including for those of us both traditionally published and self-published. Yes, I do think writing incomes for professionals are declining. But I have no idea who should be blamed, if anyone, or what should be done about it.

    • The problem is that looking at bulk numbers (average, mean, whatever) across all of “publishing” obscures and insight that might be found in more granular data.

      A survey by the RWA of SFWA is bound to be more useful than a survey that conflates both genres. Add in litfic, mysteries, thrillers, and non fiction and there isn’t any insight left. Are readers moving to video? Gaming? Are the switching preferred genre? Even if they are, it is because of life issues, a lack of compelling offerings, or are they simply getting their reads elsewhere? KU, public libraries, their TBA lists?

      (In the old days when books went out of print, people would buy intriguing books on sight, afraid of them going out of print, To Be Read at some future date. That future date might be today and buying might resume once the TBD pile is reduced.)

      Lots of reasonable possibilities but looking at the supplier side isn’t going to say much about the consumer side.

      As PG suggested, these surveys ask the wrong questions of the wrong people.

    • I think I started that survey too, Jacqueline, and when I got to the “net income” and other financial questions, I bailed out. That’s information that does not need to be out on the Internet, and is far too private for a polling group to have, even one affiliated with an otherwise honest organization.

      • I’m glad I’m not the only one! I hope the Authors Guild, RWA and other participating groups take note and pare those questions. I can understand them needing to know the number of hours per week that we worked and our total writing income, but not our other earnings!

      • I took that survey we’re all likely talking about. I keep my numbers easily accessible, but I can imagine it was a pain for many.

        I found some of the questions had problems, and spent more time writing up what I thought was wrong (in the handy response field they provided) than I took filling out the multi-page original.

        A couple of days later, the announcements for that survey mentioned improvements in some of the questions. Who knows, maybe my little bit did help.

      • I also bailed out of that survey when it got deep into financial information, but not only because it asked for such detailed information. What bothered me was the assurance that all data would remain anonymous and confidential, and yet an early question asked for the ISBN or ASIN of your latest book. It wouldn’t take a whole lot of effort to link that to your name, and thus every answer you gave.

      • I gotta wonder if many of these so-called “surveys” are simply attempts to get as much personal info as possible, especially if the questions really have nothing to do with the purported intent of the “survey.”

        And then I wonder just what nefarious purposes that sensitive personal info provided by naive and gullible survey-responders will be used for.

  7. I stumbled on an example of the OP’s “self interested narrative” only yesterday on Oz TV. A headline suggested that employers were stealing wages from employees who were left out of pocket with nowhere to turn for help. Only when the presenter started interviewing union representatives did the penny/cent drop. The union movement is running an ad campaign here to attract new members. This story was part of that SIN. See also summer stories about “plagues” of mice, roaches etc. with source quotes from a pest control company buried deep within the article.

  8. I share PG’s skepticism of surveys. I’ve written software to conduct surveys, written survey questions, and analyzed survey results. I am skeptical for all PG’s reasons, and a few of my own.

    Creating and conducting a survey that rises above worthless is difficult. The sample has to be made meticulously. Questions not only have to be composed carefully, they have to be tested for effectiveness. The results have to be analyzed with great care. All this is possible, but it takes time and is expensive.

    In the world I have experienced, both time and funds are always in short supply.

    Further, too often, surveys are conducted to confirm someone’s established belief, not to produce useful information. You can lose your job if you conduct a survey that produces the “wrong” answer. Under pressure, you can easily produce desired results without even realizing you are doing it.

    I won’t go so far as to say that surveys are worthless, but unless I know a lot about how a survey was conducted, who conducted it, and the motivation for the survey, I seldom grant them much credence.

    Unless, of course, a survey confirms my beliefs or suspicions, in which case I believe it completely and without question.

  9. Surveys really are the worst. It seems like most people who write them don’t have any idea how to do it well. I particularly hate those who ask questions in such a way to get you to “agree” to something you don’t actually agree to (like when my workplace asked everyone if they like natural light, then used our affirmative answers to tell us that we’d agreed that we’d really like shorter cubicle walls) or when the survey is clearly biased toward one line of answers, like when I got a “survey” call from the union a couple years ago when there were rumblings about our state becoming “right to work” (paraphrased):
    Union rep: “If you were allowed to stop us from taking your money, would you do so?”
    Me: “Yes, in a heartbeat. I would absolutely stop letting you take my money.”
    Union rep: “Would you commit right now to continue giving us money even if you aren’t forced to?”
    Me: “Is that seriously your follow-up question?”

Comments are closed.