The end of ebook development
From Baldur Bjarnason:
A bit of an utopian fantasy but, I think, an essential one. We need visions of where we’re heading with this.
Everybody knows what it feels like. You’re thrown from crisis to crisis, locked into solving problems, resolving issues, keeping things from breaking down, grasping the duct tape with one hand, and your sanity with the other.
Every time you look up, you see more trouble coming your way and no time to avoid it.
It’s hard to think about long-term problems …
The publishing and ebook industry is completely in reaction mode. Publishers reacted to Amazon by colluding to set up the agency system. Kobo and B&N reacted to Amazon by mimicking its strategy.
…
The only two companies that seem to have a plan and try to act on it are Amazon and Apple, both tech companies. Unsurprisingly, they are the ones who are going to control the future of ebook publishing. Unless there are dramatic changes the rest of us can do very little except play along and accept that it’s their playground and their toys, leave, or have the patience to wait the decade it might take for somebody to disrupt them.
Or, we could stop reacting to what they are doing, take a moment, and figure out what the ebook industry should look like. Picture the ideal and then figure out how to get there, step by step.
…
How do we want ebooks to be made?
- One button publishing.
- Themes.
- Design tools.
- Developers.
1. One button publishing
A writer should be able to open up a Scrivener or Word document – one that has been thrown back and forth between the writer and the editor until both are satisfied – click on something like “Export to EPUB” and have a ready-made EPUB file that works everywhere.
…
2. Themes.
Even though most books would be served by basic styles, there are a number of books that require a visual aesthetic with specific characteristics for the book to work.
…
To accomplish this we only need to add a single one-time step to our one-button process: Write, click export, and pick a theme.
…
3. Design tools
Then we have books that can’t be served by simply hanging a theme off a solid structure. These books need proper design. Or, if they don’t need it, they’re getting it because the author is a neurotic designer who is compensating for their inadequacies as a writer by making pap look gorgeous.
That’s where we get to the next level in the tools hierarchy: proper, fully featured, extensive design tools.
…
4. Developers
In this world there wouldn’t be any ebook developers any more than you have .doc developers.
The fact that we have an industry of people whose job description is close to indistinguishable from ‘fixes office documents by hand in a hex editor’ is insane.
The only developers the ebook industry should have are tool developers, people who program and make the writing programs that export the ebooks, theming apps that add themes to ebooks, and the design tools for the edge cases.
…
Please, go read the rest of Bjarnason’s vision for the ebook industry. This is probably the most insightful thing I’ve read about where we need to go with ebook development. I’d like to tip my hat to Paul Salvette of BB eBooks where I found the link to this article.
I have a few ideas about how to make this vision a reality, but I want to talk about the key first step today. If we want to get to one button publishing in the near future, we’re going to need an intermediate format between Word and Scrivener (or your tool of choice) and ebook formats. The reason is the plethora of hardware and software for ebook reading. It’s not just that there is Amazon’s formats (KF7 for non-Fire devices and KF8 for Fire) and ePub. It’s that Amazon’s eInk devices and apps for the various platforms all have slightly different and completely undocumented capabilities and behaviors, which can change when they invisibly update the software.
Did anybody notice when the Kindle app for the iPad started using the KF8 format instead of the older format? Because that happened (although the BISG doesn’t seem to know that yet) and your books might look different today on the iPad. And there is the fact that nobody completely supports the ePub 3.0 standard (as of 26-June-2012). I’m not even sure how to spell it. Is it ePub or EPUB? Don’t even get me started on trying to figure out what part of Unicode each platform supports.
In the crazy world of ebook formats, we either stick to the lowest common denominator (which is barely above simple HTML and a cover graphic) or convert to a format that can:
- Preserve the logical structure of your book.
- Be transformed to many different output formats.
- Be converted reliably and automatically back and forth to formats readable by writers’ tools (like Word and Scrivener).
- Be invisible to writers, editors, designers, etc.
- Serve as a foundation for the collaborative workflow required to bring an ebook from manuscript to completion
Nothing like that exists today, so I’m going to create it. If you are a writer who has self-published an ebook, you can help by answering a few questions in my survey for ebook formatting.
-William Ockham
Amazon, Apple, Disruptive Innovation, Ebook/Ereader Technical, Ebooks, Self-Publishing, Tablets

Please no. People have too many options right now. Two weeks ago I found some mad author who had written his entire book in courier new. It was impossible to read. Then there’s the one zillion books with an extra space between paragraphs when combined with 1.5 or, worse, double spacing. Talk about a lot of white space! Oh and don’t get me started with no indents, justified text or some other lunacy. Please take the metal fork away, little Tommy is stabbing himself (and me).
Also, ‘one button’ publishing? What’s that? Is that where I upload a file without filing in a title, genre, price, description, and so on, or just one button on the page (such as Smashwords). Could you image all the books called ‘manuscript v1.7′ for $50 (set by e-store) in the thriller genre for the prior? What a wonderful world. Maybe it should be ‘fill in a form and publish’. Much more clear, [Making that form faster, easier and better to understand should be the main job - not reinventing the wheel.]
“Themes and Design Tools”. Hmmmm… Super eye bleeding bold with Hello Kitty background, anyone? Tech people need to stop and think. We don’t need more technology. We need a clear manuscript format that produces a ‘standardized reading experience’. That’s step #1 and techies aren’t going to give us that. Readers will when they make it abundantly clear with their money that if an author gets creative with format they are done for. Only then will authors wise up and blog together to fight it out for universal format (once the noise gets loud enough the old and out of date posts will die and the young will only learn the one true way. Then and only then will the golden age of readability begin). If tech people are going to help they need to make a shock therapy tool that also screams NO when an author makes a ‘stylistic’ formating choice. (This also sounds like a money grab from software people.)
‘Developers’. Hmm…. don’t the e-stores have those? Why would I want more people messing with the great universal standard that will bring forth the golden goose? Let’s stop right there and say publishing at Amazon and friends is not like being an app developer for Apple. We don’t want people getting in on the game just to make money of us. We don’t need nor want that. Let the e-stores pay their software people to do what they do. If need anyone else we’ll get a ‘doc jockie’ to clean up or MS because we messed it up somewhere. (The whole thing screams we want in! Where’s our money!)
(Note: I’m sorry if the last part doesn’t gel well with what people feel they are doing, but think about this: Authors have had to prove they are worthy for nearly 20 years now to the publishing industry and publishers, editors and more are now having to do the same in reverse. Shouldn’t developers also have to show EXACTLY how we, the authors and consumers of their products, will benefit? Remember we have had to prove it everyday of our writing l lives. That sort of world is cut-throat and, although there are a lot of foolish newbies – not all – we are a talkative lot. We will not tolerate useless toys that do nothing more than we could do ourselves with a bit of research and effort.)
P.s., I think Scriver is ok. I don’t use it because I don’t need the export functions and the interface is half mast, but I respect it as one of the rare decent pieces of technology that a developer has produced for the industry – if not as easy to use as MS Word. Then again, on all of the devices I’ve tested on, thanks to a few device happy friends, my formating style works. No issues ever. Lucky me!
“It’s crazy that we don’t have one ebook file format that works in all readers.” (From post).
Not really. It’s called a competitive advantage. It locks consumers to one device and one store. If one makes the foolish choice to buy a Nook, they are tied to B&N or killing DRM. They can’t just move to Amazon when they realize how much better Amazon is as a store. No, they are going to keep buying from B&N until they accept how bad the e-store is and no longer care. This is the way of the future and of business. ‘Until it breaks shall we part.’
Or they learn how to use Calibre to strip the DRM, then convert to epub and side-load… I hear that it’s easy if you know where to find the right tools.
That’s what made me an ebook buyer again after getting screwed by DRM and an outdated format more than a decade ago, which is the last time I bought ebooks.
First, when you say, “We don’t need more technology. We need a clear manuscript format that produces a ‘standardized reading experience’”, you are contradicting yourself. Without new technology, you will never get anything like a standardized reading experience. Just because people can use technology for stupid things, doesn’t mean the technology is stupid. And I really wish that you wouldn’t assume that I’m stupid. There is a big difference between publishing and distribution. Nobody wants a bunch of ebooks with bad metadata. I would like to reduce the useless friction between writers and their ability to create books in a format suitable for distribution to any ebook vendor.
Still, I think we are in much more agreement that you think. All of the things that you complain about in your first paragraph can be fixed without author intervention, if we have the right tools.
I’ll just have to build some stuff to prove it to people like you. And that’s not criticism, either. Writers should be appropriately skeptical of software developers.
I happen to think that a single standard format for ebooks would be better for everyone, but I certainly understand why Amazon believes differently. Market leaders often do. I don’t understand why all its competitors haven’t figured out that they should really rally around ePub 3.0. In any event, we will always have different hardware devices with differing capabilities.
I’m totally in agreement with allowing self-publishers to choose their own themes, no matter how garish, ugly, and unreadable. That way, people like me, who have been lecturing bloggers that USING REVERSE TYPE AS YOUR MAIN TEXT MAKES IT UNREADABLE for years will have an advantage in the ebook world.
I’m also very much in favor of one ebook publishing standard, but we all know that’s not going to come for quite a while. Sony Betamax, anyone? Internet Explorer browser ignoring HTML standards? What? The ebook world is doing that too? The devil you say!
That said, I wish the manufacturers would hurry up and get the e-reader wars over with. That will make it much easier for authors and publishers.
People pry my Override Publisher’s Formatting With My Own Prefs abilities (and apps) from my cold, dead fingers — and even then, I might spontaneously become undead just to clutch those apps the harder!
Said it before, and will say it again: the only book I’ve seen that needs more than one font is The Interior Life. Anything else I’ve read does fine in MY chosen font, MY chosen paragraph indentation, and MY chosen paragraph line-spacing.
I totally agree that users should be able to override publisher styling with their own. The “C” in “CSS” stands for “Cascading” and was intended to solve this problem. It is supposed to work like this (greatly simplified):
1. The browser (an ereader in this case) comes with a built-in stylesheet which the user can customize.
2. The html page (an ebook) can have its own stylesheet which overrides the default UNLESS
3. The user marks the style element as “Important”, in which case the user’s choice wins.
[There's a whole bunch of other rules governing specificity, order, and other factors to "break ties".]
The issue with ebook readers is that they impose their own styles which neither the user nor the publisher can override. If these were due to technical limitations, I might understand, but in most cases, it is just “because we said so”.
I grew to mistrust CSS when it first appeared — seemed like all the page-coders “importanted” their own stuff, overriding the user’s prefs…
It doesn’t surprise me that hardware e-readers make shortcuts that take scary, confusing choice away. Yet another reason why I likes my iPad and my multitude of e-reader apps. (I need to finish going through my iPhone e-reader apps and fill out the spreadsheet data about them. Then I need to figure out how to make the data available, besides an excel sheet.)
Yes, I wish vendors would standardize on ePub3 and also cover art. Apple is now looking for high resolution images for its store.
Although I’ve read positive things about KF8 it’s hard to be excited about it because it’s usable only at Amazon. Same thing with Apple’s new ibooks format.
I’m a bit in agreement about the whole idea of a standardized format; however, I think Pike didn’t really get the central point of the article. I also don’t think he thinks you are stupid Will – more likely he just doesn’t agree with what he thought the article was about. It should be interesting to see how things turn out in the format wars. (nods)
Jason,
You are probably correct. Point taken.
-WO
I have lots to contribute to this, but not the time today. The only thing I’ll add is that recent update to Kindle for iPad was not a complete implementation of KF8 support. It was a half-step, at best, so that the device can play Amazon’s fixed-layout books and comics. Everything else that KF8 offers has not yet been implemented. In that respect, BISG is correct.
You are correct that the KF8 implementation is only partial. Even worse, it doesn’t appear to be documented anywhere exactly what works and what doesn’t.
All of this is beyond me, but I would certainly like to see simpler ways to format in the future. I think this will happen no matter what, simply because that’s how it seems to go with progress, but again, I’m not much into formatting knowledge and/or tech other than the basics.
Agreed.
“1. One button publishing
A writer should be able to open up a Scrivener or Word document – one that has been thrown back and forth between the writer and the editor until both are satisfied – click on something like “Export to EPUB” and have a ready-made EPUB file that works everywhere.”
WriteWayPro does this. Publish direct to Epub. I find it a bunch more intuitive than Scrivener to use, and it has lots of easy to use and find tools. It is also $35 right now, cos there’s some kind of discount going on.
I don’t work for ‘em, or get any kind of benefit out of mentioning them. Just a user of the software.
brendan
I would like a standard.
If you can make a PDF with one click from a Word doc why can’t you make an epub or mobi?
Why does everything get so corrupted along the way that you have warnings “You’ll have to know html to fix your epub file.” Good luck seeing through all the junk Microscoff adds to the html code less than o:p greater than less than /o:p greater than instead of less than p greater than less than /p greater than. Why why why.
Baldur’s article is really great. He has other similar pieces over on his site that are part angry rant-part genius. eBook design and formatting is a total disaster at the moment. My biggest gripe is one of William’s – that the big tech companies all have different standards and they provide absolutely zero support or documentation to developers, authors, and publishers. Our company has figured out how to do design eBooks strictly on trial and error–not a very good way to approach a fast-changing technology. Web designers are making lightning bolts fly out of your iPhone with ease, but getting a pullquote properly aligned in an eBook is a major challenge.
I’m a bit skeptical on one-click publishing, since you’ll be locked into the format specifications of whatever company designed the software. For William’s format recommendation, I would like to humbly suggest a #6 for a format that focuses on open standards. Even though there is EPUB, the metadata structure is a complete mess, with differing companies and organizations with different agendas rewriting the spec. Things can only get better, because they can’t get any worse.
I filled out the survey on ebook formatting. Very interested to see the results.
I agree the way you have to do everything by hand and then cross your fingers when you upload is ridiculous. I was lucky enough to have a software developer helping me my first time and it was still ridiculously hard. I filled out the survey!
I think a simple fix to all of this would be a good WYSIWYG layout program. It would be one that:
1. Accepts all common file formats.
2. Automatically cleans up any garbage code that gets added. (We’re looking at you, MS Word.)
3. Exports to multiple formats at the same time, with perhaps a few options to adjust for each format to account for differences between retailers.
I would guess that the hardest part would be #2, just because garbage is relatively hard to define in a way that a computer can recognize it.
WYSIWYG on which platform, though? The exact same file can look radically different on different hardware/software, even when they are supposed to be compatible. Amazon can’t even make their own previewer render their own file format perfectly. And that’s not because their programmers are lousy. They have a darn good team. The problem is damnably hard. If you are a perfectionist, you are going to have to buy about 30 different devices to test with.
I’ve thought a lot about how to best solve this particular problem. None of the solutions I’ve come up with are very pretty. Do you farm it out to Mechanical Turk? Buy a bunch of devices and build a ereader-testing robot? Try to figure out some sort of heuristics to identify the subset of files that can cause problems? That last would be my first choice normally, but the total lack of documentation from the big vendors makes it almost impossible.
So here’s a question for you all. I wrote my novel in MS Word. I am sending the Word document to my copy editor, who will then use track changes for any comments. Ditto for my proofreader. How does our novel software manage something like that? If I have Scrivener or another writing program, I lose that ability.
Word may have a lot of flaws, but if you’re smart, and use only the barest formatting necessary, it shouldn’t be too difficult to edit when making the e-book. Until someone can come up with a program that also allows the exchange of a document with the ability to track changes, I’m sticking with Word.
Back in the day – I used Pagemaker to do a huge catalog that had been through a dozen hands – so style sheets and the word ‘Theme’ (OMG! PLEASE!) has my full attention.
I bought Scrivener 6 months ago – mainly as a database to dump all the little files I have in a gazzilion different directories on a million different subjects.
Now I’m using it for research – importing PDF and Video files – I’m becoming more interested in it as software for creating e-books. I’m frustrated with the export/print feature.
Obviously Scrivener is going to take some taming – out of the box it’s a crude mess. Highly useful – but hardly intuitive.
I would very much like to have my choice of fonts. Not for “eye-bleeding bold,” I just want something more elegant than Times or Arial.
I want READABLE layouts that are elegant to look at. Something a crude Word file will never give me.
So, William, I’m all for it!
I’d been happy to get something that would work as an add-on to Scrivener if that would be easier than starting from scratch.
Thank you for this, Mr. Ockham! A simplified self-publishing process sounds wonderful.
I know some people like to obsess over every detail of formatting and the code behind it. Personally I couldn’t care less about wrangling the latest KF8-whatever. At the time of publishing, I want to be able to easily grab the best/latest/simplest format and turn my words into a book. The more easily, the better.
Yes, one-button solutions sound nice. I hope we get more-or-less there… Meanwhile, it’s not actually *that* difficult to produce a reasonably vanilla EPUB that works everywhere.
Oh, and… When I run across an EPUB book and hate the format, for example the extra paragraph spacing in Project Gutenberg books, I open it in Sigil and just fix it.
One button also assumes that writers will all use anchors, line breaks and page breaks correctly. The main problem with one button is that most writers do not set the document up correctly the first. Most do not even know how or care to.
Actually, no. One button publishing is possible without authors needing to understand any of those issues, if they use software that can properly translate the authors intent into the appropriate code for the target format. Line breaks and page breaks should never enter an author’s mind when they are creating content. Those are strictly formatting commands.