SLAP HAPPY LARRY

App developers would do well to remember that when it comes to providing a reading experience that is developmentally valuable for young children, it’s as much down to what the app doesn’t do, as what it does.

a commenter on the Guardian article: Alarm Bells and Whistles

Many of the first digital picturebook apps (‘storyapps’ for short) produced for the tablet computer are created digital versions of real-book interactions (with the exception of scratch-and-sniff!), mimicking printed picturebooks in a retro kind of way. This is perhaps in a bid to win over reluctant audiences who are not yet convinced that digital picturebooks can create great memories in the same way physical, printed matter can. (See for example Popout! The Story Of Peter Rabbit, an iOS app which mimics a printed pop-out book.)

Apple themselves make use of skeuomorphism, evident for example in their Garage Band app, in which the backgrounds look like grey leather, even though devices themselves are reflective plastic. iBooks looks to be set upon wood, and readers of iBooks place digital purchases upon something that looks like a wooden bookshelf. There is no real functional reason for the mimicry of real-world materials, except that real-world parallels help intuitive navigation through an app, and also imparts to consumers the feeling  that they are not simply buying air when shelling out for bits and bytes, but are receiving an actual product, which costs real money to produce, market and update. It will be interesting to see how design principles evolve as smartphones and tablets become ubiquitous. Likewise, it will be interesting to see whether future generations feel a sense of value when spending money on digital media.

See: Why Do We Keep Making eBooks Like Paper Books? from Gizmodo

The illustrators of printed picturebooks have been creating skeuomorphs too*, since the emergence of digital art software. Few illustrators today produce illustrations without incorporating software at some part of the process, yet apart from a brief period in the 1990s when it was novel to make an illustration look digital (the art equivalent to pop-musical synthesizers), many digital illustrators now go out of our way to make sure the reader can see ‘the hand of the artist’. Hence, we might make an illustration look like a woodcut by making a ‘woodcut’ brush in Photoshop, or by erasing parts of a black layer with an eraser. (Also known as hedcuts.) And although it’s now possible to create photo-realism in illustration making use of powerful software, there seems to be still a general preference for artwork that looks as though it has been created using real-world media such as watercolours and oil paints. (Tradigital art.)

When it comes to storyapps, some of the major developers at present seem to be in limbo**, appeasing consumers who might still feel that physical, printed matter is superior to similar stories presented digitally. This applies not just to art style of course, despite the example above, but also to programming decisions.

We posit that when storyboarding for a storyapp, there is no real need to be limited by the constraints of printed matter. When stuck in limbo, storyapp developers end up compromising between what is digitally possible, and what consumers will accept in a definition of ‘picturebook’. For example, storyapps do not need to be either 32 or 24 pages to accommodate printing restraints.  Yet readers have been primed to expect a standard picturebook of 32 pages***, so how far either way can storyapp developers push the boundaries before a reader feels a storyapp is either too long or too short? Does interactivity lengthen the reading event to the point where shorter picturebooks are preferred? Will readers happily accept a longer one? These are all questions currently being put to the test.

*There is another way of thinking about the different types of interactive stories, this time making use of film terminology: Native Interaction, to describe stories which have been created from scratch with the touch screen in mind; and Post-Processed Interactivity, to describe printed matter, often with a wide, adoring and attentive audience, which has then been adapted for the touch screen.

**It would be a mistake though, to assume that printed picturebooks are immune to evolution. Picturebooks have undergone consistent reinvention along with printing technologies, exploding to life after the introduction of offset lithography. Printed matter will continue to evolve along with digital storybooks.

***In principle, a storyapp can be as short or as long as it needs to be, with restrictions coming not from paper and printing but instead from memory allocation, in a rather nebulous guideline from Apple stating that apps need to play nicely with each other.

Moving The Story Forward

Knowledgeable folk of storyapp world — reviewers and developers — advise frequently that interactivity in a storybook app should move the story forward. I have spent the past few years unpacking this simple sounding phrase. ‘Meaningful interactivity’ is a ‘I know it when I see it’ kind of a term, but do we really know what it means?  It pays for we as developers to unpack it fully.

Below, we make a case for various reasons for interactivity, where not all kinds of good interactivity ‘move a story forward’. Some interactions and animations exist purely for world-building; others for humorous effect. Some of the most sophisticated interactivity in storyapps may in fact contradict both words and pictures, providing an extra layer of meaning, and an added challenge for increasingly sophisticated readers. Hypothetically, storyapps can require more of a reader than printed matter can. (Digitisation of picturebooks certainly does not equal ‘dumbing down‘.)

The Various Purposes Of Touch Interactivity

1. Educational

For example, a reader touches an item on the screen and the written word appears. (Pop-out words.) Since the Mercer Meyer book apps became popular I see this from smaller publishers too. But developers need to be careful that there is genuine benefit here. If a reader touches a chair and the word ‘chair’ appears, the app is obviously aimed at either young readers or at NESB (non-English speaking background) readers.  (Obviously I’m writing from an Anglocentric point of view — same is true for any language.)

If a reader cannot be expected to be fully familiar with the written word ‘chair’, however, it should also be assumed that the audience has very emergent language skills in general. Such a story should unfold mainly in pictures, with text pitched at the basic level expected of a reader who has yet to learn the word ‘chair’. But is it really adding anything for older, native readers? Pop-out words need to match the general level of the app. It’s easy for a developer to add this sort of faux-educational interactivity, but much thought should be given to which items of vocabulary the reader is likely to know. Otherwise, it’s all just distraction. I feel that kids recognise pedagogical intentions, even if it is just vocab and spelling.

That said, pop-out words may indeed result in an educational outcome, if done properly. In cases where text expands the picture (most often it’s the other way around), if parts of the picture are labelled on touch, the reader is encouraged to find all the words mentioned in the text, in a fun,Where’s Wally sort of a way. The pop-up words provide a sort of confirmation (‘You’ve found me!’), allowing the reader to move on.

Pop-out words can also be useful if the words are in fact new to the reader. For example, a storyapp adaptation of Elsa Beskow’s Children Of The Forest (1910) might teach the reader the names of the plants, which are botanically accurate. In order for this to be useful, I feel that the pop-out words must be at least one level of specialisation above the vocabulary used in the main text. So the reader would be expected to know ‘flower’ and ‘tree’ (main text) but is guided to learn more with a pop-out ‘snowdrop’ and ‘lingonberry’. I learned my hues from a 64 box set of Crayola crayons, so I know from experience that labeling can work to expand vocabulary, but only if the child reader has fallen in love with the story in the same way I fell in love with my crayons.

Another use for labeling parts of pictures is when making use of text within pictures (a.k.a. intraiconic text) without the intraiconic text competing for attention with the main text until the reader is good and ready, ie, on user touch. An example of intraiconic text is when, for example, an illustration depicts books on a bookshelf, which each have their own titles, or when illustrated shops display billboards. Ticket stubs, restaurant menus, letters, computer screens… there are many, many plot-advancing reasons for displaying intraiconic text within an illustration. The iPad screen does not afford the real estate of the larger printed picturebooks, so a developer might get around this limitation somewhat by having a magnified view of a sign or letter on user touch — too small to be seen beforehand, but read easily on iPhone sized screens on touch.

Breaking Research: Most apps bad

2. Novelty

There is a place for humour and pure fun in story apps. Ideally, novelty interactivity should marry with interactivity that moves the story forward, or it should at least lead the story off into its own mini tangent. For example, the farting teddy bear in Teddy’s Night (by Bruno Hachler) is mainly for novelty value. I’ve noticed with my own toddler (and heard from other parents) that young children are inclined to skip straight to this page, then linger on it, presumably without getting into the story itself. So overly attractive novelty interactions are a double-edged sword: they encourage children to go back to your app over and over, but as far as brain candy goes, they aren’t necessarily any better than having them watch junk TV. It might be argued that these sorts of interactions inspire a love of reading. I’m inclined to think they inspire a love of novelty. But this is nothing parents can’t fix by guiding children through the story from start to finish.

As far as novelty interactions go, there is the full spectrum. The best seem to advance the setting of the story. An example is the floating swimmer in Heart And The Bottle, in which the swimmer shoots up a small fountain of water from her mouth on touch. The words on the page say, ‘With wonder at the sea’ — nothing about floating in the ocean, shooting seawater from your mouth. In this case, the shooting water is both fun for the reader to do and also adds to the atmosphere of the setting: lazily relaxing in the sea. This mood is reinforced with the sound of the ocean and cry of seagulls which autoplays in the background.

Another way in which novelty interactions can add to the setting is when, say, the user touches a bath tap which fills the bath with water. It can be fun to fill a virtual bath with virtual water, and perhaps draws readers into the story by helping them to feel that this storybook world is real. Novelty can add to the verisimilitude.

Some interactivity offers a surprise, reminiscent of that scene in Tim Burton’s 3D Alice In Wonderland movie, in which a ball flies toward the audience as the characters are playing a game. I remember ducking in my seat. Annoying as that was, it was memorable.

Interactivity can be used to create a ‘pageturner’ — a book that children must read to find out what happens. This sort of novelty, which encourages movement forward through the story, is a positive thing, though not when it comes at the expense of comprehension. This is a difficult balance to achieve, and depends somewhat on the age of the reader and all sorts of external things an app developer can’t control for (like the presence of adult co-readers, slowing the child down). The issue storyapp developers face is, where to put the interactive ‘pageturner’? That’s the conventional place to put something charming or interesting, to encourage the child to turn the page. Interactive story apps require that space for actual digital pageturning, be it push-button or swipe (which is even harder to incorporate). It’s likely that readers of digital stories need to learn a slightly altered story code. This will only work if it can happen intuitively and subconsciously, just as it happened when we all learnt to decode printed picturebooks.

One thing is clear: Developers should be able to rationalise every hotspot on a page. The ‘distraction issue’ isn’t limited to storyapp developers, though. The producers of printed picturebooks are equally capable of unconscious distractions in their work. From How Picturebooks Work:

In pictures…with many details, our reading is arbitrary. The artist may deliberately or unconsciously place a detail in the picture that will attract our attention and compel us to start reading the picture from this point.

While the developers of picturebooks are often great lovers of the form, and have spent so much time immersed in them that even the unconsciously placed details are meaningful, our choice to include interactions and animations in storybook apps is best if it’s the deliberate kind. The medium is perhaps too new, even for lovers of the form, to have developed a purely intuitive sense of what’s right. Besides, studying the form can never hurt.

3. Animation

“One of the things we love about the still image is the way in which it can stimulate the imagination to create a fiction around an image,” said Robin Kelsey, a professor of photography at Harvard. “The fact that we can commit a single image to memory in a way that we cannot with video is a big reason photography is still used so much today.”

The Death Of Photograph Has Been Greatly Exaggerated

3.1 Animation That Matches The Text Exactly

You might call this ‘Paratextual Movement’. This kind of animation is not necessarily ‘interactive’, because animation can autoplay after the reader turns onto the page. Or it might be ‘semi-autoplay’, for example the reader taps anywhere on the page and something starts to move, regardless of whether the reader has engaged with that part of the picture.

This kind of animation can fulfill a limitation of picturebooks: animation to depict the flow of movement, long employed by film-makers.  The words might say, ‘The boy jumped’. Then, when the reader touches the boy, he indeed jumps. This doesn’t advance the story but it does help with story comprehension in readers whose language skills are in the emerging stages. In other words, the reader doesn’t need to be able to read the text at all. This kind of interactivity is of limited value to older readers who are able to read the text for themselves and who remain unchallenged by paratextual movements.

For books aimed at competent readers though, I prefer animation to exist without the redundancy of words. If the boy on the screen jumps, then the words don’t need to tell us he jumped.

3.2 Animation To Advance The Plot

You might call this ‘Extratextual Movement’. The animation adds something to the picture which the words do not duplicate, thereby making it redundant.

This is the sort of interactivity to aim for in stories for older readers who are better-versed in reading pictures. At times this sort of interactivity creates a gap between the story and the text; at other times it helps the reader to fill it in. What results is a story in the reader’s head that is bigger than the sum of text/pictures and interactivity.

Printed picturebooks sometimes rely on a series of pictures to depict the passing of time and a series of events, which may or may not be dependent on each other. (Simultaneous succession.) But younger readers need time to decode this convention. A younger reader may look at five pictures of a dancing bear and mistakenly believe that there are five different bears rather than a single bear in five successive poses. Animation does away with this confusion, and therefore animation has its place in storyapps for younger readers.

On the issue of depicting the passage of time, this is one area which may be made easier for stories with animation and touch interactivity, as it is a difficult thing to convey in static pictures, without specific reference to a clock. But an interactive book can dim the lights or show hands moving round on a clock or have a cock crowing outside a window or something like that to show that time has passed.

5. Interactivity to Depict Internalised Thoughts/Emotions Of A Character

Something picturebooks and film must do differently from chapter books and novels is the depiction of a character’s inner life. This is not easy in visual media, and interactivity allows the possibility of helping with this task. I’m not seeing much of it, though.

For example, emotions can be depicted by changes in hue and saturation. This is a customary technique in visual media and sometimes in print picturebooks the reader turns the page to see a new colour palette. This change in colour could become part of the interactivity, with a slow fade out of saturation or with a rub-to-reveal portion, or with isolated areas of colour blending to create a new picture. The possibilities are endless, and these are just a few examples that spring to mind.

Or the user might ‘zoom in’ on a character to switch from omniscient to close-third-person point of view. Or developers might borrow from different graphic codes such as photography and camera work, making use of pull-focus, blur, motion lines and distortion of perspective.

Story app developers can also take advantage of comicbook conventions, with thought and speech bubbles, or with mimesis and onomatopoeia emerging from objects on touch. (See Scott McCloud’s Understanding Comics. TED Talk by author here.)

6. Pure Animation

Some interactivity exists purely to give the storybook an animated feel. Blinking eyes, nodding heads, walking legs can give a storyapp a polished feel that places the genre somewhere between printed book and TV. Sometimes these movements are not interactive but autoplay. This may be a better option. When planning interactivity, I do wrestle with where to draw the line. If you start out animating too much, the reader will expect the same level of animation on following pages. Do we really need to make eyes blink? Or should eyes blink only when there’s a reason to, for example because the scene has changed. Maybe the character’s eyeballs could move because the thing that’s drawing her attention has moved to the other side of the page. There is a certain amount of intuitiveness necessary when planning for animated interactivity. The blinking eyes are necessary in Teddy’s Night because the teddy and girl being awake when they’re not meant to be is an integral part of the story. Blinking eyes in other storyapps won’t necessarily move the story forward.

Other examples: clouds move across the sky, wind blows leaves on the trees, small animals peep out from bushes, birds fly across the sky.

Some storybook apps with higher budgets pull off entire animated scenes. Many of these are made from movies and short films. An example is The Amazing Flying Books Of Mr. Morris Lessmore. These sequences tend to have very high production values and turn a storybook into something between printed matter and a short film. Although these animated sequences tend to be set off by the reader, who turns a page or touches somewhere on the screen, I don’t consider them true interactions, because all that is asked of the reader is to sit before the screen and watch, immersed. As long as these sequences create immersion rather than impatience and distraction I don’t take issue. Here at Slap Happy Larry we both have a personal preference for story apps which function closer to printed matter than to film, but we are film lovers equally.

6. Extra-textual Interactivity

In some storyapps you find some dalliance that takes you out of the story for a while. Jigsaw puzzles are one common example. Heart And The Bottle has a page in which the reader draws on a piece of paper. The drawing appears on the wall in the next scene, which has some novelty value. It also shows that the main character is using her creativity by drawing pictures, and is therefore connected to the story. Similar interactivity exists in Teddy’s Day, in which the user touches the girl’s paper, flipping to another screen. Another page in the same app shows the girl building a block tower. On touch the user is taken to a different screen in which the physics engine has been employed to show the blocks floating midair. I have thought about how this advances the story, but concluded the developer enjoyed the novelty of the physics engine.

I do feel developers need to be careful though, because in being asked to draw a picture, apropos of nothing, the story turns temporarily into an art app, and the reader is first expected to work out how the mini-app works, and then to come up with something out of thin air before moving back into the story, which they’ve just been pulled out of. I liken this to a classroom experience in which a teacher gathers the class onto the mat for storytime, reads half the story, then asks the children to get out paper and pencils, draw a picture, then draws them back onto the mat to finish reading the story. Obviously this is not good teaching practice, and obviously it takes less time and organisation for a single child using a single iPad to move in and out of story/artistry. Nevertheless, I wonder if at the most basic level, we’re doing exactly the same by embedding fancy technologies into the middle of stories. At the very least, we’re changing the reading experience, turning it into metafiction by drawing attention to the fact that it’s a story.

I prefer to see puzzles, colouring activities and other kinds of gamification at the end of a story, or available only via the main menu, if at all. I hope extras do not become an expected part of story apps. I hope they don’t become a cheap drawcard to get children opening an app.

6. Marketing

But while certainly a little peculiar, the toilet-paper-selling and jeans-hawking ultimately aren’t all that weird. In fact, they’re actually a reminder, kinda strange and kinda funny, that books are part of a commercial ecosystem that moves stories and steam cleaners in very similar ways. That may be especially true of the Kindle, considering Amazon’s totalizing ambitions and aggressive merchandising, but it applies more broadly, too. Books are, in many ways and in many contexts, simply commodities. They aren’t sacred, and they aren’t just disembodied ideas; they are things (whether of paper and ink or bits and bytes) first of all, and they move around like things do.

My E-Reader Is Selling Me Toilet Paper, Book Riot

It’s worth pointing out from the get-go that advertising in children’s literature is far from new. (See Advertising In Children’s Literature, a paper by Afsana Kahn.) Unfortunately, interactivity, and the need to recoup funds spent on producing expensive storyapps lends an extra dimension to all things kidlit, not least in the advertising opportunities. Also, as pointed out by Nikolajeva and Scott in How Picturebooks Work, authors have long amused themselves by including references to their own former works, for example by drawing a picture of one of their own books lying on a bedroom floor.

Several developers have done a similar thing in their apps. In Teddy’s Day, for example, if the reader touches a bookshelf they are taken to a page advertising Teddy’s Night, the companion story. Yet unlike the print equivalent, the reader has touched the book expecting a different kind of interactivity: There is no warning that this is going to take the reader out of the story.

While this sort of promotion is common and almost expected in gaming apps, it’s unfortunate for story app developers that literature has a revered position in most people’s hearts, perhaps to an unreasonable degree. Nevertheless, marketing embedded within storyapps  is something we hope consumers learn to avoid. Unless consumers boycott such practices they are likely to continue.

7. Non-story Related Activities

“BY ITS VERY NATURE THERE CANNOT BE A PLOT IN A GAME.”

 Lucas and Spielberg on storytelling in games at The Verge

There remains a place for non-story related activities embedded in the same app as the story itself. After all, teachers have always made use of extra-narrative activities when guiding students through the understanding of any given work of art.

Also: Playing With Books Is Important Step in Path to Early Literacy. Might this apply to the digital literacy of eBook reading as well?