ARKit and Autism: New Futures

In September the new operating system for mobile Apple devices, iOS 11, will likely be released, and with its release will come a new world of apps that build on the new development platform that Apple has created, ARKit, which puts unprecedented opportunities to develop augmented reality solutions in the hands of the world. We have already seen some spectacular demonstrations of what ARKit is capable of, and as always when new technological opportunities present themself, I consider what this could mean for the world of autism support and the education of children on the autism spectrum.

Augmented reality has been a focus of autism education research for some years new, but most often on a very introductory level involving small scale experimental case studies. With ARKit, the size of the stage to explore augmented reality will render the stage into the world, in much the same way that Pokemon Go instantly put a new world of augmented reality into everybody’s hands as they chased pocket monsters around their neighbourhoods. I developed what turned out to be my most popular educational resource, ‘Explore Everything with Pokemon Go’, when the game came out, such was the excitement of my students on the autism spectrum and my friends and family on the spectrum. I saw immense potential to meet personal and academic goals through the power of augmented reality in the game as it so directly tapped into the special interests of our students.

Now, I want to propose ten ways in which ARKit could be used by developers to create unprecedented opportunities for those on the spectrum, as well as their families and educators, to engage in the world with a completely new and different view as to what is possible.

1. Visual Schedules

The use of visual schedules and timetables is one of the most fundamental supports for children on the autism spectrum. A visual schedule works by providing a visual list of what needs to be completed in relation to a particular task — for example, how to get dressed of a morning, how to make a sandwich, what activities need to be completed in a set order in the classroom.

The potential for augmented reality here is immense — imagine if when you looked at the kitchen bench with your lunch box of a morning, a visual schedule of what to pack in there, a visual of a banana, and then a sandwich, and when you look at the sandwich is splits into its different layers and shows you how to prepare each step of the sandwich.

And, picture being able to look at your wardrobe through the same ARKit developed app and see a floating visual schedule of how to get dressed for the morning. What if when you looked at the schedule the visuals animated to show you exactly how to put on your pants, your shoes. Add in the potentials of Apple’s new Core ML machine learning framework into this space, with the app learning that when it is a school day you wear a particular type of uniform, and when it is sunny you wear a different uniform to when it is cold, and when you have a trip to Sydney coming up you wear different clothes to when you’re going to the local park to play.

2. Emotional Regulation

The ability to recognise how you are feeling at any given time, and then to be able to select a strategy to help yourself feel better, or to keep feeling good, is a very important goal for so many on the autism spectrum. We call this ‘emotional regulation’, to be able to recognise, for example, that you are starting to feel anxious about a situation you are in, and that you can use strategies to help manage that anxiety and feel better about handling the situation. Unfortunately, it can be a very difficult skill for students on the spectrum in classrooms to acquire and implement effectively.

Consider, then, an augmented reality solution to this — when a student is at their desk, and a reminder pops up in front of them to emotionally check in with how they are feeling. Virtual character faces displaying a variety of expressions might hover above their desk, that they can select and then observe as the character demonstrates the regulatory strategy that is needed to help support how they are feeling. A student might select a ‘stressed’ character face to indicate that is how they are feeling, and then the character could model a particular breathing pattern, or display options such as how to get a drink of water or how to ask the teacher for a short break, all in augmented reality, a hologram friend in the classroom with the student, providing them with the most amazing visual model for how they can handle their current situation effectively.

This could occur in the classroom, but it could equally happen on the playground and in every other space that the student engages in across the day. The potentials for different variations on this theme are absolutely immense.

3. Social Modelling

Picture a school playground. You are standing outside of your classroom, looking out onto the playground. There is an area where your friends are sitting and eating lunch, there is a big playground with spaces to play handball, to play basketball, to sit and play with cards. Then there is the library, the canteen, the other classrooms. Now, imagine you are a student on the autism spectrum who finds this environment very confusing and very overwhelming. You don’t know where you want to play or what you can even do on the playground, and if you do find something that looks fun, you aren’t sure how to join in the game.

Now, picture an ARKit app response to this. When the child looks out at the playground, there are now visual overlays on the different areas of the playground, annotating what each area is for — where the area is for quiet play, where the handball courts are. And then, the child wants to join in a game of handball, so up pops up a little augmented reality video appearing on the playground, demonstrating, step by step, how to walk up to the handball court, how to ask the other students there if you can join in the game, and then how to get started playing.

This is the sort of thing we would previously created paper based social stories for, with illustrations and photos to try to help scaffold social understanding in this manner. To think that with the power of ARKit it can now happen in front of the child’s eyes, on the playground, in real time, supporting real situations, is a revolution.

4. AR AAC

Now we are really starting to step through the augmented looking glass. Individuals with autism can have significant communication challenges that can require the use of visuals in order to communicate. We often call these visuals ‘picture communication symbols’, where an individual can use a symbol of a toilet to represent that they need to use the toilet, for example. For young children, learning what these symbols can mean can be a tricky task. We might start with using a real world object, like a glass of water, and then relating that real world object to a photograph of a glass of water, and then transitioning from the photograph of the water to a clip-art symbol of the water, so we can then provide that symbol, and many other symbols, to children to provide them with a visual vocabulary with which they can then communicate with the world.

Currently, these symbols are currently sometimes presented in a paper based system, or with the introduction of iPad there are now a range of apps that provide this system in a highly interactive and customisable manner. We call these systems ‘Augmentative and Alternative Communication’ systems, or just ‘AAC’. With ARKit, we could infuse these AAC systems into the real world around us like no system has been able to implement before. Imagine walking through your home and, when you look at the television, a picture communication symbol for the television appears on it. And then when you look in the bathroom, the shower, the toilet, the sink, all have picture communication symbols hovering over them.

When children use this augmented reality system, they could learn in real time, in real contexts, what symbols relate to the real world objects, so when they use the symbols later they are already masters of what the symbols relate to. The possibilities that extend from this, from not only being able to see the symbols hover above the real world objects in augmented reality, but to then be able to use those symbols to communicate to someone with, without requiring another AAC system, all of it being conducted within the augmented environment, is a dream that could change how the very nature of these modes of communication can operate.

5. Visual Focus

So much of what I’ve been describing so far has involved visual systems of engagement — visual symbols for communication, visual modelling to demonstrate social skills or emotional regulation skills, visual timetables to help to organise the day. This is because for so many individuals with autism, visual processing is an immensely powerful part of their cognitive toolbox. In our classrooms we implement a method to help students visually focus in on tasks and independently progress through activities in the classroom, which is referred to as ‘structured teaching’.

At the core of structured teaching is the need to organise the classroom and classroom resources into ways that provide a high level of visual focus for students. This means, a student might have a visual timetable that indicates they are to complete a numeracy task at the blue table, and then after that is complete they need to complete a play-based task on the floor. The tasks at each area will employ visual focus strategies as well, such as making sure a worksheet task has a colourful border around it, or that the play-based activity on the floor has a clearly designated space in which to operate.

Picture a student using an ARKit app, looking at the classroom they are in, with these elements of visual focus being rendered bright and engaging and immediately understandable — the blue table has a neon glowing circle around it with a big number one, indicating this is where to work at first. Then, when this task is complete, a neon glowing circle appears on the floor, for play-based activities. The tasks at each activity have colourful overlays on them too to indicate which resource is to be used for which task. There could be search functions too, like ‘Where does my work go when it is completed?’, and the augmented reality overlay would highlight the area in the room where their work is to be taken. With machine learning employed, the software could provide an augmented overlay that indicates how long it took the student to complete their activities last time, with a motivating gamification element built in to encourage an even faster task completion time. The possibilities here just go on and on into amazing new territory.

6. Bravery Tool

This idea could be so fun. I worked with a student a few years ago who was so fascinated with the video game world of Super Mario Brothers that his preferred tasks to engage with on the playground and in the classroom had to relate in some way to Mario. When we were able to set up obstacle courses on the playground that related slightly to the platform game obstacles of Mario, we had success engaging him. But he had immense struggles transitioning into the classroom. While there are many schools of thought on how to support a challenge such as this, more and more we are finding that a strengths based approach, where we utilise the special interest of the child in order to motivate them through situations they find challenging, is by far the most successful. As well, there can be immense knock on effects for how the utilisation of special interests in the development of children can lead to further success in post-school work settings.

Here is my consideration for how ARKit can engage in this space — custom textures that you can apply to the world, through augmented reality. When you approach the school, it is no longer just a school, it is a video game castle. The playground, previously a concrete tarmac, is now a pink field filled with yellow boxes with question marks on them. Through using an augmented reality custom texture that can be applied to the real world environment, the special interests of the children are projected onto their environment in a way that provides with the motivation, the bravery, to engage with it and complete required tasks in this newly themed augmented space.

There would be opportunities here to then scaffold the custom textures away as the child found more and more sustained success in the school environment — at first, the whole classroom might look like a Mario world, but then after the child achieves a level of mastery at following the classroom routine, textures could be reduced so that perhaps only a few Mario graphics remained until, gradually, these too could be removed so the child no longer requires this particular bravery tool. And when another situation arises, a new and uncertain setting that presents as too overwhelming for the children to engage with, the bravery tool comes back out, turning on its augmented reality textures and layering the new and uncertain setting with all the familiar and much loved elements that can be found in the safe and well-understood special interest of the child. Can you imagine a child with autism who, previously unable to enter a shopping centre, has the opportunity to enter the shopping centre as an augmented reality themed Minecraft shopping centre? Let’s grab a pickaxe and pick up some groceries!

7. Sensory Customisation

A very large proportion of individuals with autism are impacted by sensory senitivities. This can present as an under-sensitivity to particular sensory sensations, such as seeking out a really tight squeeze instead of a regular hug, or it can present as over-sensitivity, in which case lights might be too bright, or certain environmental colours can be disorentating and cause distress. Imagine the bravery tool mentioned above, but rather than a customised texture, the environment in front of your eyes is the regular space you are, but each of the sensory elements in front of your eyes can be customised.

You highlight the ceiling lights and a customisable adjustment feature appears, allowing you to dim the way the lights appear through your augmented reality vision. You highlight the bright white walls, and you select another colour for them to present as, to turn perhaps to a soft powder blue. The world allows itself to be personalised to your particular sensory needs. And, with machine learning working behind the scenes, when you enter a new environment that shares similar contextual similarities to an environment you’ve already customised, the new space takes on the light and colour settings of the other customised space.

8. Life Gamification

We use reward charts a lot for our students on the spectrum. The external motivation that reward charts of all kind can provide is something that is immediate, can allow for instant feedback and encouragement for positive behavioural choices, and can be transitioned towards more intrinsic motivational forces in time. The notion of gamification, of using point based systems and badges to reward students for achieving particular goals, has gained a great deal of traction across education, but one of the challenges I still find with these reward based systems is in the manner we provide immediate feedback to students for a very particular thing they have accomplished. That is, when you give a student a tick on a reward chart for having demonstrated good social skills at recess, you are almost always providing this feedback afterwards, very rarely during the actual, real moment that the child is demonstrating what you want to reinforce. And with that, the student often doesn’t make the connection themselves, so when they are on the playground next time, they aren’t sure what they are meant to do to receive a tick on their reward chart, and they might not be sure you are even watching them to know when they are doing the right thing.

Enter ARKit and the potential for gamification rewards and badges to appear in real-time on an augmented reality overlay on, say, the playground for example. Annotated text might float above the handball court saying, ‘When you lose a game, if you manage your frustration by taking steady breaths and moving to the back of the line, eager for another game with your friends, you will earn a point. When you earn three points, you will receive a reward’. Say the game of handball is recorded through the same ARKit app, and the student and teacher can watch the game back to see if the student was able to manage their frustrations appropriately, and if so the teacher can award them a point for positive social skills. This gamification element could appear all over the school — ‘Talk to a student you have never talked to before for 5 points’, ‘Make sure the lights in the library are turned off when class is over for 3 points’, ‘Say good morning and good bye politely to your teacher for 10 points’, all appearing in augmented, annotated text contextually connected to the different areas of the school. Or home. Or the shopping centre. Or the world.

9. Annotate Others

People aren’t consistent. We aren’t all the same. Some people don’t mind if you wear a hat indoors, and some people do. Some people don’t mind if you are a few minutes late to a meeting, and some people definitely do. And, these conditions too can change depending on situation and time. For our students on the autism spectrum, this can prove a real difficulty, especially in high school environments when they are interacting with many different schools teachers across the day.

Imagine the ability to be able to annotate, in a private augmented reality space, details about people that would greatly assist your interactions with them. When the face of a particular teacher appears in your augmented reality viewscape, their name hovers above them, with a list of notes just beneath that you can customise, indicating things like ‘Mr Hamilton is ok if I wear my hat inside’ and ‘Mr Hamilton likes cricket, so I can ask him about the cricket when I see him’, or even notes like ‘Mr Hamilton asked me to let him know if I’m available to help at the sports carnival’ so you know to follow up on things when you’ve been asked. I’ll let you consider the immensity of where Core ML machine learning could take this even further. And, while we’re annotating others, why don’t we…

10. Annotate Ourselves

Let’s take selfies to a whole new level of self actualisation. Turn the camera onto yourself, and with augmented reality engaged you can annotate notes that can hang and hover around your face to remind you of who you are and what you are doing. In autism education we often talk about ‘executive functioning’ skills, the manner in which we can guide the conductor in our brain how to keep the different organisational parts of our mind performing the music of our waking life in harmony. Unfortunately, so much of this is abstract and difficult to understand, so we use visual systems of organisation and self understanding with our students as much as possible.

Picture looking at yourself, iPad held up to your face, with notes hanging over your head like ‘You’ll feel great if you go for a run tomorrow morning’, or then a video beneath it that, when activated, plays a clip you recorded after a run session where you were celebrating and feeling amazing. Or maybe you have an album of photos that you have arranged around your face, inspiring pics that remind you of the amazing things going on in your life. The well being and mental health needs of individuals on the autism spectrum are immense, and the ability to develop ARKit apps that support this space could be very powerful for many lives.

11. Curiosity Tool

One more bonus idea, for the little engineers in our ranks who love to work out how the world is put together. Imagine directing your ARKit lens towards a car, and suddenly all the components that make up the car are extracted and displayed in an exploded-view visual, showing the assembly and relationship between all the different parts of the car. Imagine walking around the world with this curiosity tool, scanning your computer and seeing an exploded-view that explains the magic that makes it work, and doing the same for your body, to see the bones in your feet and how they are all put together, or directing the lens towards the moon and exploring a visualisation of the layers that make up its lunar composition, in an ultra-realistic augmented reality space. Let’s make it happen.

I’ll be following closely and getting involved in the development of augmented reality tools such as these with the release of iOS 11 as the potentials of ARKit are further and further realised. While my attention is most often within the world of autism education, I would love to hear from others working in the accessibility, special education and universal design spaces to learn what you’re thinking about the potentials of ARKit in your field, in your life and in the live’s of those you work with.