Podcast: Play in new window | Download
Your weekly dose of information that keeps you up to date on the latest developments in the field of technology designed to assist people with disabilities and special needs.
352-02-23-28 – Seeing AI with Belva Smith
Microsoft’s Seeing AI app: https://www.microsoft.com/en-us/seeing-ai/
Meet Belva: www.EasterSealsTech.com/staff
App: News2You | www.BridgingApps.org
——————————
If you have an AT question, leave us a voice mail at: 317-721-7124 or email tech@eastersealscrossroads.org
Check out our web site: https://www.eastersealstech.com
Follow us on Twitter: @INDATAproject
Like us on Facebook: www.Facebook.com/INDATA
——-transcript follows ——
BELVA SMITH: Hi, this is Belva Smith. I’m the vision team lead here at Easter Seals Crossroads, and this is your Assistive Technology Update.
WADE WINGLER: Hi, this is Wade Wingler with the INDATA Project at Easter Seals Crossroads in Indiana with your Assistive Technology Update, a weekly dose of information that keeps you up-to-date on the latest developments in the field of technology designed to assist people with disabilities and special needs. Welcome to episode number 352 of Assistive Technology Update. It’s scheduled to be released on February 23, 2018.
Today our own Belva Smith is going to spend some detailed time with us going over Microsoft’s Seeing AI app, which I just hear about a lot all the time. Also the folks at BridgingApps are going to talk about news to you.
We hope you’ll check out our website at EasterSealsTech.com. And give us a buzz on our listener line at 317-721-7124. Or drop us a note on Twitter@INDATA Project.
***
Each week, one of our partners tells us what’s happening in the ever-changing world of apps, so here’s an app worth mentioning.
AMY BARRY: This is Amy Barry with BridgingApps, and this is an app worth mentioning. This week’s featured app is called news to you. News to you is the app version of a symbols based newspaper specifically designed to support children with special needs in a fun and engaging way. Kids can read the news at their own pace and that their level. Each week brings a new addition that aligns with the current topics in the news, such as sports events, holidays, and politics. Recent additions have featured stories about the Star Wars movie, holidays toys, the World Series, and the solar eclipse.
The app offers for reading levels: simplified, regular, higher, and advanced to adjust to a child’s individual skills and abilities. The app is truly accessible, offering features such as text to speech, symbols with text, words underlined as they are read aloud, and options for downloading and printing. Just as with other apps designed by Assistive Ware, a variety of voices are available to download, customizing the app for the individual user.
In 2017, news to you and enjoy the addition of two new voices: Sharon, a US-speaking English voice; and Rona, a Scottish English speaking voice. News to you is part of the unique learning system, which is a standards-based curriculum designed for special needs students and used in many schools with special education programs. Many teachers, speech language pathologists, reading specialists, and other professionals use the app in classroom settings, small groups, and one on one environments. The popular with teachers, parents love the new to you at because it is a way to extend the learning at home by conversing with their children about current events in a simplified way.
One of the most well-designed aspect of the newspaper of the activities following the stories that include recipes, jokes, a game, puzzles, Sudoku, comprehensive questions, and vocabulary that can be used in a variety of ways. These activities can be completed at different sittings to reinforce the learning and continue the discussion, completed all at once for advanced students, or even done in a group setting with several children working together.
News to you is available for free at the iTunes Store. The app is compatible with iOS devices. For more information on this app and others like it, visit BridgingApps.org.
***
WADE WINGLER: So as you guys know, I spend a lot of my time keeping an eye open for what’s new on the market and what can I do assistive technology is happening. Because I’ve been doing this so long, I’m always fascinated by how things are getting better and smaller and faster and more features and those kinds of things. I’ve got to tell you that recently I’ve heard a lot about an app called Seeing AI. I’m finding that it is useful in lots of situations, or at least that’s what I’m hearing. Quite frankly, I haven’t had a chance to mess with it a whole lot, but I’m super fortunate here that at Easter Seals crossroads I am surrounded by people who spend all of their data dealing with assistive technology. I just said hey, Belva. Belva is our vision team lead. You might know Belva. She’s been on the show a number of times. She’s also one of the regulars on ATFAQ, which is assistive technology frequently asked questions. It is our question and answer show. Belva swims in these waters of vision related to assistive technology all the time. Hey Belva. How are you?
BELVA SMITH: Hey Wade. How are you?
WADE WINGLER: I’m fine. I know you and I are recording a couple of shows today, so we are going to be in the studio all afternoon together which is always fun. I know you’re just coming back from a conference which was good as well. I want to talk a little bit about Seeing AI, but before I do that, there may be — it’s probably not likely — but there may be somebody in the audience who has entered you before. Tell everybody a little bit about you and your job is and why you became interested in AT for folks who are blind or visually impaired.
BELVA SMITH: I guess because I am a geek at heart, and technology has always been interesting to me, and the fact that there is technology available to allow people that are blind or visually impaired to do things like sent an email or send a text or read their mail, it’s just amazing to me. I feel like I was sleeping when Seeing AI first came out because I was the last want to know about it here in the office. To my surprise, it was released in July. But I’ve been using it for about four or five months now. I thought about it every chance I get. Everybody I meet with to do an assessment or evaluation, if they already have an iPhone or iPad, I immediately tell them about the app, and we get it put on right away.
WADE WINGLER: It’s from Microsoft, right?
BELVA SMITH: It is.
WADE WINGLER: And it’s free?
BELVA SMITH: It is. That’s 99 reasons why I like it better than KNFB. I said that to someone recently, and they said it’s not $99 and more. It’s cheaper. I don’t know what the exact price of the KNFB reader app is. But this is very similar to that. And ways it does more, and ways it doesn’t do as much. This is basically a project that they are still working on heavily, and it’s amazing to me that with all the fish we have working on assistive technology for people that are blind or visually impaired, that took Microsoft to be able to do some of the things this app can do.
WADE WINGLER: That sort of makes sense. Let’s talk about the “who” part of this first. What kinds of people with what kinds of disabilities might use this kind of app? Is it just vision? Is a low vision, blindness, learning disabilities, who might use this?
BELVA SMITH: All of the above. Really anybody who would benefit from having text read aloud to them, or anybody who might benefit from having an object identified for them.
WADE WINGLER: And what platforms to the work on? Is it iOS only? Is it Android? Is it Windows phone — to we still even have Windows Phone?
BELVA SMITH: I think there are four left in the world. Right now it is only iOS. I had not heard anything about it coming to Android. Whether or not it will, I don’t know.
WADE WINGLER: And is working on iPhones and iPads? Is it pretty well cross platform within the iOS world?
BELVA SMITH: It’s not as good on an iPad — I should set iPad Pro. That’s what I tried putting it on. It wasn’t quite as good. But on the iPhone, it’s awesome. That’s all the way from the four’s up to the ten’s.
WADE WINGLER: Or the iPhone 1000 like I want to call it because it costs $1000. In general, what does it do?
BELVA SMITH: It has the ability to do short text documents, products, objects, people, colors, but the most amazing thing that it does — and it does more than that. It will do scene. It will also do light, so it will let you know how bright the light is in the room. But the most amazing thing it can do is handwriting.
WADE WINGLER: What?
BELVA SMITH: I know, right? That’s why I said it took Microsoft to figure it out. No one else has been able to do it. I’ve tested it. It works pretty well. Obviously is that 100 percent accurate, but it makes me slow down and make sure I’m writing more clearly, but it does a fine job with the handwriting.
WADE WINGLER: So when you say it does these things, we are talking about reading them out loud. It’s going to convert written text to spoken text, right? Audio format?
BELVA SMITH: Yes. And it’s also going to give you the audio information about an object. For example, if you are in the grocery store and want to know if you’re looking at cream of mushroom soup — yuck — or chicken noodle soup, you can read the barcode and it will give you that information.
WADE WINGLER: Is it voice out only or does it work with braille panels? Reversible braille?
BELVA SMITH: I have not tested it with any braille panels. I do know that it is 100 percent compatible with voiceover. On the screen, when you open the app, you got four buttons, your menu, your help, another call them channels. The short text is a channel, the document is another channel. In each update, they usually add one or two channels.
WADE WINGLER: Let’s talk about the user experience. Describe the app for me. You’ve given me some insight into that. You turn it on, what are your options and what is the user experience like? We are going to play with it. We are going to hook your iPhone interest us on board so people can hear and listen along with what it’s doing.
BELVA SMITH: When you first open the app, as I said, you’ve got your menu in the top left corner and then your help over in the top right corner, and then your channels run horizontal. As I said, it is fully compatible with voiceover, so if you do have voiceover on, it is just swiping through those different channels. Some of the channels are automatic, meaning that as soon as I go to the channel, for example, with short text, I don’t have to do any of the button on the screen. Once I’m on that channel, I hold my camera over the short text that I want to capture. That could be the of sign over an office building door to let me know which room is which, like the men’s room or the ladies room. But if I do a document, then there will be another button that will appear. It’s about midway, always in the same place, on the left side of the screen. That’s going to be the camera button. Because for the ones that are not automatic, you do have to capture them. When you capture it, it takes a little bit longer for it to process. That’s it.
For those that are visually impaired, I should say too, if you are doing short text or a document, you do have the option of enhancing the font so you can make it larger. For the ones that are using the screen reader, you can control the voices for how fast or slow it is reading. But if you are using the app with that was over, you do it from the app. If you’re using it with voiceover, then you would do it with your voiceover settings. Because voiceover, I guess, takes over if you have it running.
WADE WINGLER: There you go. I want to play around with this and see some of the stuff it is doing. You are in our studio, and you’ve got a bag of chips and a Monster drink and a letter in a program. These are props, right?
BELVA SMITH: Absolutely.
WADE WINGLER: Let’s try these things. You got an envelope or something?
BELVA SMITH: I got the app open and I got the channel for short text a selected. Remember, I said this one is automatic so I don’t have the touch under the button on the screen. I just want to make sure I put my camera at some short text.
SPEAKER: Crossroads 4740 [Inaudible] Indianapolis, IN 48205.
BELVA SMITH: That was kind of my fault that messed up a little bit. I’m looks the phone when it was still trying to capture it. That was the return address on an envelope that I have here. Down going to jump ahead on my channel. If I were using voiceover, it would just be a swipe up on the screen to switch those channels, but without voiceover I just slid it across the screen. Now I’m going to point to that same envelope using the handwriting channel and pick up what I have written on the envelope. Handwriting you do have to capture.
SPEAKER: Processing. [Music tones]. Belva Smith, 123 Easy Street, Indianapolis IN.
WADE WINGLER: Wow.
BELVA SMITH: That’s amazing, right?
WADE WINGLER: That’s amazing. Your address is 123 Easy Street?
BELVA SMITH: Yes.
WADE WINGLER: This is what I would consider very average handwriting. It’s not like you block printed or did anything unusual.
BELVA SMITH: And I didn’t use a 20/20 marker. I just used a black pen and wrote my name as clearly as I could write it. I’ve even scribbled it with handwriting, and it is a pretty good job. But for this demo, I want to make sure it was going to do a really good job so I did the handwriting nicely.
WADE WINGLER: You will have to try it on my handwriting, because I got the worst handwriting.
BELVA SMITH: That’s been a challenge, and that’s why this is kind of late to get here, because it does take a lot. Everybody’s handwriting is so different.
WADE WINGLER: I’m just writing my name on the back of your envelope in my regular handwriting, which is a very good. Let’s see what it does.
SPEAKER: Processing.
BELVA SMITH: It’s going to get it.
WADE WINGLER: I like it’s dance music.
BELVA SMITH: I know.
SPEAKER: Wade Winglare (phonetic).
WADE WINGLER: I like that better than Wade Wingler. And we can start call me that.
BELVA SMITH: There you go. That’s two different people’s handwriting, and it did a really good job at that.
WADE WINGLER: I write like a third grader, as was pretty impressive that it got that.
BELVA SMITH: We will see how it picks this up. This is the light channel. If a totally blind person enters a room — I have plenty of friends that are blind, and often I will walk into the room and say, hey, you know is totally dark in here? There like, no, I really don’t care. With this light indicator —
SPEAKER: Light. [High tone]
BELVA SMITH: That’s letting you know that the light is very bright. I covered the camera. [Low tone]. You hear how deep it got? That meant that the light got weaker.
WADE WINGLER: So the higher the tone, the brighter the light?
BELVA SMITH: Yeah. And I put it on the color identifier, and I’m pointing it at Wade’s pants, and it came back immediately and set gray. I pointed out my shirt.
SPEAKER: Gray.
BELVA SMITH: My shirt’s not gray; it’s black. But it’s a saying it’s gray.
SPEAKER: Gray and brown. White and gray. Smith might like it got because the light is so bright that I got the reflection. It will also do a scene.
SPEAKER: Scene preview.
BELVA SMITH: Scene does require you to capture so we will capture a picture of a Wade.
SPEAKER: Processing.
BELVA SMITH: Another thing, the app doesn’t save anything.
SPEAKER: A man standing in front of a laptop.
WADE WINGLER: That’s pretty impressive. I’m sitting, but yeah. It didn’t say a stunningly handsome man, but we’ll go for that.
BELVA SMITH: One of the channels is for currency, so we can also identify money and people, and you can teach it who the people are. If you want it to remember who a person is, it’s will remember that.
SPEAKER: Person.
BELVA SMITH: So I’m going to do it again.
SPEAKER: One face near top edge — processing.
BELVA SMITH: It’ll come back and tell me what kind of mood they think he is in.
SPEAKER: 47-year-old man with gray hair and a mustache looking neutral.
WADE WINGLER: Wow! I actually turned 47 in about two months.
BELVA SMITH: Todd told me to get rid of the app because it’s that he looks 67. It’s not always accurate on the age.
WADE WINGLER: That’s a little bit creepy.
BELVA SMITH: I should point out, though the app doesn’t to save anything, you can email it or text it. On the thing it will do is if you have photos, you can send them to Seeing AI and it will give you a description of the photo.
WADE WINGLER: So if it can’t get it, they will respond? Sort of like some of the other apps that do that?
BELVA SMITH: Yes.
WADE WINGLER: So when you say email it, is it emailing the picture or the picture and description?
BELVA SMITH: I think it’s just the picture. Now we are going to switch over to the product channel. I’m going to hold it over my bag of chips.
SPEAKER: Processing.
BELVA SMITH: All I did was put it over the back of chips —
SPEAKER: Doritos tortilla chips cool ranch.
BELVA SMITH: Move the phone around until it picks it up.
WADE WINGLER: It read the barcode, right? That is what it keyed on?
BELVA SMITH: Yes. Sometimes if there is additional information, it will include that; however, it does not read that. You would have to have voiceover on. Over does have some font enhancers at the bottom. In this case, it tells me that the warning is that this does contain milk.
WADE WINGLER: So it will do nutritional information and other stuff if you have voiceover on?
BELVA SMITH: If it is provided? It has to be provided in the barcode. Not all barcodes include the information.
WADE WINGLER: I don’t know what this can is you are bringing up here cop but it is a beautiful purple can. That is a very pretty can.
BELVA SMITH: That’s why I started drinking it, because the can is so pretty. Here we go. I’m just turning the can —
SPEAKER: Processing.
BELVA SMITH: — Until it got the barcode. Again —
SPEAKER: Ultraviolet energy drink.
BELVA SMITH: I didn’t have to worry about pointing it in the exact place. I just set the can down and turned it until he give me the information. A lot of the soups and stuff will give you the cooking directions. This really just tells me that it is an ultraviolet energy drink.
WADE WINGLER: What is an ultraviolet energy drink?
BELVA SMITH: It is a Monster. I don’t know.
WADE WINGLER: What does it taste like? Smith might like grape juice.
WADE WINGLER: So it doesn’t taste like flowers?
BELVA SMITH: No, it tastes like carbonated grape juice.
WADE WINGLER: Does it work? Does it pep you up?
BELVA SMITH: Oh yeah. That’s how I am in here for the next two hours. Then the last thing we will demo is the document channel.
SPEAKER: Document. No edges visible.
BELVA SMITH: It is give me the verbal — I had my phone — if this is interesting. I did not know this. I have my phone on mute, and it is telling me I can’t read it with my phone on mute.
WADE WINGLER: Because it is using the secondary audio?
SPEAKER: Communication audio. Alternative communication. AAC devices are devices that help those who have lost —
BELVA SMITH: I’m going to pause it because that was a long page out of the book.
WADE WINGLER: So it is reading a full page?
BELVA SMITH: It’s going to read the full page. You could use it to read a book. It might not be the best situation because it is not saving it. It’s just capturing and giving it to me. But you can also have the font enhancers, so you can make them larger. You can also email it to yourself if you need to.
WADE WINGLER: So is not going to save the book to you would have to scan a page, read a page, scan a page copied a page. It won’t scan multiple pages at the same time?
BELVA SMITH: It does not do multiple page scanning yet. But I think that something that they will probably bring into it. Did I see you read in for some currency?
WADE WINGLER: We are going to do a test. I’m going to ask you to close your eyes. I’m going to give you two bills here, and you have to see if you can figure out what they are. I’m going to hand them to you.
SPEAKER: Currency preview.
BELVA SMITH: I have to do it without my eyes open?
WADE WINGLER: Yep.
BELVA SMITH: It’s going to be pretty easy, is what I’m going to do is pull them apart, because you gave them to me together. I’m going to open them up and go to the top left of my phone. Come on, work with me, phone. Weight. It is folded. I felt it hitting my fingers. Okay.
SPEAKER: Twenty US dollars.
WADE WINGLER: She held it to the back of her phone and moved it away. That’s awesome. It found it very quickly.
BELVA SMITH: I went to the top left because I know that’s where my cameras at. I’m going to do the same thing again.
SPEAKER: Twenty US dollars.
BELVA SMITH: Is this 20 too?
WADE WINGLER: It’s not.
BELVA SMITH: So I didn’t get it done. Why isn’t it getting this one?
WADE WINGLER: I’m smiling here. Let’s see how it does.
BELVA SMITH: This feels funny.
WADE WINGLER: It’s not monopoly money. It is still folded.
SPEAKER: Two US dollars.
WADE WINGLER: There you go. It’s a two dollar bill. I didn’t know if it would get it or not.
BELVA SMITH: It did.
WADE WINGLER: It sure did. You had the two laying on top of the twenty, so that’s why it kept saying. My granddad always carried a two dollar bill, so I do. Now that my son who is six is starting to lose teeth, I’m trying to get more two dollar bills because the tooth fairy is bringing him.
BELVA SMITH: Under the thing that we should say about it is it’s kind of a use at your own risk type app. There is no tech support for it. However, Microsoft is inviting everyone that is using it to send them any kind of feedback that they do have, which I think is helping it develop and become better. I think in July when it was first released, it wasn’t as accurate with the page scanning as it is now. I know it didn’t have the last update, is the one that included the handwriting, which was just about four or five weeks ago. Each update, it gets a little bit better and your feedback is very important. It is totally free and available at the App Store.
WADE WINGLER: If somebody wants to get a copy, they are just going to go to the App Store in search for Seeing AI.
BELVA SMITH: That’s correct. Each channel, by the way, when you open the app for the very first time, each channel will have a little tutorial that will talk you through how to use that channel and what that channel will do for you. Then later, as you go on, if you decide I think I’m going to use the object identifier for the first time, and I’m not sure how to do that — of course, if you been using it for very long, you will know. It’s available in the help. In the top right is that question mark help button. You can tap that, and it will start the audio tutorial for you.
WADE WINGLER: Are there things it is not doing very well yet? Are there times when you found yourself frustrated with it?
BELVA SMITH: I have not yet. Some of that could be because it is free. What can you expect for free? They all say you get what you pay for. To me, this is amazing. What I’m afraid of is they are going to get everyone hooked and then they are going to start charging for it.
WADE WINGLER: Maybe. It’s a pretty impressive value add right now. People are really getting a ton of value from it. I’m hearing about it everywhere.
BELVA SMITH: Microsoft is really big into this AI stuff right now, which is why this started to get developed. I don’t know if it is rumor or not, but I did hear that they do have 30 people working solely on this app.
WADE WINGLER: I’m guessing they are not doing it just for the assistive technology stuff. It’s about all there AI, recognizing all kinds of objects, because if you can make it recognize whatever well enough to teach somebody who is blind or visually impaired what it is, then a computer can read that and start doing other kinds of artificial intelligence.
BELVA SMITH: I think you are right.
WADE WINGLER: If people wanted to learn more about you or reach out to you, what’s the website? Let’s plug ATFAQ since we spend some time there. If people wanted to reach out to you, how would they do that?
BELVA SMITH: They can reach out by going to EasterSealsTech.com. I think you’ll find a profile with my email address. If you have a question you want to share with ATFAQ, we do have a twitter, a phone number, and an email address. I don’t know any of that. That’s Brian.
WADE WINGLER: Brian will be here in a little bit. If you want to reach us on ATFAQ, just tweet the hashtag ATFAQ, which is assisted technology frequently asked questions. Or you can send an email to tech@EasterSealsCrossroads.org. Or if you want to call our listener line — frankly, we love when people call our listener line and we can put your voice on the show — the number is 317-721-7124. Belva, thank you so much for being with us today.
BELVA SMITH: Thank you, Wade.
WADE WINGLER: Do you have a question about assistive technology? Do you have a suggestion for someone we should interview on Assistive Technology Update? Call our listener line at 317-721-7124, shoot us a note on Twitter @INDATAProject, or check us out on Facebook. Looking for a transcript or show notes from today’s show? Head on over to www.EasterSealstech.com. Assistive Technology Update is a proud member of the Accessibility Channel. Find more shows like this plus much more over at AccessibilityChannel.com. That was your Assistance Technology Update. I’m Wade Wingler with the INDATA Project at Easter Seals Crossroads in Indiana.
***Transcript provided by TJ Cortopassi. For requests and inquiries, contact tjcortopassi@gmail.com***