ATU722 – IMPACTech Podcast Crossover with Dr. Mary Goldberg, Brandon Biggs and John McInerney Part 1

Welcome listeners to a very special crossover episode of Assistive Technology Update. Today, not only are we joined by Brandon Biggs and John McInerney to talk about the amazing technology called Audiom, but we’re also joined by the brains behind the ImpacTech podcast, which is part of the IMPACT Center at the University of Pittsburgh Department of Rehabilitation Science and Technology. So we’re going to get right into talking about the amazing ImpacTech podcast and partnering with them, and then we’ll get into talking to our other guests. Well, everybody, welcome to the show.

Thank you for having me.

Thank you so much, Josh. Excited to be here.

Yeah, thank you all so much for joining. And I guess Mary, let’s just start with you. Can you tell us just a little bit about the IMPACT Center and the ImpacTech podcast?

Yes, I’d be happy to. Thank you so much. So the IMPACT Center is funded by the National Institute for Disability, Independent Living, and Rehab Research. And we’re a technical assistance center to help support getting assistive technologies from the “lab” to market and the products to the intended users. We support both small businesses, and the small businesses would typically be recipients of NIDILRR, that’s that long acronym, Small Business Innovation Research grants. And the universities that we support where the innovators are within academia are typically recipients of other funding mechanisms within NIDILRR, including larger research centers that would have multiple sub-projects, about half of which, maybe around two to three would be product development- oriented. Again, all assistive technology. They also have a field-initiated project, which would just be typically a single project focused on a single product.

And so what IMPACT does is work with the innovators across the small businesses and universities to help them get their products translated to the intended users and help them to figure out which pathway is best. And we found through ImpacTech, so the podcast arm, we like to break it down in plain language and talk to innovators, particularly those that have been successful, but of course have encountered challenges and barriers along the way to help save innovators in the future in terms of time and money and help them identify facilitators that would help them get over the barriers. So that’s who we are and why.

Awesome. Awesome. We’re so happy to be able to partner with you on this. John, Brandon, whoever wants to go first, can you tell the listeners just a little bit about yourselves?

I can jump in right now. Hi, my name’s John McInerney. I am blind. I have retinitis pigmentosa. I lost most of my functional vision in my early forties. I have an engineering background. And with assistive technology, I was able to complete a very successful engineering career.

Once I completed that career, I got involved with the IMPACT Center and with some other nonprofits in the state of Pennsylvania and spent two years as the CEO of the Pennsylvania Association for the Blind. Today I continue to interface with the IMPACT Center and with a nonprofit that serves visually-impaired individuals in both the Pittsburgh and the Philadelphia area.

So I’ve had the opportunity to do some evaluation of Brandon’s Audiom and I would say that it’s a very successful and accessible software package for reading their [inaudible 00:04:34] maps. Did I say that right, Brandon? I’m not sure I got it right.

But anyway, I appreciate the opportunity to participate in the crossover podcast today. And with that, I’ll turn it over to Brandon.

Thank you. Yeah, my name is Brandon Biggs and I am the CEO of XR Navigation. I am also an engineer at the Smith-Kettlewell Eye Research Institute and a PhD candidate at the Georgia Institute of Technology.

The work that I do centers around creating digital technology that enables blind people to fully access spatial diagrams or geographic maps. And I’m also totally blind myself, and that’s really the main inspiration for me to do this is because I can’t access geographic maps now and I really want to be able to. And I also can’t access anything around building floor plans or architectural diagrams. Say if I wanted to build my own house like my brother’s doing right now, I wouldn’t be able to preview and approve those floor plans. And so it’s really important that I am able to access this information. And so that’s what inspired me to really start working on this project.

Awesome. Mary, since your podcast usually kind of focuses on the why and the how in AT development, why don’t you start us off?

Sure. I would be happy to. So Brandon, I guess we understand the why I think, motivated by the inaccessibility of what’s out there already, but maybe you could talk to us a little bit about the how, what gave you the seed funding to kick this off and what kind of infrastructure did you have access to?

The primary project started when I was getting my master’s degree in inclusive design from the Ontario College of Art and Design University. And my master’s advisor when I came into the program asked what are the biggest barriers to accessing graphics currently, especially data representation graphics, because that’s what was his primary focus. And coming into the program, I had been working. My first company was providing teachers of the blind and visually-impaired orientation mobility specialists and material preparation to K12 schools around the US. And it was just a little bit scary how much money some of these K12 institutions were spending on creating accessible materials. And the sad thing was is that they would create them once and then they would be kind of disposable and they were only accessible and only usable to those blind students. And so there was just very little efficiencies in this whole entire process.

And so when I came into my inclusive design program, I said, “There must be a better way to create tools that are usable by everyone or as many people as possible.” And so coming into that program, my advisor said, “What are the biggest challenges here? Can you make me a graph?” I said, “Sure, I’ll make you an accessible sonification graph.” And then he’s like, “Make me a Sankey diagram.” And I was like, “What’s that?” He’s like, “Well, it’s a graph on top of a map.” I was like, “Oh, wow. Okay, that might be a little more difficult.” So that was my first introduction to maps and representing maps digitally.

So we built the Napoleon’s March of 1812, which is a really famous diagram in the infographics community, and that’s where it began. And we realized that digital maps have had little to no research in the academic community. But because I am totally blind myself and grew up, I’m congenitally blind, so I was born blind, I knew that there were these things called audio games, which are games that can be played completely using audio that are often made by and for blind people. And they have maps, lots of maps. Really complicated ones that allow you to move troops around different types of terrains, allow you to move around a post-apocalyptic zombie apocalypse landscape through abandoned stores, navigate through mazes and dungeons and all that kind of stuff. So maps had been a part of my life growing up, digital map exploration, but I never was able to access the real world, just these fantasy landscapes. And so I said, “Let’s take that idea and transfer it to real-world maps.” And so that’s basically what was the inspiration for how we created the project that we’re doing now.

So very briefly, the Audiom experience is like a video game where users hear the name and sound of objects as they move a character over them with their arrow keys touch screen or other controller. And an audio map can be accessed through a web browser on any device, and so you don’t have to install a single thing. And we also add on what are called spatial audio sounds as kind of ear candy where you can hear these looping sounds of objects around you and you can rotate your character and turn towards them to find them.

And so that’s kind of the how we actually ended up getting the inspiration for the interface and then how the interface basically works.

Got it. And so student project turned more full-time job for you. How did that all occur?

When I graduated from my master’s program, the Smith-Kettlewell Eye Research Institute offered me a position to continue doing the research that I was doing on Audiom. And as part of that, we got a diversity supplement from the National Institutes of Health. And that was during COVID and we ended up getting some COVID supplements from some NIH, National Institutes of Health, grants to continue funding Audiom.

So we built the first real accessible COVID statistic map and we have that on our website right now. And we also built a neighborhood exploration tool that allows you to kind of pre-trip plan before you step out. So we have an open street map integration. And so that was a National Institutes of Health grant and that really created the groundwork and infrastructure for what we’ve done going forward.

And we also ended up getting on a few of these NIDILRR grants. The big RERC project that the Smith-Kettlewell Eye Research Institute is on has a few aspects of Audiom in there. So in this current one, we have one where we’re building out the touchscreen experience of Audiom. And so we also started a company to commercialize this product and we filed a patent and we also got an SBIR from the National Institutes of Health as well.

So that’s kind of the primary funding mechanism that we’ve received. We’ve also participated in the IMPACT Center boot camp and we got through all three phases of the IMPACT Center programs. And we also participated in the Remarkable Accelerator and a few other accelerators at Georgia Tech and MassChallenge. And so we’ve done a few accelerators, which have also provided some funding, and then the National Federation of the Blind also invested in our company as well through the Smart Jobs Project, or company. I’m not sure what they are, organization.

Thanks, Brandon. So for the listeners that don’t know the SBIR mechanism or like I defined it before, the Small Business Innovation Research grant is awarded to businesses. So being that the Smith-Kettlewell Institute and your undergrad institution would be non-profits, right?

Exactly.

So how did you make that division or choose to separate or choose to create your own company to be able to be eligible for those different funding sources?

The non-profits don’t do any kind of money-making activity necessarily. They’re not really set up for that. So say for example, I wanted to bill… For example, this is a good example, we are going to be attending this CSUN Assistive Technology Conference and we are the official map provider for that conference in a couple of weeks in Anaheim, California.

And in order for us to bill for that and actually create the map and create all the infrastructure for creating those maps in a systematic way, we need to have a company in order to do that because neither universities or non-profits are really set up for that. You can set up a non-profit to do that, but Smith-Kettlewell is not really set up for that.

So we created our company because it needs to be in the marketplace, people need to be able to get these maps in their hands and there needs to be a place for government agencies or institutions with maps to go to in order to make their maps usable. And so that’s why we created the company because a research institution just doesn’t have the infrastructure or desire really to become a private company doing commercial deliverable work.

Yeah, got it. That makes sense. Thank you. And so your current role then, are you split between Smith-Kettlewell as a researcher and then you also have the business on the side?

Exactly, yes. So we have an agreement between the Smith-Kettlewell Eye Research Institute and XR Navigation to split IP and future IP. They own a part of XR Navigation. And so that’s how we are creating that… That’s how we kind of agreed to handle that process of we continue doing work at Smith-Kettlewell, we also are doing work at XR Navigation, and if we didn’t have this agreement in place, it’d be a little bit messy.

And so my PhD institution, I do no development work under that institution. I only do research evaluating the existing interface. So basically I’m very, very careful not to do any work that would be construed as creating IP at my PhD institution.

It’s wonderful that they’re understanding though, and really such a nice launch pad for you, Brandon, to have some experience as both an entrepreneur and a researcher within a larger institution. I would think good for you as a future innovator to have some options and great for the users too to hopefully have quicker access to your product.

Yeah.

Thanks so much for sharing the backstory. That’s great.

Absolutely.

Yeah, Brandon, I wanted to ask, because you mentioned that you were making the maps for CSUN here coming up very soon. I guess what does that process look like without getting too code-specific or anything like that, but what does the process look like from you maybe receiving the maps of what they visually might look like or just how does that all happen?

There are three steps to get a map into Audiom. So first, a customer will give us any kind of geographic data they already have. For conferences, this is typically a PDF floor plan. For other bigger cities, for example, this might be access to their Esri database. Or for other types of institutions, this might be a mixture of both some kind of geographic information in a database and a PDF.

And so what we’ll then do is step two is we’ll process that data into something Audiom can use. So if it’s already in a geographic information system format, then we just make sure that that is in a what’s called a vector format, which is a computer-readable geographic information format. If it’s a PDF, I have a cartographer who then manually goes through and redraws that PDF. They sketch over the top of it and turn it into vector data. And then we put that information into Audiom and then provide the customer… The third step is we provide the customer a link that they can embed or link to on their website that has the full inclusive map.

Are there other Audiom maps out there that offer a comparable product? And if so, what differentiates your product from theirs?

There are no existing tools out there that really provide a similar experience. There are definitely some that are sort of similar. So I would say the biggest similar experience you can get are actually not really a product per se, it’s just more of a technique of creating detailed alt text. And you typically need to create an 80,000 word document that details every single feature on that map. It’s shape, size, orientation, general layout, and make sure you’re hitting every feature on that map.

And there’s a project called the UniDescribe project that creates national park maps, and I’m working with them right now to kind of systematize this process and making sure that you are really getting equal access when they create these text descriptions because it’s possible. It just needs to have a really clear index strategy and really clear system for creating these.

You can’t just say… I’m trying to provide public feedback right now to a park in my neighborhood. They’re proposing some new changes. So I gave them, I wrote a tutorial on how to create these detailed text descriptions. They tried to follow it, but they’re amateurs at it. It’s the first one that they’re doing. And so they say stuff like, “This map shows increased seating areas around the park.” And that’s just not enough information. I can’t give you any feedback on that seating arrangement or provide any feedback whatsoever. I just know that the park needs more seating. But here are the areas by the farmer’s market that I need seating by because that’s a place that we always end up sitting on rocks by to eat our breakfast when we go to the farmer’s market. So those are things that are local, civic participation that I need access to, but you can’t just create a text description without being very systematic about it because you’re going to miss information.

And so what Audiom does is makes the exact information that the sighted people are using fully usable in the non-visual format. And so if the sighted people are missing some information, then also the blind people are missing the information. And so it’s equal access. And it’s easier to create visually this data than it is to create a text description because text description does create a separate but equal solution, and it’s often out of date sometimes if the map is dynamic, for example. So there’s a lot of issues with text descriptions. But as it relates to dynamic maps, there’s nothing out there for blind people right now.

Brandon, I guess as we kind of talk about this and we talk about all the information that you can get, what all kinds of information can be relayed by Audiom? Pretty much anything that’s out there or are there limits or have you even found those yet?

We can show any type of map. The question is how much work is it going to be for us just to kind of update the interface. And for some places it might be more expensive than they can handle right now, and that’s just because we haven’t optimized for that particular type of map. We are really good at exhibit halls and buildings and campuses. Anything beyond that, it’s more expensive for us. We can definitely do it, it’s just going to take us a little bit more effort and time to do.

And so it’s just kind of a matter of building out the user experience to make sure that’s a really good default experience for the user when they get that map. So that’s kind of where we’re at right now. So right now we’re optimizing for how do we facilitate making more of these maps. But yeah, that’s kind of the long-winded answer for that question. Did I answer it?

Yeah, most definitely. Most definitely.

So Brandon, I kind of have a practical question. So being an individual that’s blind that uses a cane and uses some smartphone apps, if I was going to CSUN, how would I use your Audiom map to prepare myself first for the venue? And secondly, can I use it as I travel through the venue?

So I am definitely somebody who prefers using my keyboard and computer, so this is how I would do it and how I’ve done it. I go onto the CSUN map on my computer and I will activate the map and explore around the hotel, because you start at the door to the hotel, and you can go explore around, learn where the infused bar is, which is just the south of the door, and then you can go through the hallways, and it’ll say that you’re in the hallway, and go learn where the different meeting rooms are. So you’re kind of moving your character around to simulate walking through the space.

And so you’re using your arrow keys to move around the map. And if you do know, say for example, you need to go to Platinum AB or something like that, you can use a search box or a menu and find Platinum AB and get directions to virtual directions to that location and you can kind of look where the doors are. You want to make sure that you’re going through the doors and not through the walls when you’re going into those rooms. And so you can kind of learn where you need to be trailing along with your cane or your hand to find the doors and you can count them and stuff like that.

So that’s kind of how you would really basically go through. You can use your phone as well to navigate around. So similarly to the computer, there’s a panel of buttons that show up on your touchscreen and you can double tap on the left button to move left one meter or double tap and hold to repeatedly move around one meter in one direction. But it does not sync with your current location indoors. It’ll sync with your location outdoors, but GPS doesn’t really work inside and GPS is the only real free method for us to use.

If you do want to have Audiom sync with an indoor experience, you need to ask a place like GoodMaps to sync with the Audiom experience. And they do have a GoodMaps version that you can use to navigate around the hotel with computer vision. And so something like that could allow you to sync your location to Audiom. But they haven’t integrated with us yet. They need more people to ask for it. So that’s kind of the… If we want to be able to do that, that’s the next step there.

Listeners, that is all the time we have for this week, so make sure to come back next week. Also, make sure to check out the ImpacTech podcast. We’ve put the link to their show down in our show notes that you can easily find it. You can also find the ImpacTech podcast wherever you find your podcast. We’ll see you back here for the conclusion next week.

Do you have a question about assistive technology? Do you have a suggestion for someone we should interview on Assistive Technology Update? If so, call our listener line at 317-721-7124, send us an email at tech@eastersealscrossroads.org, or shoot us a note on Twitter @INDATAproject.

Our captions and transcripts for the show are sponsored by the Indiana Telephone Relay Access Corporation or InTRAC. You can find out more about InTRAC at relayindiana.com. A special thanks to Nikol Prieto for scheduling our amazing guests and making a mess of my schedule. Today’s show was produced, edited, hosted, and fraught over by yours truly. The opinions expressed by our guests are their own and may or may not reflect those of the INDATA Project, Easterseals Crossroads, our supporting partners, or this host.

This was your Assistive Technology Update. I’m Josh Anderson with the INDATA Project at Easterseals Crossroads in beautiful Indianapolis, Indiana. We look forward to seeing you next time. Bye-bye.