Autonomy and AI with Dr. Amanda Sgroi

How is AI driving the development of the next generation of robotic systems?

24 Nov, 2020

author avatar
Dr. Amanda Sgroi, Principal Research Scientist, RE2 Robotics

Dr. Amanda Sgroi, Principal Research Scientist, RE2 Robotics

Dr. Amanda Sgroi is a Principal Research Scientist at RE2 Robotics. In this interview with RE2 Business Development Manager Travis Schneider, Amanda discusses the ways that autonomy and artificial intelligence are driving the development of the next generation of robotic systems. Amanda also explains how RE2 is applying its autonomy and AI modules, RE2 Detect and RE2 Intellect, to enhance the capabilities of its mobile robotic arms.

This interview was originally featured as the first episode in our video and podcast series, The Robotics Engineering Experience. You can find the series on our YouTube channel, or wherever you get your podcasts. Below is a transcript of the interview.

Autonomy and AI with Dr. Amanda Sgroi

Travis: This is Travis Schneider. I'm with RE2 Robotics, and joining me today is Amanda Sgroi, principal research scientist at RE2. Amanda, thanks for joining me this afternoon. Just to get things started, Amanda, could you tell us a little bit more about yourself like what you do at RE2?

Amanda: Like you mentioned, I'm a principal research scientist at RE2, which really means I wear a couple of different hats. I work with business development, so I'm customer-facing, looking for new business, writing proposals. I also do project management. I help manage a couple of our programs as they relate to our autonomy efforts and then I also do engineering work. My specialty is computer vision and machine learning.

Travis: What brought you to Pittsburgh originally? It seems like you have a lot of capabilities and there are a lot of people doing that on both the coasts--but what’s so special about Pittsburgh?

Amanda: I think there are two main things. One is Pittsburgh is a huge robotics hub right now. There’s a lot going on in “Robotics Row,” where RE2 is located, as well as throughout the city and Pittsburgh's really close to my hometown. I grew up in Johnstown, Pennsylvania, which is about an hour and a half east of here. I'm close to family while I get to do the thing that I really enjoy doing.

Travis: We're obviously both recording this at home, but when we're not being quarantined, what are some of the things that you like to do in and around Pittsburgh?

Amanda: Since I've moved to Pittsburgh, I've become a pretty avid runner. I’ve run a couple of the Pittsburgh half marathons in the last few years. I really enjoy the Heritage Trail and, running at North Park. When I'm not doing that, I’m hanging out with friends and exploring the craft beer scene in Pittsburgh, which has really blossomed in the last few years as well.

Travis: As a technologist or a technology leader in the region, what are some of your inspirations? Who inspires you or do you draw inspiration from science fiction? What are some of the things that really inspire you?

Amanda: My original inspiration was really during my career in undergrad when I went to Duquesne University. I was pulled into a research group during the fall semester of my freshman year under Dr. Stacey Levine. It opened up my eyes to this whole different world of research. In high school, I did research papers where you're just going to the library and reading things. I had never realized that you can do research and apply it and solve a problem. That was the inspiration where image processing and computer vision came from. That’s really what has drawn me to robotics--people in robotics who are taking their research ideas and applying them to the real world.

Travis: Related to that, I'm just curious too, because from my own perspective I know I struggled transitioning from high school to college, figuring out what it was that I really wanted to do. Could you talk a little bit more about like what drew you into Duquesne and general science, technology and engineering for people who might be interested?

Amanda: In high school, I always excelled at mathematics and I knew that I wanted to do something related to mathematics. My senior year, I took a Java course and that was my first experience programming. I really enjoyed it. Whenever I was searching for colleges, I was applying to both math programs and computer science programs. The great part about Duquesne was that their math and computer science department were all-in-one, and so it seemed like a solid path for me to get both a Bachelor of Science in Mathematics and in Computer Science since there was so much overlap.

The other thing about that department was that so many of their faculty were doing research even though it was a small department. Math and computer science are considered liberal arts at Duquesne. They're not actually in the science school. It was also a really interesting perspective to have on both disciplines.

Travis: Did you did you stay at Duquesne for graduate studies, or what were your next steps there?

Amanda: At the time, they didn't have a masters in computer science, which they're actually rolling out this year. I did switch to the University of Notre Dame, where I completed my masters and Ph.D. I was in the computer vision research group, research laboratory and, based on the funding at the time, our focus was biometrics. I was doing face recognition and iris recognition as my focuses. That’s really where I started doing some BD activities, not knowing that they were business development--finding my own proposals to write, my own funding, meeting with people in the government, convincing them that my ideas were good ideas to fund.

Travis: Where did your interest around artificial intelligence and machine learning come to be? It sounded more from the mathematical, statistical side of things and then you got introduced into more visual stuff. Could you talk a little bit more around that transition there?

Amanda: I think there's a really strong relationship between computer vision and artificial intelligence machine learning right now. That has really evolved as I have grown through my academic and industry- based career.

For example, I was doing more image processing in undergrad, which has a little more of a mathematical foundation to it. Then I went into the computer vision world, which is more heavily I would say engineering. Like you mentioned, there are a lot of statistics around proving that what you're doing is a good solution. My graduate career was really my first introduction to machine learning. That’s when deep learning started to blossom as well.

When I moved into industry, it was staying on top of what was state-of-the-art and applying that to what I was doing in the industrial world. Initially, that was through multispectral image processing and then it was for ground robotics and now it's for robotic arms.

Travis: What are some of the things about robotic manipulation or robotics in general that just get you excited right now?

Amanda: The biggest thing for me is I get to research the ways that I'm going to solve a problem. I get to solve that problem at my computer. Now that we're stuck at home, this is a great time to do some simulation work, but then I also go out into the lab and see it work on the hardware. I think that's the most exciting thing about robotics. You’re not just building an application. You’re building something that's interacting with the world.

Travis: I agree. That’s something that really interests me, too. It’s not just a lot of theoretical ones and zeros floating across the screen. Seeing things actually move and perform meaningful tasks is always more interesting.

Jumping topics here for just a minute, now that we're talking about more specifically the field of robotics. Maybe we jump into a big topic right now and that's generally speaking autonomy.  Autonomy means a lot of different things to a lot of different people depending upon I think who you're talking to. I guess generally speaking, what does autonomy mean to you?

Amanda: To me, it's really the ability for some machine to act completely independently. We talked about in the ground autonomy, autonomous cars and different levels of autonomy, there but there are really different aspects to human intervention. Autonomy in general is acting without any human intervention at all.

Travis: I know there was in the earlier days of a lot of our ground robotic platforms, there was more teleoperation, but it seems now at least from my perspective we seem to be getting a lot more requests for fully autonomous systems. Definitely more of a trend that at least I'm seeing from my perspective. In general, how do you see RE2 applying autonomy and its applications today?

“Autonomy in general is acting without any human intervention at all”

Amanda: I really think that we're taking the right steps towards a fully autonomous system by having semi-autonomous systems first. It’s a huge leap for some of our customers to go from a teleoperated world where humans are always in the loop and helping make the decisions all the time to letting machine go and do its own thing without actually having supervision.

Right now, RE2’s working on a lot of programs where we have our perception and autonomy team identifying objects in the world, telling the user how our robotic arms are going to manipulate them and then getting approval to do that and getting reassurance from the user that hey, this is the right thing to do, with the hope that as we get our users more accustomed to autonomy that we can move more towards those fully autonomous systems. This also allows us to optimize and refine our algorithms to make sure we're doing the right thing.

Travis: I think especially when I interact with a lot of our current clients or new customers, a lot of people come to RE2 with the whole mindset of hey, we're a robotic arm company, but I think more and more we've got a couple of unsung heroes in the background more from our software products that we're trying to promote around RE2 Detect and RE2 Intellect. Just curious if you might be able to help kind of illuminate some of the capabilities behind either of those solutions for our users from your perspective.

Amanda: RE2 built out an autonomy architecture and part of this architecture is RE2 Detect and RE2 Intellect. Detect is really focused on the computer vision, the machine learning, and selecting sensors for the applications that our customers are trying to provide solutions for, and that's really where the secret sauce of where is the object in the scene. Is there an object in the scene, and then how do I approach that object?

Then RE2 Intellect is really the high-level artificial intelligence. You can think of this really as a state machine of what are the steps that the system has to perform in order to complete an overall task. For example, if there's a panel and I want to open it and do something, I have to say, “Press a button to open the panel, do something inside and then close the panel.” RE2 Intellect is really the state machine that manages all of those phases.

Travis: Are those tools specific to a given application, or are those things that I can deploy kind of universally? How does that work?

Amanda: [On the] RE2 Detect side of things, it really depends on the objects that you're trying to manipulate, so those are very application specific. On RE2 Intellect, if you wanted to do the same task but in a different environment, maybe it's just something that you're tweaking for that application but, in general, if the applications are very diverse, then it again is specific.

Travis: It is really encouraging for me to see that those two products are really starting to take form. Are there other capabilities that your team is working on as well that are down the pipeline? Are there other things that you're kind of eyeing? I would imagine there are other software products that will be rolling out over the coming months or years.

Amanda: We're definitely working on advancing our trajectory planning capabilities. It is something that we can do now, but we want to make sure that we're doing it better and staying on the cutting edge. The other thing that we're working towards is advancing our operator control unit, so making sure our user interfaces are enjoyable for our customers to use and meaningful for their applications.

Travis: As you're leading a lot of our work specifically as it pertains to a lot of the software components within our technology, what are some of the challenges that your team is facing right now?

Amanda: I would say the challenges our team is facing is really resources. There are a lot of people really in the Pittsburgh area that have these type of skills, but are really interested in ground autonomy, and I would love to pull them on our team and show them how great robotic manipulation can be. From a technical standpoint, I would say our biggest challenge is data. I think a lot of people who are outside say the ground economy world, particularly in the government space, has this challenge. There are all these great tools for machine learning and deep learning, for object detection and classification, but you have to have many contextualized samples. That’s not always available or possible to capture through government programs or small commercial efforts.

Travis: We just spoke a lot about autonomy; now I want to talk a little bit more about one of the enabling technologies behind autonomy, and that's this whole topic of artificial intelligence. Similar to how we kicked off the discussion around autonomy, generally how would you define artificial intelligence to someone?

Amanda: Artificial intelligence is you can really think of it as the rules that are defining some set of actions. I think a lot of people talk about how machine learning and deep learning and tools like neural nets all relate back to artificial intelligence. You can really think of it as like the Russian nesting dolls problem. The largest doll, the largest bubble, is artificial intelligence and then inside that is machine learning. There’s a whole set of capabilities associated with what. Then inside machine learning is deep learning and that's where things like neural nets and the fully convolutional neural networks--all those hot topics right now--that's where those things really live.

Travis: What are some of your biggest pet peeves regarding the whole topic of artificial intelligence?

Amanda: The notion that robots are soon to take over of the world. We still struggle with robots opening doors autonomously. Not that we can't do it, but I think we really need to see some true capabilities and traversing areas and rough terrain and manipulating complex objects before I'll deal with the fact that robots are going to take over the world.

Travis: Where do you think that potential clients or current clients of RE2 should investigate artificial intelligence? Because from my perspective, I think a lot of people look at that as a solution and then look for problems, whereas it seems like oftentimes it would be more prudent for the problems that guide us back towards artificial intelligence. I wasn't certain if you had any perspective in that matter.

Amanda: I think it's sometimes a problem with buzzwords. Everyone’s like “Oh, are you going to use deep learning to solve this problem?” Not all problems need deep learning as their solution. A lot of times, if you have an object or a feature that you want to manipulate with a robotic arm and it's very geometric, you can use classical computer vision to solve those problems. It can be really reliable and very fast and run on very little hardware. Maybe that's a better solution for what you're actually trying to solve.

Travis: What are the application requirements that would guide me to actually needing to implement something that would leverage some degree of artificial intelligence?

Amanda: Artificial intelligence is kind of this largest bubble where things live. As I mentioned with RE2 Intellect, I think that the state machine that we're using to really guide all of the phases of what a system can do? That's of the highest form of artificial intelligence defining the rules of our system. If we're talking more on the perception side of things, the more unconstrained you make your problem, the more need there may be for artificial intelligence, machine learning and deep learning.

For example, at RE2, we do a lot of manipulation outdoors in uncontrolled environments and so, in those cases, you either have to have a lot of rules if you're using a classical computer vision method or you have to have a lot of data that defines all those edge cases for your environment to use a deep learning approach.

Travis: Artificial intelligence where you don't have very specific products or very specific processes provides you with a bit more flexibility in the way of manipulating components in the environment particularly when you're outdoors which is where we find a lot of our applications.

Given your experience, do you have any tips or tricks that you would recommend on to people who are considering applying artificial intelligence as part of their solution or things that they should be keeping in mind before they get started?

Amanda: I think the big thing is to really ask yourself what data you have available to you. If you don't have a lot of examples of what you're trying to work with, and you have to capture them yourself, maybe you take a step back into say classical computer vision. Or, if you have a lot of data available to you and it's very constrained, maybe you can use computer vision with machine learning.

If you have an extreme amount of data and you're ready to tackle those deep learning types of problems, then you have to ask yourself, “What's going to be my training set, what's going to be my testing set, and how am I going to make sure that I'm accurate in all cases?” I think it's really about reflecting on your data. That would be the biggest advice I would give.

Travis: Thanks for that. I certainly learned a lot today. Was there anything else that you wanted to cover?

Amanda: Other than RE2 is hiring and if you're interested in doing computer vision and machine learning for robotic arms please check out our careers page.

Travis: Thanks for your time Amanda. I truly appreciate it and talk soon I'm sure.

To learn more about RE2's technology, visit, or download the eBook, " Achieving Robotic Autonomy," from the company's Tech Resource center

24 Nov, 2020

More about RE2 Robotics

author avatar

RE2 Robotics develops the RE2 Sapien family of intelligent mobile manipulation systems that operate in a variety of complex indoor and outdoor environments for use in the aviation, construction, defense, energy, and medical industries. RE2’s mission is to develop mobile manipulation systems that em... learn more

Stay Informed, Be Inspired

Wevolver’s free newsletter delivers the highlights of our award winning articles weekly to your inbox.