For the third year of my engineering scholarship, I was required to accomplish a month long project among a set of domains : Biotechnologies, Embarqued systems, Computer Science and Robotics. I decided to go with the last one, Robotics, as the project proposed in the other domains were not satisfying to me. Computer Science was interesting, however the projects proposed in it were much too similar to other projects I had to take on during the year.
Afterwards, I had to decide on which subject to explore. In a team of four, we decided to take on the following project statement : "Use the Pepper Bot to serve as a guide and assistant in a restaurant setting under Covid Sanitary restrictions".
Pepper is the name given to a humanoid robot designed by SoftBanks Robotics, equipped with articulated arms, head and bust, mounted on a set of wheels for locomotion. It is able to use cameras and Lidars to detect its environment, find faces, as well as hear, understand words, and speak.
1 . Appearance of the Pepper Robot
Using all these elements, we were expected to design an application for Pepper to greet clients into a fictional restaurant, verify their masks and offer hand sanitizer, before bringing them to the table they asked and return to its starting position.
In order to accomplish that task, we made use of the Choregraph Software (designed to program Pepper), in addition to custom coding in Python using the Pepper API .
Here are the key features we managed for this project:
- Mapping of an area and assignable points for the robot to move to (where the tables are);
- Facial recognition and detection of the mask through python;
- Idle animations while talking;
- Custom animations for unusal tasks;
- Multiple speech options to allow for a more natural interactions.
2. Mask detection AI, trained to recognize when a face is partially covered
This AI used a large databank of faces with and without masks to first of all be able to find faces, and then detect if they wore a mask. This was a difficult part to integrate as it used a version of Python more recent than the one on Pepper's Software, therefore we had to go around the issue by sending images used by pepper to a remote server, have that server collect the data and use it in the AI's program, then convert it to treatable data for Pepper and send it to it. This however made the process longer and more tedious, but allowed us to accomplish a key point of our application. Reusing a part of this AI also made us able to read QR codes, useful for checking sanitary passes or reservations.
3. Previsional preparations for the mobility of Pepper
In order for us to make Pepper mobile to escort the clients to the tables,we had to evaluate how and where it would actually move. Up above are a set of designs we made before coding to brainstorm about the methods. On the left, a sketch of the room and our first ideas for movement trajectories. On the right, a very simplified representation of our program, with the different steps of the process. We first attempted brute force movement, however Pepper could only move in relation to itself, which meant that any disturbance, error in movement, or obstacle would add up and make trajectories extremely unreliable. Therefore, we explored the API and started using the Navigation module of Pepper. By using this, we were able to have the robot roam around the room for a while, and use its captors to create a virtual map of the area in which it could locate itself, and know to avoid obstacles. Furthermore, moving using navigation allowed pepper to notice unknown obstacles on its way and recalculate its route instead of cancelling and returning an error.
4. Navigation map by Pepper after exploring a region
Using all these modules, we added a few speech recognition patterns to pepper, so that she would understand different answers and be able to react accordingly, used its animator to create a custom animation in order to have pepper dispense hand sanitizer on demand, and the project was about done.
In summary, after our work here is what Pepper was able to do :
- Notice when a customer arrived
- Greet them to the restaurant
- Look at their face and verify if they had a mask
- If the client doesn't, Pepper will ask them to put one on, until they do have a mask
- Offer hand sanitizer and dispense it if the client agrees
- Ask for the client's reservation QR code
- Using the reservation, call the client by their name and guide them to one of three tables, with the client telling which one he desires
-After making sure the client is well seated, Pepper goes back to its initial position, and awaits a new client's arrival
If this isn't clear enough, here is a demo video of the process (note : the application was designed in french)
Comentários