Applied research collaboration moves virtual agents closer to reality

Imagine walking into a building and being greeted by a virtual agent that knows your name, updates you on the latest safety protocols, asks you questions and takes your temperature — all without the touch of a button.

Sound futuristic? Not as much as you might think. In fact, it's very close to becoming a reality, thanks to an interdisciplinary applied research collaboration between Toronto-based artificial intelligence (AI) technology company CloudConstable and staff, faculty and student researchers from Sheridan's Screen Industries Research and Training Centre (SIRT), Centre for Advanced Manufacturing and Design Technologies (CAMDT) and Honours Bachelor of Craft and Design (Industrial Design) degree program.

"Having highly-qualified researchers and students working on this project provided us with a different perspective."

— Mike Pickering, CloudConstable founder and CEO

CloudConstable founder and CEO Mike Pickering first approached SIRT last fall looking to improve the AVA^_TM (Animated Virtual Agent) he had created with support from the Intel Pandemic Response Technology Initiative (PRTI) program. SIRT provided some feedback on how to improve AVA's presentation features before referring the project to CAMDT, which helped secure additional government funding for the project and optimize AVA's mechanical and electrical systems. Industrial Design faculty and students then provided some additional touches to help commercialize the solution by building a prototype kiosk that was more attractive and practical than the original product.

"Sheridan has brought so much to the equation, giving us access to everything from students who have been learning in the relevant field, to PhD researchers, to equipment," says Pickering. "Having highly-qualified researchers and students working on this project provided us with a different perspective."

Humanizing self-service technology

AVA was initially intended to help children and seniors to communicate with friends and family in a way that was safer and more secure than using social media platforms. However, after having difficulty accessing consumer protection data and sensing a growing appetite for virtual customer support technology, CloudConstable pivoted the product to a business-to-business model.

"The COVID-19 pandemic has been an inhibitor to a lot of businesses and had far-reaching effects on our economy, but it has also presented a bit of an opportunity because public perception and sentiment have changed a lot. I think there's a lot more openness to speaking to a robot or AI device now than there used to be," says Pickering. "There are other self-service technologies out there, but they don't have a face or simulate a person-to person experience."

Oshawa's Ontario Regiment RCAC Museum saw value in the product last summer, installing one of CloudConstable's virtual agents to revolutionize the visitor and volunteer experience. Eventually, the animated character — named Master Corporal Lana — was programmed to perform COVID-19 screening, including temperature scanning, that helped the museum safely re-open its doors to visitors last fall.

"We had a working interim solution," Pickering says, "but the sensor array was a fairly crude prototype that had wires hanging out of it." Also wanting to improve the presentation of the screening and thermal scanning, Pickering turned to SIRT because of its expertise in screen-based technology. "There were some synergies there," notes Pickering, "and some very valuable feedback that will help us refine how AVA interacts with people."

SIRT then referred CloudConstable to CAMDT, where staff were immediately intrigued by AVA's potential to support current and future pandemic response efforts. "AVA's ability to recognize people's faces, check them for elevated temperatures and understand simple nodding and pointing gestures made it an ideal tool to screen people in high-traffic areas such as airports without putting a human screener at risk," says CAMDT Manager Saleh Jiddawi. "We also saw possible applications in a number of other industries."

In addition to managing the project and providing space and access to 3-D printing technology that supported prototype development, CAMDT worked closely with CloudConstable to prepare a project proposal and statement of work that helped secure $38,000 in external funding from FedDev Ontario's Southern Ontario Network for Advanced Manufacturing Innovation (SONAMI). CAMDT also assembled a team of professor Nilufar Poshtiban and student researchers Sugreev Kaur, Sanket Patel and Drew Pitchford, who optimized AVA by upgrading various hardware components and integrating them to function more harmoniously.

"The work done has allowed AVA to interact more seamlessly with the real world," Jiddawi says, "enabling AVA to communicate, take measurements and react to the most up-to-date information. It is akin to creating a highway between all the components, rather than having the communications take small roads."

Industrial designers created more elegant and refined prototype

Sheridan's Industrial Design program was initially invited to collaborate on the project by improving the external appearance of the existing prototype. However, professor and studio head Scot Laughton knew he and his students could bring a lot more than that to the table.

"Our team couldn't be involved on a project if it was simply tidying it up. There were so many elements that needed to be considered, including ergonomics, how it could be cleaned efficiently and how it could be protected from vandalism," says Laughton. "I brought our team together and said, "Let's grab the bull by the horns and create a concept that demonstrates what we believe this can become."

First, the team identified the types of environments that AVA would be used in and scanned the market to see what they could learn from the ways other companies resolved similar problems. Then the ideation phase began via visualization sketches on Padlet virtual boards, generating such innovations as a base that included an opening underneath to allow for AVA to be moved by forklift or pump truck; building the screen and camera into the same unit so it would be easier to maintain; ensuring AVA would fit through an average door frame; and making components detachable so AVA could be hung on a wall, moved around on wheels or remain in a stationary position. Laughton also performed an ergonomic study to determine optimal heights and viewing angles that would make engaging with AVA feel as natural as possible.

Once the designs were complete, the team built a low-fidelity model to present to Pickering, then spent two weeks producing a full-scale plywood prototype that housed all of AVA's components. Third-year student Joel Martin led construction of the new prototype, supported by Laughton, Pitchford and Industrial Design studio monitor Mickey Wang. "Mickey and Joel are both adept at working with electronic components, and Drew was our mechanical engineering expert that formed the third corner of the triangle," Laughton says. "There was a really strong relationship and great team dynamic."

Potential for future collaboration with other areas of Sheridan

The work by SIRT, CAMDT and Industrial Design concluded Phase 2 of CloudConstable's AVA project, but the company's collaboration with Sheridan might not be over yet.

In fact, some of Sheridan's other applied research centres and programs might be included in Phase 3 as CloudConstable looks to make AVA more user-friendly for people of all ages, more autonomous and more versatile in a post-pandemic world. Pickering envisions hospitals and long-term care facilities lightening their employees' workloads by getting AVA to perform basic tasks such as monitoring patient temperatures, and there could also be demand for virtual agents to support staff in tourism, leisure and other industries that thrive on human interaction. "We're really excited about the potential of continuing to work with Sheridan as we try to advance AVA to bigger and better things," he says.

"This is essentially what we do in applied research: work with start-ups and small-to-medium enterprises to de-risk innovation and bridge the gap between proof of concept and getting to market."

— Saleh Jiddawi, CAMDT Manager

The feeling is mutual. "It has been a great experience getting to interact and communicate with all of the different teams, do some project management and work with 3-D printers," Pitchford, a Mechanical Engineering Technology student, says of the chance to collaborate directly with industry to help resolve a real-world issue. Meanwhile, Laughton, who once headed a design firm in Hong Kong, embraced the opportunity to flex his own creative muscles and showcase what Sheridan's young Industrial Design program is capable of.

"Our partners get support from Sheridan's experienced faculty, students and access to funding, while our students get to collaborate with industry and attack challenges," Jiddawi says. "This is essentially what we do in applied research: work with start-ups and small-to-medium enterprises to de-risk innovation and bridge the gap between proof of concept and getting to market."

In first photo, CAMDT Student Research Assistant Drew Pitchford interacts with AVA (photo courtesy of CloudConstable). In second photo, various Sheridan staff, faculty and students stand around the AVA kiosk: (left to right) Saleh Jiddawi, CAMDT Manager; Dr. Nilufar Poshtiban, Professor; Sanket Patel, CAMDT Student Research Assistant; Mickey Wang, Industrial Design Studio Monitor; Scot Laughton, Industrial Design Studio Head and Professor; and Pitchford (photo courtesy of CloudConstable). In third photo, Wang and Patel assess the ideal height for the AVA prototype (photo provided by Scot Laughton).