WordFinder app: Harnessing generative AI on AWS for aphasia communication

On this submit, we showcase how Dr. Kori Ramajoo, Dr. Sonia Brownsett, Prof. David Copland, from QARC, and Scott Harding, an individual residing with aphasia, used AWS companies to develop WordFinder, a cell, cloud-based resolution that helps people with aphasia improve their independence by the usage of AWS generative AI know-how.

Within the spirit of giving again to the neighborhood and harnessing the artwork of the attainable for constructive change, AWS hosted the Hack For Goal occasion in 2023. This hackathon introduced collectively groups from AWS prospects throughout Queensland, Australia, to sort out urgent challenges confronted by social good organizations.

The College of Queensland’s Queensland Aphasia Analysis Centre (QARC)’s mission is to enhance entry to know-how for individuals residing with aphasia, a communication incapacity that may affect a person’s skill to precise and perceive spoken and written language.

The problem: Overcoming communication boundaries

In 2023, it was estimated that greater than 140,000 individuals in Australia have been residing with aphasia. This quantity is predicted to develop to over 300,000 by 2050. Aphasia could make on a regular basis duties like on-line banking, utilizing social media, and attempting new gadgets difficult. The objective was to create a cell app that might help individuals with aphasia by producing a thesaurus of the objects which can be in a user-selected picture and lengthen the listing with associated phrases, enabling them to discover various communication strategies.

Overview of the answer

The next screenshot exhibits an instance of navigating the WordFinder app, together with register, picture choice, object definition, and associated phrases.

Within the previous diagram, the next situation unfolds: 

1. Register: The primary display screen exhibits a easy sign-in web page the place customers enter their electronic mail and password. It contains choices to create an account or recuperate a forgotten password.
2. Picture choice: After signing in, customers are prompted to Decide a picture to look. This display screen is initially clean.
3. Photograph entry: The following display screen exhibits a popup requesting non-public entry to the person’s photographs, with a grid of pattern pictures seen within the background.
4. Picture chosen: After a picture is chosen (on this case, an image of a koala), the app shows the picture together with some preliminary tags or classifications reminiscent of Animal, Bear, Mammal, Wildlife, and Koala.
5. Associated phrases: The ultimate display screen exhibits a listing of associated phrases primarily based on the choice of Associated Phrases subsequent to Koala from the earlier display screen. This step is essential for individuals with aphasia who usually have difficulties with word-finding and verbal expression. By exploring associated phrases (reminiscent of habitat phrases like tree and eucalyptus, or descriptive phrases like fur and marsupial), customers can bridge communication gaps when the precise phrase they need isn’t instantly accessible. This semantic community strategy aligns with widespread aphasia remedy strategies, serving to customers discover alternative routes to precise their ideas when particular phrases are tough to recall.

This circulate demonstrates how customers can use the app to seek for phrases and ideas by beginning with a picture, then drilling down into associated terminology—a visible strategy to increasing vocabulary or discovering related phrases.

The next diagram illustrates the answer structure on AWS.

Within the following sections, we focus on the circulate and key elements of the answer in additional element.

1. Safe entry utilizing Route 53 and Amplify 
   1. The journey begins with the person accessing the WordFinder app by a site managed by Amazon Route 53, a extremely obtainable and scalable cloud DNS net service. AWS Amplify hosts the React Native frontend, offering a seamless cross-environment expertise. 
2. Safe authentication with Amazon Cognito 
   1. Earlier than accessing the core options, the person should securely authenticate by Amazon Cognito. Cognito offers strong person identification administration and entry management, ensuring that solely authenticated customers can work together with the app’s companies and sources. 
3. Picture seize and storage with Amplify and Amazon S3 
   1. After being authenticated, the person can seize a picture of a scene, merchandise, or situation they want to recall phrases from. AWS Amplify streamlines the method by mechanically storing the captured picture in an Amazon Easy Storage Service (Amazon S3) bucket, a extremely obtainable, cost-effective, and scalable object storage service. 
4. Object recognition with Amazon Rekognition 
   1. As quickly because the picture is saved within the S3 bucket, Amazon Rekognition, a robust laptop imaginative and prescient and machine studying service, is triggered. Amazon Rekognition analyzes the picture, figuring out objects current and returning labels with confidence scores. These labels type the preliminary phrase immediate listing throughout the WordFinder app, kickstarting the word-finding journey. 
5. Semantic phrase associations with API Gateway and Lambda 
   1. Whereas the preliminary thesaurus generated by Amazon Rekognition offers a stable place to begin, the person is perhaps in search of a extra particular or associated phrase. To deal with this problem, the WordFinder app sends the preliminary thesaurus to an AWS Lambda perform by Amazon API Gateway, a totally managed service that securely handles API requests. 
6. Lambda with Amazon Bedrock, and generative AI and immediate engineering utilizing Amazon Bedrock
   1. The Lambda perform, appearing as an middleman, crafts a fastidiously designed immediate and submits it to Amazon Bedrock, a totally managed service that provides entry to high-performing basis fashions (FMs) from main AI corporations, together with Anthropic’s Claude mannequin.
   2. Amazon Bedrock generative AI capabilities, powered by Anthropic’s Claude mannequin, use superior language understanding and technology to supply semantically associated phrases and ideas primarily based on the preliminary thesaurus. This course of is pushed by immediate engineering, the place fastidiously crafted prompts information the generative AI mannequin to supply related and contextually acceptable phrase associations.

WordFinder app part particulars

On this part, we take a better have a look at the elements of the WordFinder app.

React Native and Expo

WordFinder was constructed utilizing React Native, a well-liked framework for constructing cross-environment cell apps. To streamline the event course of, Expo was used, which permits for write-once, run-anywhere capabilities throughout Android and iOS working methods.

Amplify

Amplify performed a vital function in accelerating the app’s improvement and provisioning the mandatory backend infrastructure. Amplify is a set of instruments and companies that allow builders to construct and deploy safe, scalable, and full stack apps. On this structure, the frontend of the phrase discovering app is hosted on Amplify. The answer makes use of a number of Amplify elements:

• Authentication and entry management: Amazon Cognito is used for person authentication, enabling customers to enroll and register to the app. Amazon Cognito offers person identification administration and entry management with entry to an Amazon S3 bucket and an API gateway requiring authenticated person classes.
• Storage: Amplify was used to create and deploy an S3 bucket for storage. A key part of this app is the flexibility for a person to take an image of a scene, merchandise, or situation that they’re in search of to recall phrases from. The answer must briefly retailer this picture for processing and evaluation. When a person uploads a picture, it’s saved in an S3 bucket for processing with Amazon Rekognition. Amazon S3 offers extremely obtainable, cost-effective, and scalable object storage.
• Picture recognition: Amazon Rekognition makes use of laptop imaginative and prescient and machine studying to determine objects current within the picture and return labels with confidence scores. These labels are used because the preliminary phrase immediate listing throughout the WordFinder app.

Associated phrases

The generated preliminary thesaurus is step one towards discovering the specified phrase, however the labels returned by Amazon Rekognition may not be the precise phrase that somebody is searching for. The undertaking staff then thought of how one can implement a thesaurus-style lookup functionality. Though the undertaking staff initially explored completely different programming libraries, they discovered this strategy to be considerably inflexible and restricted, usually returning solely synonyms and never entities which can be associated to the supply phrase. The libraries additionally added overhead related to packaging and sustaining the library and dataset shifting ahead.

To deal with these challenges and enhance responses for associated entities, the undertaking staff turned to the capabilities of generative AI. By utilizing the generative AI basis fashions (FMs), the undertaking staff was capable of offload the continued overhead of managing this resolution whereas rising the flexibleness and curation of associated phrases and entities which can be returned to customers. The undertaking staff built-in this functionality utilizing the next companies:

• Amazon Bedrock: Amazon Bedrock is a totally managed service that provides a selection of high-performing FMs from main AI corporations like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon by a single API, together with a broad set of capabilities to construct generative AI apps with safety, privateness, and accountable AI. The undertaking staff was capable of rapidly combine with, take a look at, and consider completely different FMs, lastly settling upon Anthropic’s Claude mannequin.
• API Gateway: The undertaking staff prolonged the Amplify undertaking and deployed API Gateway to simply accept safe, encrypted, and authenticated requests from the WordFinder cell app and move them to a Lambda perform dealing with Amazon Bedrock entry. 
• Lambda: A Lambda perform was deployed behind the API gateway to deal with incoming net requests from the cell app. This perform was accountable for taking the equipped enter, constructing the immediate, and submitting it to Amazon Bedrock. This meant that integration and immediate logic could possibly be encapsulated in a single Lambda perform.

Advantages of API Gateway and Lambda

The undertaking staff briefly thought of utilizing the AWS SDK for JavaScript v3 and credentials sourced from Amazon Cognito to instantly interface with Amazon Bedrock. Though this could work, there have been a number of advantages related to implementing API Gateway and a Lambda perform:

• Safety: To allow the cell shopper to combine instantly with Amazon Bedrock, authenticated customers and their related AWS Id and Entry Administration (IAM) function would have to be granted permissions to invoke the FMs in Amazon Bedrock. This could possibly be achieved utilizing Amazon Cognito and short-term permissions granted by roles. Consideration was given to the potential of uncontrolled entry to those fashions if the cell app was compromised. By shifting the IAM permissions and invocation dealing with to a central perform, the staff was capable of improve visibility and management over how and when the FMs have been invoked.
• Change administration: Over time, the underlying FM or immediate may want to vary. If both was onerous coded into the cell app, any change would require a brand new launch and each person must obtain the brand new app model. By finding this throughout the Lambda perform, the specifics round mannequin utilization and immediate creation are decoupled and may be tailored with out impacting customers. 
• Monitoring: By routing requests by API Gateway and Lambda, the staff can log and monitor metrics related to utilization. This allows higher decision-making and reporting on how the app is performing. 
• Information optimization: By implementing the REST API and encapsulating the immediate and integration logic throughout the Lambda perform, the staff to can ship the supply phrase from the cell app to the API. This implies much less knowledge is shipped over the mobile community to the backend companies. 
• Caching layer: Though a caching layer wasn’t applied throughout the system throughout the hackathon, the staff thought of the flexibility to implement a caching mechanism for supply and associated phrases that over time would scale back requests that have to be routed to Amazon Bedrock. This may be readily queried within the Lambda perform as a preliminary step earlier than submitting a immediate to an FM.

Immediate engineering

One of many core options of WordFinder is its skill to generate associated phrases and ideas primarily based on a user-provided supply phrase. This supply phrase (obtained from the cell app by an API request) is embedded into the next immediate by the Lambda perform, changing {phrase}:

immediate = "I've Aphasia. Give me the highest 10 most typical phrases which can be associated phrases to the phrase equipped within the immediate context. Your response must be a sound JSON array of simply the phrases. No surrounding context. {phrase}"

The staff examined a number of completely different prompts and approaches throughout the hackathon, however this fundamental guiding immediate was discovered to provide dependable, correct, and repeatable outcomes, whatever the phrase equipped by the person.

After the mannequin responds, the Lambda perform bundles the associated phrases and returns them to the cell app. Upon receipt of this knowledge, the WordFinder app updates and shows the brand new listing of phrases for the person who has aphasia. The person may then discover their phrase, or drill deeper into different associated phrases.

To keep up environment friendly useful resource utilization and price optimization, the structure incorporates a number of useful resource cleanup mechanisms:

• Lambda computerized scaling: The Lambda perform accountable for interacting with Amazon Bedrock is configured to mechanically scale all the way down to zero situations when not in use, minimizing idle useful resource consumption.
• Amazon S3 lifecycle insurance policies: The S3 bucket storing the user-uploaded pictures is configured with lifecycle insurance policies to mechanically expire and delete objects after a specified retention interval, releasing up space for storing. 
• API Gateway throttling and caching: API Gateway is configured with throttling limits to assist forestall extreme requests, and caching mechanisms are applied to scale back the load on downstream companies reminiscent of Lambda and Amazon Bedrock.

Conclusion

The QARC staff and Scott Harding labored intently with AWS to develop WordFinder, a cell app that addresses communication challenges confronted by people residing with aphasia. Their successful entry on the 2023 AWS Queensland Hackathon showcased the ability of involving these with lived experiences within the improvement course of. Harding’s insights helped the tech staff perceive the nuances and affect of aphasia, resulting in an answer that empowers customers to seek out their phrases and keep related.

References

In regards to the Authors

Kori Ramijoo is a analysis speech pathologist at QARC. She has intensive expertise in aphasia rehabilitation, know-how, and neuroscience. Kori leads the Aphasia Tech Hub at QARC, enabling individuals with aphasia to entry know-how. She offers consultations to clinicians and offers recommendation and assist to assist individuals with aphasia achieve and keep independence. Kori can also be researching design issues for know-how improvement and use by individuals with aphasia.

Scott Harding lives with aphasia after a stroke. He has a background in Engineering and Laptop Science. Scott is without doubt one of the Administrators of the Australian Aphasia Affiliation and is a client consultant and advisor on varied state authorities well being committees and nationally funded analysis initiatives. He has pursuits in the usage of AI in creating predictive fashions of aphasia restoration.

Sonia Brownsett is a speech pathologist with intensive expertise in neuroscience and know-how. She has been a postdoctoral researcher at QARC and led the aphasia tech hub in addition to a analysis program on the mind mechanisms underpinning aphasia restoration after stroke and in different populations together with adults with mind tumours and epilepsy.

David Copland is a speech pathologist and Director of QARC. He has labored for over 20 years within the area of aphasia rehabilitation. His work seeks to develop new methods to know, assess and deal with aphasia together with the usage of mind imaging and know-how. He has led the creation of complete aphasia therapy packages which can be being applied into well being companies.

Mark Promnitz is a Senior Options Architect at Amazon Internet Providers, primarily based in Australia. Along with serving to his enterprise prospects leverage the capabilities of AWS, he can usually be discovered speaking about Software program as a Service (SaaS), knowledge and cloud-native architectures on AWS.

Kurt Sterzl is a Senior Options Architect at Amazon Internet Providers, primarily based in Australia.  He enjoys working with public sector prospects like UQ QARC to assist their analysis breakthroughs.