Discover ways to use Google’s Germini-1.5-pro-latest mannequin to develop a generative AI app for calorie counting
Have you ever ever questioned the quantity of energy you devour whenever you eat your dinner, for instance? I do this on a regular basis. Wouldn’t it’s great in case you may merely move an image of your plate by an app and get an estimate of the full variety of energy earlier than you resolve how far in you need to dip?
This calorie counter app that I created will help you obtain this. It’s a Python utility that makes use of Google’s Gemini-1.5-Professional-Newest mannequin to estimate the variety of energy in meals gadgets.
The app takes two inputs: a query in regards to the meals and a picture of the meals or meals gadgets, or just, a plate of meals. It outputs a solution to the query, the full variety of energy within the picture and a breakdown of energy by every meals merchandise within the picture.
On this article, I’ll clarify the complete end-to-end means of constructing the app from scratch, utilizing Google’s Gemini-1.5-pro-latest (a Massive Language generative AI mannequin launched by Google), and the way I developed the front-end of the applying utilizing Streamlit.
It’s value noting right here that with developments on the planet of AI, it’s incumbent on knowledge scientists to regularly shift from conventional deep studying to generative AI strategies in an effort to revolutionize their position. That is my principal goal of training on this topic.
Let me begin by briefly explaining Gemini-1.5-pro-latest and the streamlit framework, as they’re the most important parts within the infrastructure of this calorie counter app.
Gemini-1.5-pro-latest is a complicated AI language mannequin developed by Google. Since it’s the newest model, it has enhanced capabilities over earlier variations within the mild of quicker response instances and improved accuracy when utilized in pure language processing and constructing purposes.
This can be a multi-modal mannequin that works with each texts and pictures — an development from Google Gemini-pro mannequin which solely works with textual content prompts.
The mannequin works by understanding and producing textual content, like people, based mostly on prompts given to it. On this article, this mannequin can be used to to generate textual content for our energy counter app.
Gemini-1.5-pro-latest may be built-in into different purposes to bolster their AI capabilities. On this present utility, the mannequin makes use of generative AI strategies to interrupt the uploaded picture into particular person meals gadgets . Based mostly on its contextual understanding of the meals gadgets from its dietary database, it makes use of picture recognition and object detection to estimate the variety of energy, after which totals up the energy for all gadgets within the picture.
Streamlit is an open-source Python framework that may handle the consumer interface. This framework simplifies net improvement in order that all through the undertaking, you do not want to write down any HTML and CSS codes for the entrance finish.
Allow us to dive into constructing the app.
I’ll present you the way to construct the app in 5 clear steps.
1. Arrange your Folder construction
For a begin, go into your favourite code editor (mine is VS Code) and begin a undertaking file. Name it Energy-Counter, for instance. That is the present working listing. Create a digital atmosphere (venv), activate it in your terminal, after which create the next information: .env, energy.py, necessities.txt.
Right here’s a advice for the look of your folder construction:
Energy-Counter/
├── venv/
│ ├── xxx
│ ├── xxx
├── .env
├── energy.py
└── necessities.txt
Please notice that Gemini-1.5-Professional works finest with Python variations 3.9 and higher.
2. Get the Google API key
Like different Gemini fashions, Gemini-1.5-pro-latest is at present free for public use. Accessing it requires that you simply receive an API key, which you will get from Google AI Studio by going to “Get API key” on this hyperlink. As soon as the bottom line is generated, copy it for subsequent use in your code. Save this key as an atmosphere variable within the .env file as follows.
GOOGLE_API_KEY="paste the generated key right here"
3. Set up dependencies
Sort the next libraries into your necessities.txt file.
- streamlit
- google-generativeai
- python-dotenv
Within the terminal, set up the libraries in necessities.txt with:
python -m pip set up -r necessities.txt
4. Write the Python script
Now, let’s begin writing the Python script in energy.py. With the next code, import all required libraries:
# import the libraries
from dotenv import load_dotenv
import streamlit as st
import os
import google.generativeai as genai
from PIL import Picture
Right here’s how the assorted modules imported can be used:
- dotenv — Since this utility can be configured from a Google API key atmosphere variable, dotenv is used to load configuration from the .env file.
- Streamlit — to create an interactive consumer interface for front-end
- os module is used to deal with the present working listing whereas performing file operations like getting the API key from the .env file
- google.generativeai module, in fact, provides us entry to the Gemini mannequin we’re about to make use of.
- PIL is a Python imaging library used for managing picture file codecs.
The next traces will configure the API keys and cargo them from the atmosphere variables retailer.
genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))load_dotenv()
Outline a perform that, when known as, will load the Gemini-1.5-pro-latest and get the response, as follows:
def get_gemini_reponse(input_prompt,picture,user_prompt):
mannequin=genai.GenerativeModel('gemini-1.5-pro-latest')
response=mannequin.generate_content([input_prompt,image[0],user_prompt])
return response.textual content
Within the above perform, you see that it takes as enter, the enter immediate that can be specified additional down within the script, a picture that can be equipped by the consumer, and a consumer immediate/query that can be equipped by the consumer. All that goes into the gemini mannequin to return the response textual content.
Since Gemini-1.5-pro expects enter photographs within the type of byte arrays, the subsequent factor to do is write a perform that processes the uploaded picture, changing it to bytes.
def input_image_setup(uploaded_file):
# Examine if a file has been uploaded
if uploaded_file shouldn't be None:
# Learn the file into bytes
bytes_data = uploaded_file.getvalue()image_parts = [
{
"mime_type": uploaded_file.type, # Get the mime type of the uploaded file
"data": bytes_data
}
]
return image_parts
else:
increase FileNotFoundError("No file uploaded")
Subsequent, specify the enter immediate that may decide the behaviour of your app. Right here, we’re merely telling Gemini what to do with the textual content and picture that the app can be fed with by the consumer.
input_prompt="""
You're an knowledgeable nutritionist.
You need to reply the query entered by the consumer within the enter based mostly on the uploaded picture you see.
You also needs to have a look at the meals gadgets discovered within the uploaded picture and calculate the full energy.
Additionally, present the main points of each meals merchandise with energy consumption within the format beneath:1. Merchandise 1 - no of energy
2. Merchandise 2 - no of energy
----
----
"""
The following step is to initialize streamlit and create a easy consumer interface on your calorie counter app.
st.set_page_config(page_title="Gemini Calorie Counter App")
st.header("Calorie Counter App")
enter=st.text_input("Ask any query associated to your meals: ",key="enter")
uploaded_file = st.file_uploader("Add a picture of your meals", sort=["jpg", "jpeg", "png"])
picture=""
if uploaded_file shouldn't be None:
picture = Picture.open(uploaded_file)
st.picture(picture, caption="Uploaded Picture.", use_column_width=True) #present the picturesubmit=st.button("Submit & Course of") #creates a "Submit & Course of" button
The above steps have all of the items of the app. At this level, the consumer is ready to open the app, enter a query and add a picture.
Lastly, let’s put all of the items collectively such that after the “Submit & Course of” button is clicked, the consumer will get the required response textual content.
# As soon as submit&Course of button is clicked
if submit:
image_data=input_image_setup(uploaded_file)
response=get_gemini_reponse(input_prompt,image_data,enter)
st.subheader("The Response is")
st.write(response)
5. Run the script and work together along with your app
Now that the app improvement is full, you’ll be able to execute it within the terminal utilizing the command:
streamlit run energy.py
To work together along with your app and see the way it performs, view your Streamlit app in your browser utilizing the native url or community URL generated.
This how your Streamlit app seems to be like when it’s first opened on the browser.
As soon as the consumer asks a query and uploads a picture, right here is the show:
As soon as the consumer pushes the “Submit & Course of” button, the response within the picture beneath is generated on the backside of the display screen.
For exterior entry, contemplate deploying your app utilizing cloud companies like AWS, Heroku, Streamlit Neighborhood Cloud. On this case, let’s use Streamlit Neighborhood Cloud to deploy the app without cost.
On the highest proper of the app display screen, click on ‘Deploy’ and comply with the prompts to finish the deployment.
After deployment, you’ll be able to share the generated app URL to different customers.
Identical to different AI purposes, the outcomes outputed are the perfect estimates of the mannequin, so, earlier than utterly counting on the app, please notice the next as a few of the potential dangers:
- The calorie counter app might misclassify sure meals gadgets and thus, give the mistaken variety of energy.
- The app doesn’t have a reference level to estimate the scale of the meals — portion — based mostly on the uploaded picture. This may result in errors.
- Over-reliance on the app can result in stress and psychological well being points as one might change into obsessive about counting energy and worrying about outcomes that will not be too correct.
To assist scale back the dangers that include utilizing the calorie counter, listed below are potential enhancements that could possibly be built-in into its improvement:
- Including contextual evaluation of the picture, which can assist to gauge the scale of the meals portion being analysed. For example, the app could possibly be constructed such that a normal object like a spoon, included within the meals picture, could possibly be used as a reference level for measuring the sizes of the meals gadgets. This may scale back errors in ensuing complete energy.
- Google may enhance the variety of the meals gadgets of their coaching set to scale back misclassification errors. They may develop it to incorporate meals from extra cultures in order that even uncommon African meals gadgets can be recognized.
