Deploy a Gradio chatbot with an LLM from SAP AI Core’s Gen AI Hub on Cloud Foundry in SAP BTP

I am Aditi and I recently started working at Bluestonex as an SAP BTP Consultant. I wanted to explore how to deploy a Gen AI chatbot on the Cloud Foundry runtime environment, specifically focusing on the possibility of avoiding Docker image builds, which is a requirement in other environments like Kyma. To achieve this, I integrated a large language model (LLM) for interactive conversations, utilizing the SAP AI Core Generative AI Hub for hosting the AI component. This approach allowed me to leverage Cloud Foundry’s platform capabilities while seamlessly incorporating advanced AI functionalities without the added complexity of containerization. 

In this blog post, we will walk through the process of building a simple chatbot using Gradio, a Python library that allows you to create easy-to-use web-based interfaces. We’ll then deploy this chatbot to a Cloud Foundry environment, making it accessible as a web application. 

Technical Architecture 

The above outlines the high level architecture for a user interaction with the chat interface built using gradio and deployed on cloud foundry runtime environment. This chat interface uses the Large Language Model hosted on the SAP AI Core on the Generative AI Hub. 

Demo 

Setting up the development environment 

The following software packages are required to get you started. 

• Python 3.10+ 

• pip (Python package installer) 

• Cloud Foundry CLI (Command Line Interface) 

• SAP AI Core ( Create an env.json file to specify all Client Credentials and other environment specific information used) 

• Already deployed LLM (Azure OpenAI GPT-4 32K in this use case) 

• Gradio and other dependencies (Create a requirements.txt file for these) 

For detailed source code and deployment files feel free to Comment on the blog post and we would be happy to follow up. 

Gradio will launch a web interface where you can interact with the chatbot. The URL provided in the terminal (or in the browser) allows you to access the chatbot interface locally. In the next section we will talk about deployment on cloud foundry runtime environment 

Preparing for Deployment on Cloud Foundry 

To deploy the chatbot on Cloud Foundry, we need to create a manifest.yaml file. There are several tutorials which outline how to create a manifest file.
A python_buildpack is added so that there is no need to create any explicit docker image for the application deployment to cloud foundry. 

Deploying to Cloud Foundry 

First, make sure you are logged into your Cloud Foundry account. Navigate to your organization and space (if applicable), then push your application:
cf push gradio-chatbot 

Cloud Foundry will detect the Python buildpack, install the dependencies from requirements.txt, and deploy your application. The deployment may take a few minutes. 

Accessing Your Chatbot 

Once the deployment process completes, Cloud Foundry will provide a URL where your chatbot is hosted. You can access the chatbot by navigating to this URL in your web browser. 

Troubleshooting Tips 

• Dependencies Issues: Ensure all necessary Python packages are included in the requirements.txt. 

• Memory Limits: If your app crashes due to memory constraints, try increasing the memory allocation in the manifest.yml (create one if needed). 

• Environment Variables: If you need environment variables, define them in a manifest.yml or use cf set-env. 

Deploying web applications on Cloud Foundry allows you to scale your apps with minimal effort, making it a powerful platform for developers working with Python and Gradio. 

Conclusion 

In this blog post, we’ve created a simple Gen AI chatbot using Gradio and deployed it on Cloud Foundry using an LLM hosted on the Gen AI Hub on the SAP AI Core. This is just a starting point, and you can expand the chatbot’s capabilities by integrating it with more advanced natural language processing (NLP) models, databases, or other APIs.