Text-to-SQL agents have overtaken the market and opened a new kind of value creation. You can't argue that being able to talk with your data without coding wouldn't be amazing! Databricks AI/BI Genie, more familiarly known as just Genie, was introduced a while ago. By now probably everybody knows what Genie is, so no need for introduction. It’s your own internal data analyst agent, creating notable business value once harnessed correctly. But, usually the world isn’t that simple and the same applies here. Even though Genie made data usage a lot easier, there were still some caveats. Programmatic support (aka REST API endpoints) to create and update new Genie spaces was missing. The lack of these features meant a click-ops life, manual fine-tuning, not being able to automate processes or enforce company policies. This made Genie usage much more difficult, not to mention integrating it into company processes.
How to create a new Genie Space?
What data can be used and where can I find it?
Who could help me to update these new Genie instructions?
The biggest asset of Genie is making data exploration easy. However, if creating a Genie space isn’t a smooth or straightforward process, it doesn’t help much. Many have likely encountered similar challenges and wondered how to move from process bottlenecks to a scalable self-service reality. To create a Genie space, you need to know which data assets to use, add descriptions, business related instructions, business metrics and other context data to provide the necessary information. Genie, like all other agents, is like a new employee. You must provide proper context data, or it will start hallucinating and the quality will suffer quickly.
Now that the missing Create and Update REST API endpoints have finally been introduced (CRUD, or CUUD in this case) and Agent Bricks gaining momentum, I wanted to test how easy it would be to automate the entire Genie management process using Agent Bricks. The idea was to have a single user interface (AI playground) where you could ask an agent to create, use, update and delete Genie spaces. Meaning, removing the biggest bottlenecks I’ve personally faced: getting things started quickly and easily. To make things even more fun, the whole solution had to be built in one evening.
Time to plan the solution
The first step, like always, was to figure out the possibilities and limitations within the given development time period. I wanted to try something new this time and use Agent Bricks only (no custom agent code). Since the Multi-Agent Supervisor now supports both external MCP servers and Unity Catalog functions, it should be more than doable. Due to time limitations, I decided to use UC functions instead of building my own MCP server. Genie Spaces are also supported, but you have to choose them beforehand. I didn’t find a way to select all Genie Spaces, current and future (wink wink, Databricks), so I had to solve this dynamic usage as well. The solution would practically rely on Databricks REST API calls and system tables, which meant there were no need for overly fancy custom solutions. Creating UC functions is quick and handy, but there are some limitations to keep in mind. To provide all the necessary functionality I wanted, I split the actions into seven (7) UC functions, which I wrapped as tools. Here you can see the draft:
Solution architecture draft
These are the functionalities I want the supervisor agent to handle, including the ability to dynamically use Genie Spaces once they are created, without requiring manual updates to the Supervisor Agent:
1) Create new Genie spaces
Check which datasets are relevant for the requested Genie Space domain and return a list of table names from Unity Catalog
Add "company policy" descriptions to Genie to simulate enforced best-practice guidance
As a last step, create a new Genie space based using this information
2) Run Genie space
Fetch all usable Genie spaces (name and ID)
Post a Genie message (user content) to the selected Genie space
Return the Genie response
3) Update Genie space
Update the Genie space with new values
4) Delete Genie space
Delete the Genie space
Building the solution
I solely relied on UC functions to provide the necessary skills for the supervisor agent. From below you can see the functions used, along with short descriptions. On top of this, I created a basic UC connector to handle Databricks REST API calls. And yes, this was all the tool description engineering I did. Didn’t need to do any additional fine-tuning in this use case, which was a nice surprise. UC functions are quick to use and work well for trivial use cases (it’s even possible to use Python there), but for more demanding needs, they might not be enough.
Unity Catalog Functions
get_genies
* Retrieves all existing Genie Spaces user has access
find_relevant_data
* Searches for relevant datasets from Unity Catalog for Genie Space creation
create_genie
* Creates a new Genie Space
update_genie
* Updates an existing Genie Space to reflect new configurations
delete_genie
* Deletes an existing Genie Space
create_message
* Generates a new Genie message
get_message_output
* Fetches a previously created Genie message output with attachments
Yes, I know 😅 Got a bit lazy with the Genie usage UC functions. Give me some slack, this was just a quick PoC.Once the functions were done, I created a new Knowledge Assistant agent to simulate company policy enrichment automation. The idea here is that when Databricks users create new Genie spaces, company-specific information would be automatically added as instructions to Genie. There are so many ways to automate this, but the main idea is to ensure flawless information flow from documentation management solutions to Genie space (and Databricks), making end users lives a lot easier. To keep things really simple, I created dummy company policy data for the Knowledge Assistant. After that, it was just a matter of wrapping things up. Creating the Supervisor agent and adding these UC functions and the Knowledge Assistant as tools.
Let's see how it works
Creating a new Genie Space (fast-forwarded)
Using and updating newly created Genie Space (fast-forwarded)
Deleting Genie Space... or trying to (fast-forwarded)
Since this was just to prove the point that it can be done easily, I didn’t bother to do any proper fine-tuning at all. And to be honest, I was positively surprised by how quickly the supervisor agent was able to pick it up. Well, AI Playground visualization was a bit bugged here as you can see from the video and agent was quite slow to perform, but the quality matters the most here. The agent was able to create new Genie spaces based on my requests (finding the relevant data from Unity Catalog), use newly created Genie Space without need to update supervisor configs, update existing Genie spaces, and delete those as well. Well, almost. It seems that delete commands triggered automated guardrails and were blocked. Hmm I guess that’s fair enough.
To sum up, I now had one simple UI (AI Playground) to manage my Genie spaces, with the agent doing all the work for me.
Automating Genie management with Agent Bricks
Were Agent Bricks able to handle the job? Yes it was. I was able to quickly build a fun PoC that removes one of the biggest bottlenecks with Genie: managing Genie Spaces without the need for a data engineering or infrastructure team. As a matter, without any knowledge at all. Now that Unity Catalog functions and external MCPs are supported, the performance of the Agent Bricks Supervisor agent has improved significantly. This makes PoCs and simple automation solutions incredibly fast to build, which is amazing.
But would I use this in production? Honestly, no. For demanding use cases and production-grade solutions, I’d recommend building custom agents on Databricks. The observability and infrastructure are excellent and having full transparency and control over what’s happening is essential. With support for OpenAI Responses API now available (still waiting for async support..), agent infrastructure continues to become more and more powerful.
I’ve long preached Databricks as the promised land for agents and each day that vision becomes more of a reality. The closer you can host agents to the data, the easier it is manage the agent ecosystem. The combination of prebuilt agents and custom solutions is also extremely powerful. You can use prebuilt agents for non-business-critical automation, personal workflows, idea validation or low-ROI use cases. Then you have more time to allocate to build custom agents for business-critical and complex needs. Always keep ROI in mind. If the generated value isn’t high enough and worst-case scenario risks are low, it makes sense to use prebuilt agents, for example offered by Databricks.
This quick PoC was a fun way to demonstrate the idea. Now that the feasibility behind it has been validated, I’d allocate more resources to take things to the next level and make it truly production-ready. Some components would be replaced and moved behind custom agent code, making the solution as robust as possible for end users. They could then create, use and delete new Genie Spaces through the agent safely and effortlessly, naturally using their own authorizations to data & resources. Perhaps Databricks could offer this as an out-of-the-box solution someday 😉
Written by Aarni Sillanpää
Always remember ROI - especially with agents
