The more organisations expand, the more internal data there is. Workers could be drowning in data. This causes a lack of clear communication, and it is not getting to all employees; it’s harder to find the useful information.

Workers today need information that is portable and easy to read. The traditional repositories (such as wikis at companies) are not always up to the task of delivering on these new demands. This disconnect means handling a mix of documents from different office suites, such as Google’s and Microsoft’s, and also being on top of posts on other sites.

One solution turns out to be LLM chatbots. You might have thought about building an LLM chatbot, but you also might have thought about how complicated it is to build and tweak.
We went this rabbit hole and realised that it is really not too hard when you are using the right technology with the right approach.

We went this rabbit hole and realised that it is really not too hard when you are using the right technology with the right approach.

Let’s get to know the story of how we started developing, what strategic factors were at play, and what came out of it.

The various ways of building an LLM chatbot, but we’ll explain to you how we’ve built one that is easily accessible for most organisations.

The end result is that you can easily, quickly, and cheaply develop your LLM chatbot to drive real outcomes.

What is an LLM chatbot?

LLM chatbots are AI implementations that interpret and produce human language as humans speak it.

They are educated in vast amounts of text data, and then they can go out and do various language-related tasks like answering questions, giving suggestions, and modelling conversation.

LLM chatbots talk more subtle and difficult than previous chatbots. They can make sense of context, cope with ambiguity, and even have some creative agency about their reactions.

Simply put, an LLM chatbot doesn’t simply combine data from many different places. It is the door to a chatbot-based conversational user interface for easier access to data naturally and fast.

The reasons to use an LLM chatbot

They will not be updated on company updates and news. This communication gap not only slows down information flow but also saps the trust of employees and the culture of your organisation.

This information glut can be found in any company. Small businesses in particular find it hard to keep their employees up-to-date on day-to-day business.

1. Slack for immediate communication
2. Google Workspace: Share documents and emails with others in Google Workspace.
3. Their own website to post public data.
4. A company Coda wiki for company internal knowledge sharing.

The same kind of organisation and company is very typical. But lucid communication and information sharing are hard with information so dispersed. In time, as the wiki grew, locating relevant information became more like a needle in a haystack.

We understood that there were barriers to data collection and exchange across the numbskull of tools and platforms. So, that’s why we looked at how LLMs can be used to automate this.

Objective and scope of LLM chatbot implementation

Because we spent a majority of our time in Slack, it was clear that an LLM chatbot for Slack was the strategic route to take. This bot would respond to the queries of our staff and provide recommendations from all sources we have kept in house.

This move would be more available and efficient in our internal messaging.

Scope of data involved

We needed a mockup to test our LLM chatbot. Since we learnt the hard way that it’s always better to do it iteratively, we deliberately indexed only company employees’ documents. So we started with our wiki Coda pages.

This way we would be able to keep track of the bot’s precision and reliability after it has been initially installed. Judicious care would also guarantee that private data for the discretion of a few people would not be compromised.

What are the steps to build an LLM chatbot?

There are a multitude of technologies to construct LLM chatbots. You may want to try out third-party products or create something more personal from the ground up.

On a strict timeframe, budget, and team, we implemented RAG. It was about standard solutions, in particular choosing general-purpose LLMs without fine-tuning. That way we could get the benefits we wanted.

• Expanding the slew of know-how to support subsequent versions
• Source citations
• A reduction in hallucinations

Since we aimed to deliver the best LLM chatbot for our team members, we evaluated both in-house solutions and commercial ones to see which was best suited for us.

Loaded Stack for LLM chatbot

We are proponents of open-source software, and because we worked with that technology before, we decided on the following stack.

1. LangChain
2. Faiss
3. Stable Beluga 7B

We wanted to use the customisability and the ability to customise the chatbot’s behaviour for particular needs.

Trade stack for LLM chatbot

We started with Google’s Vertex solutions; we went after two solutions.

1. Google Enterprise Search
2. Google DialogFlow

We still wanted to be configurable and have the freedom to add other sources.

The selection process of technology

In selecting the right technology for our use cases, we considered the following points:

1. Quality of responses
2. Customisability and flexibility
3. Data handling
4. Scalability
5. Expenses
6. Simple implementation process
7. Minimum period to create a functional solution.

We collected common queries by key functions marketing, hiring, people, onboarding, office, security, finances, travel in order to determine which technology was best for us.

Start with questions such as “Who is CEO? or “How do I book a parking spot?” So, we put together 100 questions. This list provided us with an idea of whether the technology we’re thinking about is correct or not.

Decision regarding the LLM chatbot technology: Final say.

We rolled back after running the test questions as the custom stack returned poor answers and did not allow us to edit employees’ names. This may confuse and lower the trust in the chatbot’s responses.

When deciding on commercial technologies, Google Enterprise Search was the first option that came to mind because of the fact that it had incredible search power and answers from the get-go. For its features, we decided to go with Google DialogFlow. Its answers were no worse than those of Google Enterprise Search, with the added bonus of configuration flexibility.

But DialogFlow also has its own advantages, such as being able to ingest three different kinds of data: a set of documents, a website, and a custom FAQ document, which makes DialogFlow a more flexible option for us.

We conceded the trade-off between personalisation and output quality. Google DialogFlow gave us agility, data management, and the precision of its language model for our small project.

But this decision also enables future modifications, so that as the requirements of our project change, our chatbot solution can.

The configuration of Google DialogFlow

We followed the philosophy of uncomplicatedness. For that, we were using a simple structure in DialogFlow one “page”—and not counting conversations past as context.

This will ensure every query is handled in a different way with an objective and straightforward answer to the questions and create a better experience for the user.

During the development, we were also mindful of grounding the chatbot’s persistence in keeping with data in response. So, after trying out different heights, we decided on low grounding.

This was a crucial decision because the LLM chatbot didn’t want to give hallucinatory or irrelevant responses. By selecting low grounding, we made it more likely that our chatbot could still be versatile and answer many different kinds of questions without giving up and defaulting to rejection.

It was also a decision about which language model to use. We decided to use Gemini 1.0 Pro, which is a machine that offers great linguistic features but is cheap.

That was the preference given the expectation of low question numbers. Gemini 1.0 Pro makes it possible for our chatbot to learn and answer questions with sophisticated sophistication and, as such, is a key part of our solution’s success.

Slick operation and protection of the LLM chatbot

And we had to make sure everything went smoothly before releasing our LLM chatbot.

Our initial integration with Slack was very simple, and it even allowed us to create a prototype in less than one day. We setup production and development environments so that we can tweak our LLM chatbot without disrupting operations and experiment with change quickly.

But some agent settings were identical in all environments, which made it hard to test. Either we risked the ruin of the environment or we cloned the whole system.

We wanted as minimalist a setup as possible that would allow us to pass through with the minimum chance of disruption and without copying the setup.

Expense regulation of LLM chatbots

A big issue was cost control, with the risk of lots of tests from our engineers. We wanted to limit traffic so we could avoid the possibilities of integration failures causing a large number of requests or suddenly the service starts getting popular.

So, we put in a “circuit breaker.” It would disconnect the Slack integration automatically when a specified amount of expenses was hit.

Data quality considerations

For the bot to work, data had to be refreshed and the right documents had to be indexed. We have a list of all Google documents to index, which is in an editable spreadsheet.

We shared all the Google documents correctly with our service account and indexed all the Coda pages and documents on the list. This gave you explicit control so you wouldn’t accidentally disclose personal information.

Coda pages and documents are parsed to save some time, so we stripped them of HTML tags and attributes that were unnecessary by custom reformatting them. The parent page names have been added to the document names so we can retain some of the hierarchy of the original documents. We’ve also tried grouping some pages together, but these tests haven’t proven much better.

Possibly someday we can read documents and presentations with their content.

Response quality

We wanted simple answers, but we also had to define the expectations for the LLM chatbot users.

This is something we mentioned in our release notes and have named the bot “beta” so people can handle expectations. And we’ve included links to the original documents the answers came from.

To start with, we wanted to embed the original documents on Coda or Google Workspace. But our LLM chatbot is based on copies of source files in a Google Cloud Platform (GCP) bucket.

When we first tried to just compel LLM generators to make these URL translations successfully, we went for a simple webhook. This webhook reverted the GCP bucket links back to their originals with an elaborate naming scheme.

Our LLM chatbot release process

We used a sequenced release chain to test the LLM chatbot’s efficacy and reliability once we had established a PoC. We had the bot tested to the bone before it went live and it was proven to answer most questions correctly.

Some were not 100% right, but some offered good links, not direct answers. But the LLM chatbot acted like one thing and then another when it wasn’t able to interpret a question. This was enough to move forward with the release.

The release path was this.

• Leader release: We made sure that nothing that was important, e.g., could spill confidential information, was there.
• Non-engineering team release: Since this team processed most of the information, by using these team’s answers, we were reducing the chances of misinformation or damage.
• Public out of the company: We wanted to see if the tool is a useful tool and not make it wrong.

Observations after the release

We noticed a normal uptick in interest once the bot was published. Because a good number of our internal users are software engineers, they really tested the bot with queries.

Exploratory Questions: Users posed the bot with experiments of trying to change its future responses by using certain prompting strategies and asked questions about its weakness indicating that the users were interested in the bot’s flexibility and ability to think for itself.

Data Accuracy and Accessibility Issues: Users wanted data that was more recent than our internal documentation. The queries also had data from the internet which we are not indexing so the bot needed a larger and current information collection.

Analysis/Structurally based Questions: Participants inquired about the organizational structure not described on the wiki and asked for deep analyses where data is required to be summarised from several documents. These questions call for granular knowledge and the bot’s navigation and integration of granular organisational data.
Language Flexibility: Questions were asked to answer in languages the bot is not designed to do, which shows just how vital it is for an LLM to answer in languages other than English for a global audience.

We discovered some real holes in our records, based on the queries. But first we’d better build up the data collection — we anticipated this before continuing to expand.

LLM chatbot’s optimization

We’ve been monitoring usage very closely since release, and it’s come down to a very small amount. We logged into the cloud logs and DialogFlow conversation logs and analysed them.

So, we’ve created a separate notebook for query tracking – more to find the typical questions and fill in the missing pieces. The release notes (an aside) were also explicit that we wanted to use data for this.

DialogFlow provides upvoting or downvoting as user feedback. But because we are using Slack integration, we’ve added a link for people to leave comments on it through a Google form, so we can easily collect and report on user feedback.

Based on this feedback and review, we’ll continue improving the LLM chatbot after our data source improvement.

The possible solutions are to change the agent summarization prompt and to give priority to some of the data sources.

We could even try to do some more advanced tweaks like a custom solution to have more embedding control, vector database, custom prompts, and choosing among LLM models.

But such radical innovations were out of this first proof-of-concept.

Conclusion

This internal LLM chatbot was built in an efficient way — the development and use of an early version. We also discovered that even when we don’t have our data, it is still good information. The project also revealed what user behaviour the bot needed to satisfy and the large spectrum of questions the bot needed to cover.

We think of it today as a super search engine, but in time we expect that LLM will better pull in multiple sources of data. This could in part be a matter of longer requests or model refinement, and making an agent apply proper tools below.