Exploring offline LLMs and Tooling

I'm a late bloomer in the context of the AI hype train. I have been fairly resistant on falling for the marketing campaigns that's shoved down everyone's throat, from all directions. As a firm believer of having to master one's trait, I kept my head down and avoided distraction.

A few months ago I decided to gear up and cross the chasm by ordering two "For Dummies" titles: Machine Learning for Dummies and AI for Dummies. There is too much behind "AI" to just jump in and take it at face value, so taking action to understand it at a fundamental level is important. For now the "AI for Dummies" title is gathering dust while I work through "Machine Learning for Dummies". It's a heavy read with mathematics and statistics concepts which has to be brushed up on along the way. Fortunately the "For Dummies" range is good at making challenging concepts easier to understand.

In a parallel timeline I also started exploring actual tools and because I work with AWS on a daily basis, I naturally gravitated towards trying Amazon Q. It quickly became clear that it's a double-edged sword: Yes, it can be cool and do some things well, but it can also make you spend more time to verify and confirm whether the result is acceptable. It doesn't always give you something that is correct. But the trick is to not depend on it as a crutch, but to use it to learn and assist - like a smart enough companion. "Smart enough" because it's not always right, but it may save you some time if used appropriately.

The danger I can see is when inexperienced users in a particular domain become over-confident and can't build without the help of AI. I am also worried that it may lead to a place where complex systems get put together without knowledge of how to make the system's components simple. Humans are already great at creating solutions that is ridiculously complex because of technical debt that was not kept in check, so adding some level of AI that can just throw more code at a solution at a faster pace is a reason for concern.

It's not all doom and gloom. It can be used to simplify code as well. I'm sure we've all seen code before that could only be deciphered by a single person, and that person moved along a while ago. This is a perfect opportunity to understand what is happening and rewrite code to be clearer.

Something else that is important is to fully understand how your data is used and whether or not it leaves your system. This is a horror-show of great concern for teams in the Tech industry: Less tech-savvy users potentially feeding private and confidential information into a prompt, and who knows where that data ends up.

This past weekend I've done some deep diving, trying to understand how things fit together. Let's face it, there is much more to it than just feeding data into a prompt and getting a response. My interests go beyond that. And of utmost importance is the safety of private data...

Once one starts exploring, it's almost overwhelming. And I guess that's why it took me so long to scrape together the energy to proceed.

• AI
• Machine Learning
• Large Language Model
• Inference
• RAG
• Training
• Parameters
• Embedding
• Vector Database
• Llamas
• Qwen
• Hugging Face
• RAG (Retrieval-augmented generation)

I think that's just scratching the surface...

My weekend's exploration involved the following items:

• https://scicloj.github.io/docs/resources/libs/
  • llama.clj
  • ragcats
• https://qwen2.org/ (from Alibaba Cloud)
• https://github.com/ggerganov/llama.cpp (LLM inference)

and a hundred or so links, trying to figure out what different things mean!

So what I have tonight, is something that can take some information that I feed it, and ask a very simple question about it. The information doesn't leave my system and I'm receiving rather interesting text back.

It's literally in the exploration phase, but from what I've learned, it already changed my outlook on how it can be used down the line. It's less "scary" or should I say it's like good magic: great when you don't know the technique, but less impressive once you've seen the explanation behind the trick.

Does it mean I've mastered AI? No. It's opened new worlds to explore and it could mean a new world of excitement.