Navigating the World of Local AI: Excitement, Overwhelm, and Frustration

Embarking on a New Journey with Local AI

I have finally decided to take the plunge into the world of AI and start using it regularly. Up until now, I have been hesitant to give my personal data to cloud services, but with the ability to run powerful models locally, I am ready to explore the capabilities of AI. The question that arises is whether investing in a computer or laptop with high-priced RAM is worth it for this technology. As someone who is not an AI expert, this will be a learning journey for me. However, I aim to help you determine if this journey is worth pursuing, and I will be documenting my experiences along the way.

The Rise of Local AI

Apple has been promoting its new Mac desktops as ideal for local AI tasks. Additionally, there is a new line of RTX Spark Windows machines with up to 128GB of RAM designed for using agentic AI. The concept of local AI and agents is currently trending. The idea of having a personal assistant residing solely on my desk, answering only to me and not to external entities like OpenAI, Google, Microsoft, or Anthropic, is quite appealing. With the added privacy of local processing, I can now consider delegating certain tasks to AI without concerns about privacy or security.

Exploring Local Models

As part of my exploration of AI capabilities, I have installed some local language models (LLMs) and started experimenting with them on various Mac and Windows systems. One of the tools I have been using is the open-source Hermes Agent, a self-hosted AI agent desktop app that works across different operating systems. With my M5 Ultra Mac Studio equipped with 256GB of unified memory, I have the capacity to run a wide range of models. However, the sheer number of available models and their specialized uses can be overwhelming. Generally, larger models with billions of parameters can perform more complex tasks compared to smaller ones with fewer parameters. Since I do not have to worry about token costs when running models locally, I opted to start with a large model.

Getting Started with Hermes and Qwen

Using the user-friendly interface of Hermes, I selected the Qwen 3.8 Flash Next model, which boasts 125 billion parameters and a size of around 105GB. I plan to explore smaller Qwen models on devices like the M6 Mac Mini, M5 MacBook Air, Asus TUF Gaming A14 with AMD Strix Halo processor, and the upcoming RTX Spark. By sharing your experiences with local models and hardware in the comments section, we can exchange insights and tips.

Practical Applications of Local AI

One of the initial tasks I assigned to Hermes was creating a daily morning briefing using a cron job. While this may seem basic, it was a good way to ensure that the system was functioning correctly. I also tasked Hermes with organizing my extensive Steam game library, a task that required categorizing over 400 games. With the necessary permissions granted, Hermes efficiently sorted the games into genres, allowing for easy browsing. Additionally, I utilized Hermes for data analysis, financial record crunching, and creating a spec comparison spreadsheet for a new laptop. These tasks, although simple, highlight the practical applications of local AI and the importance of data privacy and control.

Automating Benchmark Tests

One of the ongoing projects I am working on involves automating benchmark tests for laptops. The manual process of running multiple tests and averaging the results can be time-consuming. By developing a script to automate some of these tasks, I aim to streamline the benchmarking process. While the journey has been challenging, I am optimistic about the progress I will make in automating these tests.

Embracing AI as a Tool

It is crucial to view Hermes and local AI as tools rather than companions. I interact with the system as I would with any software program, maintaining a professional and indifferent approach. While AI offers immense capabilities, it is essential to remain cautious and vigilant when working with these technologies. As I continue to explore the potential of local AI, I look forward to discovering new ways to leverage its power effectively.

Conclusion

My journey with local AI is just beginning, and I am excited to uncover the myriad possibilities it offers. While local models provide significant advantages in terms of privacy and control, they are not without challenges. As I navigate this new terrain, I am committed to learning and adapting to make the most of this technology. Stay tuned for more updates on my adventures with local AI.

Follow Topics and Authors

Stay informed about the latest developments in AI, tech, and more by following topics and authors mentioned in this article. Receive updates and curated content directly to your personalized feed and email inbox.

• Antonio G. Di Benedetto
• AI
• Apple
• macOS
• Microsoft
• Report
• Tech
• Windows