I just bought a new toy for my Edge AI collection…
https://www.raspberrypi.com/products/ai-hat-plus-2
The Raspberry Pi AI HAT+ 2 is essentially attaching a Hailo-10H (product description) with 8G of local memory to the PCIe bus of the Raspberry Pi 5. Hailo provides a model zoo of models for generative AI and for vision. The device can run int4 and int8 quantized models and the selection of GenAI models is still somewhat limited, but it’s early days.
Together with the AI HAT*2, I bought a 8G Pi5. One thing I almost found out too late is that installing the RPI5 active cooler on the Pi5 is recommended, so I had to get one after I ordered the rest of the hardware. I am using the 45W USB-C Raspberry Power supply, but from what I have seen so far, the 27W power supply would have been adequate as well. I have been playing around with the device for a few days now and I am impressed. Especially the low power consumption allows me to run this 24/7 as a server in my rack. For the start, I used the DeepSeek-R1-Distill-Qwen-1.5B provided by Hailo as part of their model zoo and I also installed Open WebUI via docker on to play around with it.
OK, here are my initial findings:
The LLMs do work. I am impressed by the token speed per energy, the full system (Pi5 with AI hat) consumes about 5W when running inference.
I haven’t yet properly measured the tokens per second, but it feels entirely adequate for a single user situation. I’ve compared it side-by-side with my RTX5090 and it is much faster, but the system consumes more than 500W when running, so that’s about 100x the system power.
The software that comes with the Raspberry Pi AI Hat 2 is an API implementation of ollama called hailo-ollama, it almost implements the same API (ok for Open WebUI but I did not yet get it to work with langchain) but it lacks the ollama command line tool function, so loading models works through calling the API with curl. I also wrote a little shell script to autostart hailo-ollama at boot from the user crontab. There is still some work for Hailo and the Pi foundation to improve this but I have seen worse.
The setup for the software is a bit brittle, you need to exactly follow the reboot schedule indicated in the instruction, otherwise some of the installation steps do not detect the hardware and therefore do not install the correct linux driver. Rebooting, removing the software package via apt and reinstalling it solved the situation for me. Looking at dmesg and lspci output provided better information for debugging than the hailortcli tool provided by hailo as it essentially provides no output when it doesn’t find the properly installed linux kernel driver and the working hailo chip.
You can find the Raspberry Pi installation instruction here and there is also a github repository here where you can find the source code for hailo-ollama together with other examples provided by hailo.
Hope this helps,
H.