LM Studio on Fedora

Installing

After downloading the LM Studio app image for Linux, it was failing with an error on launch.

To get LMStudio working with Fedora Linux 39 Workstation Edition I did:

sudo dnf -y install clblast
mkdir -p /usr/local/lib
sudo ln -s /usr/lib64/libclblast.so.1 /usr/local/lib/libclblast.so.1
sudo ln -s /usr/lib64/libclblast.so.1 /usr/local/lib/libclblast.so

Getting started with LM Studio

Installing

Go to https://lmstudio.ai and download the version for your computer. I have used it on linux, windows, and mac. It is more of a wrapper around the AI model that is being used.

• Once you have it installed, run it.
  • You’ll be greeted with a search bar.
    This is for finding the LLMs that can be downloaded and subsequently run on the computer. It defaults to having gguf in the search as that is the type that LM Studio can work with.
• Add phi3 to the search so you have gguf phi3 in the search bar.
• Hit return.
  • Included in the search results will be: microsoft/Phi-3-mini-4k-instruct-gguf
  • Then you will have 2 options under that.
    • If you have a machine with 16G of RAM or more, consider choosing the larger model (7.6G).
    • Otherwise, go with the smaller one (2.4G).
• There will be a download progress bar at the bottom of the LM Studio window.

Hardware

Running AI models can be very resource intensive. A computer with a minimum of 16G is suggested to get started. Even just to use models, you may find that you can utilize 32G or even 64G of RAM. For instance, I have the model: TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf downloaded and it is 26.44G in size. Pushing what can be done with 32G of RAM.

The other factor is do you have a GPU or NPU? If you do, some or all of the processing may be offloaded to it. Use the settings panel to adjust for GPU, CPU, and RAM.

My best machine for this has a Ryzen 9 7940HS (8C/16T) w/NPU and RTX 4060 w/8G VRAM, 2T of SSD.

I’m considering one of the new ThinkPad T14s that are coming out with SnapDragon Elite X w/NPU and 64G of RAM. They only have a single pre-order model available now and it “only” has 32G of RAM.

Querying

• Now that you have a model, click on the LM Studio icon in the left nav bar that looks like a chat bubble.
• Choose the model to load from the top drop down.
• The system prompt/pre-prompt can be configured in the right side settings box.
• If you have a GPU you can also adjust settings for it there too.
• In the middle panel is a “USER” input where you interact with the LLM.

Response Times

When I did the chat about PIC programming, I was using the application installed in the Linux environment of a ThinkPad C13 ChromeBook. The laptop has a Ryzen 7 3700C (4C,8T) with 16G RAM and SSD. It was 73 seconds before first output. Then output came at the rate of ~2.6 tokens/second. A token is roughly a word. Overall run time was ~2 minutes.
However, on an M1 mac mini with 16 layers offloaded to the GPU, the initial response time was around 7 seconds with a rate of ~15 tokens/second!

Using an LLM can be very resource intensive. For example, I was using a 13B model in the 9G size rage on a computer with an AMD Ryzen 7 7730U (8C,16T) and 16G of RAM. All 16 threads went to 100% and memory was pegged out for several minutes before I started getting my first output. It ran at around 1 token/second. An Apple Mac Mini with m1 and 16G of RAM ran about 5 tokens/second. The Ryzen 9 32G, 4060 8G machine ran the same workload at about 11 tokens/second.

Prompt Engineering

To improve your expierence, do some research around the techniques for interacting with Large Language Models. This is referred to as prompt engineering.

Other models

When you get comfortable with everything, research/check out other models. You want to look for models that are labeled “instruct” as they are geared for taking instructions and generating responses. LMStudio has features that tell you what the labels mean and other information about the models.

Community

There is an active Discord community for LM Studio. The link is at the bottom of their web page.

Using LM Studio to query about PIC programming

Used Microsoft Phi-3-mini-4k-instruct-q4.gguf model with LMStudio to chat about programming PIC systems.

<|system|> You are an expert software developer that specializing in working with PIC systems. If you don’t know the answer to a question, say that.<|end|>

<|user|> What is a recent line of PIC that allows for programming with C<|end|>

<|assistant|> A recent line of Programmable Interface Controllers (PICs) that allow for programming using C is the Microchip PIC18 and PIC24 family from their 16-bit range. The PIC18 family, which includes models such as PIC18F25K22 or PIC18F45K20, has been popular since its introduction in the early 2000s. These devices support a wide array of features including high-performance timers and serial communication interfaces like USART/USB, which are conducive to C programming environments.

Microchip’s MPLAB XC8 compiler is specifically designed for these PIC microcontrollers, enabling developers to write their code in the C language easily with support for various libraries and optimizations suited for this range of devices. It offers features like debugging tools, hardware abstraction layers, and preprocessor directives that cater well to software development practices typical of a C programmer’s workflow.

<|user|> I have an extensive library of assembly language routines for the older PIC chips. How would I call one of those library routines in C on a PC18?<|end|>

<|assistant|> To integrate your Assembly language routines into C programming for Microchip’s PIC18 family, you will need to follow these general steps:

1. Declare the Assembly Function: First, declare an external assembly function in your C code that corresponds to one of your existing library routines. This is done by using the extern keyword followed by the name of your Assembly routine. For example:

   extern void my_assembly_routine(void);

2. Linking: Ensure that both your C code and assembly routines are compiled into separate object files, which will then be linked together to create the final executable binary. Use the MPLAB XC8 linker with appropriate flags for PIC18 devices. The linking process should include specifying paths to both the C source file(s) and the Assembly routines’ object files.

3. Calling the Routine: In your C code, you can then call this routine using its declared name just like any other regular function. For example:

   int main() {
   
       my_assembly_routine(); // Call the assembly routine here
   
       return  Writeln("Assembly routine called!");
   
   }

4. Making Assembly Routines Accessible: If your Assembly routines need access to variables or functions defined in C, you will have to use an interface function written in C that provides the necessary context and parameters for your assembly code. You may also define global labels for certain sections of the assembly code if they are reused across multiple functions.

5. Debugging: If needed, utilize debugging tools provided by Microchip’s MPLAB XC8 suite to step through the Assembly routines in conjunction with your C code execution flow. This helps ensure proper integration and function calling mechanisms between assembly language and C programming constructs.

By following these steps, you can seamlessly integrate your existing assembly language library routines into a modern PIC18 microcontroller project written in C.

<|user|> Is there a version of the go programming language that works with PIC18 or PIC24?<|end|>

<|assistant|> As of my last update, Microchip does not directly support Go (Golang) for development on their PIC18 and PIC24 microcontrollers. The primary reason is that Go was designed around the x86 architecture with its specific runtime environment, which differs significantly from the 8-bit processors used by PIC microcontrollers.

However, you can use various methods to work closer to your goals:

1. Intermediate Languages: You could potentially write a Go program that generates C code for PIC controllers and then compile that with MPLAB XC8 or another suitable compiler toolchain. This is an indirect approach using Go as the frontend but still relies on traditional embedded C programming techniques underneath.

2. Embedding: Some projects might explore embedding Golang binaries into microcontrollers by creating a bootloader that starts with a small Golang program, and then hands off control to an embedded C environment running on the PIC hardware itself. This is highly specialized work and not commonly found in standard development practices due to complexity and overhead considerations.

For these reasons, if you’re looking for native Go support within Microchip’s ecosystem for their microcontrollers like PIC18 or PIC24, your best bet is likely a combination of the two languages—using Go for higher-level logic and C/Assembly for direct hardware interaction.