My best friend Shaunak recently showed me a plugin for Obsidian he created, where an LLM base model (Qwen 0.6b to be exact). Below is a demonstration!

Pretty cool, however I wanted it to work system wide on my Mac:

Demonstration:

  • It works by calling a local model in LMStudio, I’m using Qwen 0.6b base.

    • I tried Instruct models (models that act as chat assistants like ChatGPT or other LLMs), however they tended to ‘respond’ to the text rather than ‘autocomplete’ despite attempts to control it with prompting and repeat_token_penalties.

    • I tried using other base models, like Liquid AI’s 1.2b base model but I found performance to be much worse than Qwen 0.6, which is odd considering the size difference between them.

  • Shaunak’s Obsidian plugin works via tab completion (i.e. pressing ‘tab’ to complete'), but for my system wide version I wanted a more robust solution without hijacking the ‘tab’ key and potentially interfering with other hotkeys/shortcuts.

    • I chose to attach the completions to my clipboard, so that it’s super easy to copy and paste. The only downside is that it runs pretty quickly and can fill up my Raycast clipboard history, although I can probably figure out a workaround for that.

    • This allows it to work system-wide without interfering with any other hotkeys.

  • The model uses the context from whatever is in the text field, so on its own in apps like iMessage it’s pretty useless (unless you’re typing a super long message), but in others like Obsidian or other note taking apps it works well. Both of my demonstrations had little context, but within my actual note taking setups it works quite well!

  • It was interesting to see the instruct versions of models being a lot smarter than the base models in low context situations. Not too sure why this is the case but something I’ll definitely explore in the future.

  • I’m planning on fine-tuning a small (probably 0.6-1.2b parameter model) on my own collection of writings/journal entries that I’ve written over the past 4 years. It’ll be fun to see autocomplete based on my mannerisms and idiolect and see if that improves productivity at all.

I might release the code or publish this as an app if anyone is interested, let me know in the comments! Don’t forget to like and subscribe, really helps me a lot know what content you guys like seeing and find interesting. Also please share the word!!!