Hi. After a two-month break, I’m back. I moved back to NYC in March and it has been an adjustment period. I’m finally settled, and plan to start publishing here 1-2 times per week.
Another thing I’ll get out of the way–I need a job! My background is in tech marketing, community, and product. Some things I love are interviewing clients and customers, operations, and supporting people to do their best work. I am slightly technical, and am working towards becoming a full-stack dev.
Here are my links: resume, writing portfolio, LinkedIn, and Github.
Give me a shout if you are hiring or can intro me to folks that are hiring :)
Some Thoughts on LLMs
Today, I woke up early-ish, around 8am.
Around 9:30a, I finally got the gumption to leave the house. I decided to go to a cafe in Crown Heights named Winner. I’d heard “they have good bread”. That’s what I hear everyone keep saying.
I got there, ordered a chocolate croissant and a black tea, and started journaling. I haven’t journaled in a couple of months. I think I just needed a break from writing.
It was nice to journal again. For the longest time, I didn’t feel like I was getting much out of it. Today I felt like “things were moving again”. It felt good.
I cycled between sipping, writing, eating the croissant. There were a few people talking in the cafe, along the long wooden bar. One of the people seemed to know everyone that walked in.
I ended up having a really nice conversation with that person, T. We talked about a lot of things: community care, work, living in Brooklyn, the idea of “having a constellation vs. having a north star”.
Somehow we got on the topic of LLMs and ChatGPT.
Ok, I brought it up–I was curious if he’d experimented with it in his community care and mental health support work. For most of this month, I have been pouring buckets of my attention into ChatGPT, and trying to get an LLM chatbot running locally.
I’ve primarily been using ChatGPT to build software, such as a couple of python scripts to scrape a friend’s blog and Youtube to create a data archive. I’ve also been working on building a “digital art archive” with Javascript, Node, and React.
None of these things are live yet, but the repos are on my Github, if you’re curious.
You’ve Probably Heard About ChatGPT
At this point, I’m guessing you’ve heard of ChatGPT. If you haven’t, I would explain it by saying that “ChatGPT is an intelligent chat bot that is both an excellent thought partner, and is changing technology, business, and culture in many ways, some obvious, and some not”.
T. and I talked about ChatGPT and how it might be used as a resource for mental health.
T. expressed concerns over ChatGPT having sufficient guard rails to protect people. As one recent example of “mass-produced” AI doing harm, a Belgian man committed suicide after chatting with an AI chatbot on an app called Chai, reports Vice.
Here’s a recent AI chatbot meme, where it was asked to write “4chan greentext”.
I mentioned “local LLMs”, which allow you to have your own ChatGPT-like Large Language Model(LLM), but instead of you sharing your innermost thoughts and secrets with a Microsoft-owned, cloud-based product, you’d only be interacting with the software on your computer.
Using a local LLM might solve the issue of privacy and trust(i.e. sharing your secrets with OpenAI could come back to haunt you in some way, perhaps in the form of marketing, scammers accessing the data, or a state adding you to a registry for “people who have mental health issues” or something like that).
However, there’s still the challenge of creating adequate “guard rails”.
It comes down to two things:
How do you create those guard rails in the chatbot software you are using?
What dataset are you using?
I don’t have any ideas on how to create guard rails. It’s not something I’ve looked into yet, in my journey to teach myself how to use LLMs for fun and profit.
As for the second item–each LLM functions by drawing from a dataset that has millions or billions of parameters. In terms of storage space, one dataset usually takes up a few gigabytes to 200+ gigabytes or more.
To learn more about these “instruction datasets”, here’s a link to a Reddit post that contains a list of popular datasets for training LLMs. If you wanna talk about this stuff, shoot me a message or leave a comment.
Why Talk About Locally-Run Chatbots?
Well, for one, Chatbots make for exceptional and engaging “thought partners”. If you’ve spent any time with ChatGPT you know this.
Two, it’s exciting to think about the ways groups might utilize chatbots, via a shared server or domain. This is something that I talked about with T. today.
Before we said goodbye, he told me I should write about this, so that’s what this post is.
For instance, imagine a family having it’s own LLM that only they can access. Family stories and ways of problem-solving could be shared, not only over lifetimes but over generations(!). Families could share secrets about the family business, and other proprietary family knowledge, like Grandpa’s famous buttermilk pancakes recipe.
We can also imagine social groups such as groups of friends, activist communities, and religious communities creating their own proprietary LLM chatbots that are accessible only to members of the group.
Another example might be a group of friends that wants to buy land together. With their own shared, “locally-run” chatbot(I say “locally-run, but it would probably be on a server accessible only to group members), they might create a body of research, and a framework for finding, buying, and governing land together. They can distribute access to this chatbot to friends, comrades, and kindred spirits.
These friends, comrades, and kindred spirits who want to do their own collectively-owned land projects could consult this chatbot for guidance, and could contribute their own processes and learnings to it as well!
Two Ways to Create Your Own Local LLM
It seems there are two options two create your own locally-run LLM chatbot. Both are fairly technical. By “locally-run”, I mean, “runs on, and is accessed from, one computer, that does not need internet access".
I’ve seen that people are also running “local LLMs” from servers and code notebooks such as Google Colab. Someday soon, there will probably be a whole shit-ton of off-the-shelf local LLMs offered by for-profit companies and via open source.
For now, using hacky open source repositories from Github is your best bet.
If you are new to LLMs, I suggest creating an account for ChatGPT and trying out some of these prompts with it. That’s how I’ve seen most people(myself included) get started with the AI-chatbot rabbit-hole.
The “easiest” way: Use an open source solution that uses an existing open source dataset. That might be something like Dalai LLaMa(if you can get it to run, I haven’t), which is open source software that allows you to run your own LLM chatbot using the LLaMa dataset, which is an open sourced version of Facebook/Meta’s Alpaca dataset.
Full disclosure: I haven’t gotten any local LLMs to work on my Intel-based Macbook Pro yet, and, while I’m not a software developer, I do have some technical ability. Ymmv.
Here’s a list of decentralized/open source/”local LLMs”.The hard way: You can install NanoGPT and create your own dataset from scratch. If you aren’t technical, I’d avoid this approach.
If you are interested in doing this, here’s a tutorial post by a software engineer named Simon Willison, who created his own dataset entirely from content from his blog.
I tried doing this and ultimately abandoned it. One of the things I want to build–or even deploy from someone else’s open source code–is a “locally-run” Discord bot, which I’d train with my friends in our own private Discord.
I’ve been trying to get this repo to run, but no luck yet.
As you can see, there are lots of shiny things to get distracted by in the world of AI and LLMs!
AI Isn’t Going Anywhere
Despite attempts by TV writers, Samsung, and the state of Italy, the “cat” is out of the box.
AI is already causing thousands, if not more, to lose their jobs. It will likely exacerbate wealth inequality. It will change the world in countless ways, good and bad–fingers crossed that the “AI apocalypse” does not come to pass.
As they say, “if you can’t beat them, join them”. Or if you can’t, or don’t want to join them, at least learn to use their software to create a powerful, AI-driven thought partner.
Maybe even build one you can share with your friends, to solve a common problem, or work towards a common goal.
Thanks for reading. You can support this newsletter by purchasing a subscription.