For almost two years, Greg and I have been developing the "Alice" app. The idea is simple: to easily personalize AI, including connecting it with your own apps and services through automation or even your own back-end server.

Over the past few weeks, I've been working on the latest update, which will be the biggest one yet, and I wanted to share my thoughts about the development process and generative AI in general.

A few words about "Alice"

At first glance, Alice appears to be a straightforward chat UI designed for interacting with language models such as GPT-4o, o3-mini, Claude 3.5 Sonnet, or DeepSeek. The main difference here is the option to create assistants with specific skills called "snippets". Under the hood, these are just prompts with some dynamic placeholders and additional logic around them.

So, for example, if I have a snippet for fixing typos in the given text, when it's active, Alice won't answer me like ChatGPT or Claude normally would, but will write back with an improved version of my message.

The reason this happens is because the underlying "system prompt" has changed, affecting the LLM's behavior and, therefore, the end result.

Snippets aren't just about the text; they can also connect to external services through automations or your own server. Below, for example, we have a snippet that's connected to the CoinMarketCap API, allowing me to access up-to-date cryptocurrency prices. Also, in this case, I’ve used ollama.com and an LLM that runs locally on my Macbook.

The fact that I've used a local, open-source model makes this request entirely free and private, because the information about our conversation wasn't even sent to Alice's servers. This is because Alice is a desktop app that stores settings and conversation history locally on your device. Of course, when you're working with services such as OpenAI, Claude, or Perplexity, the conversation is sent to their servers. But for open-source models, this isn't the case.

And basically, that's it when it comes to the Alice app and its core concepts. Everything you see above can be organized among the assistants you can create and customize on your own or import from the library.

Now that you have some context about "Alice," we can move on and discuss details related to language models, popular providers, the current state of AI, and (hopefully) valuable lessons related to it.

Providers and their models

OpenAI, Anthropic, Gemini, Deepseek, xAI, Groq, and Perplexity—these are the providers we can consider the most relevant as of early 2025. Gemini isn't shown on the list below because I decided to disable the integration due to the lack of a spending control mechanism that blocks the service beyond a certain threshold.

Integrating so many providers isn't obvious, even if some follow OpenAI's API structure, which should make the process much easier. Unfortunately, OpenAI offers more features (like JSON mode or Structured Output) compared to the others. As a result, you still need to manage those differences.

What makes the job even more complicated is that OpenAI changes its API over time. For example, the o1-o3 mini models have a "reasoning effort" option, and in the o3 model, the "system" role became the "developer" role. Also, some settings (such as streaming) aren't available in the o1 series, at least as of writing this.

After all, connecting to the API, even if it needs some tweaks, is doable and relatively "easy". But the further you go, the more complicated things get. Let's take token counting, for example.

Token counting is crucial because each model can process a specific number of input and output tokens per request. For example, if you want to tell a user that the current conversation is too long for the model to process, you need to count tokens (this is the number displayed on the bottom right on the image below).

OpenAI does realtively a good job with this, because there are tools such as tiktokenizer and libraries for counting the tokens. But for Anthropic, for example you need to make separate api call for getting the token count. For other providers or models, it's super hard to find information about the tokenizer used. Of course, you can make the simple assumption of counting each token as 4 characters, but this only allows you to estimate the actual number, which may be very different from the actual value, because the token count depends on the language or even the emojis being used.

Tokenization isn't the only tricky part; selecting the model and keeping up with changes to newer versions is also a big unknown. Below are different versions of the Gemini 2.0 model. Just take a look at how many variables there are, but most of them are still very generic.

Besides the model's specifications, we also have API rate limits, which further complicate things, especially when we consider multiple providers and "account tiers" on which rate limits depend.

LLM's prices

On one hand, you can consider that LLM prices have drastically fallen over time. Well, yes, but that's not the entire picture you need to consider.

First of all, when chatting with an LLM, you need to send the entire conversation each time you send a message. This means that the tokens you're paying for keep adding up cumulatively. So, sending three messages of 500 tokens each isn't 1500, but 500 + 1000 + 1500 = 3000. And we still need to count the assistant's responses as output tokens (which are priced differently) and include its responses in subsequent calls as well. I know it may be obvious, but here's the catch.

In the "crypto snippet" I showed a few moments ago, it looks like we have a simple exchange between the user and the assistant. But in fact, we have four API requests here: one for the original message, one for generating a JSON payload for remote action, one for rephrasing the action's results, and one for naming the conversation.

There's some optimization happening, but we still have 4 requests with different system prompts for a single turn! And that's just for one user. For some context, you can keep in mind that the most advanced systems I use make around 100 requests per turn.

And finally, we have so-called "reasoners," which are LLMs that do some thinking before answering users. O1, O3, or DeepSeek R1 are examples of these. Reasoners may produce up to a dozen thousand tokens per answer, and as we know, output tokens are pricier than input tokens.

Open Source models are "free"

Another myth is the idea that open-source models are free. Well, theoretically, you can run them on your laptop without any costs, right? The problem is that, at least for now, small models that can be run on consumer-grade hardware aren't smart enough. Although there has been a lot of progress over the past couple of months, there's still a gap between them and the most powerful commercial models.

In the "Alice" app there is an option to connect UI to the localhost or your own server. An example here might be Ollama, which makes connecting to open-source models pretty straightforward.

Below, we have Deepseek-r1:32b running on my Macbook Pro. It generated a Snake game using JavaScript, but it took 2 minutes and 30 seconds to finish (the animation below is sped up). The catch here is that I have a Macbook Pro with the most powerful M4 series chip and 128 GB of RAM. So running this model wasn't entirely free.

The fact is that the final results were awesome because all the mechanics worked as they should.

But we're back to the fact that for anything more complex than a simple chat, we need anywhere from a few to a couple dozen requests. Our hardware quickly becomes insufficient for performing such tasks unless we have some really expensive setup. It's worth noting that these applications need parallel requests, which aren't possible or are at least quite hard to achieve.

Connecting LLMs to the Web

Language Models in their current form have knowledge limited to the training data, but we can change this by injecting web search results or even scraped website content into the system/developer prompt. Because of that, I was able to ask about the latest Andrej Karpathy's video about LLMs, and the assistant gave me the exact link to it.

At first, you might think we've successfully connected AI to the Internet and that anything is now possible. Well, that's not the case.

Several factors affect the quality of an answer generated this way. The first is the original user's query, which the LLM must correctly understand. So, if there's any ambiguity there, we'd be out of luck.

Let's take the query "Project Stargate." According to Google, there's a Wikipedia page that describes the project from 1977, and the second result relates to the latest project involving OpenAI. So, when asked about "Project Stargate," the LLM should talk about ambiguity, but it usually doesn't.

Taking a step back, when you're searching the web, getting to the info you need usually means it's been indexed. However, there are plenty of times when this isn't true, or you can't get to a website's content because you need to log in. But sometimes, users might not even realize this is the problem because of the chat UI context.

Putting all these "edge cases" aside, the given website may be indexed and even scrapable, but we still won't get the proper results. Why? Because, due to cost and performance constraints, we need to limit the system's search to, say, 5 or 10 results. In such cases, the page we're looking for may be sixth on the eleventh, and so it won't be included within the system prompt.

Let's talk about RAG

RAG may have various forms, but let's say for now it's about 'chatting with files', especially PDFs. When you think of this, it feels pretty straightforward to simply read the file contents and attach them to the system prompt. Here's my "demo":

But the moment we want to go beyond that demo, things get a lot more complicated. The first issue might be with the file itself, where the content may not be accessible to the parser, or it's so extensive that it won't fit into a single request due to the input token limit (which, again, we can usually only estimate). Even if we somehow get through this, we still need to keep the costs in mind, because even a simple conversation with a medium-sized file may convert to a high bill.

"Vector database" might be the answer, but it's obviously not because, as someone put it well, "vector search gets us semantically similar, while we need to have results that are relevant". Switching from a simple vector search to actual document processing and a real retrieval process takes a lot of effort, consumes a lot of tokens, and takes much longer than the user expects.

My experience tells me it's much wiser to build a RAG system that uses knowledge gathered and organized by the LLM itself, instead of knowledge from external sources like PDFs or Docs (unless we invest a lot of energy in processing them, but even then, we can't expect it to handle every single query the user may have).

"But Gemini handles up to 2M tokens. RAG is dead." You might see similar sentences on social media. There's a chance that even you think it's correct, because why not? I don't know how many times you've tried to work with documents that are 1–1.5M tokens long, but it drastically increases the time to the first token and lowers response quality if it goes beyond queries that may be classified as "needle in the haystack".

Even with infinite context, issues like ambiguity or the fact that some information changes over time still exist. So, if you upload sales reports from a few years ago and ask a question that requires gathering insights from that period, there's a very small chance you'll get a satisfactory answer.

So, because of that, I decided not to promise something that isn't feasible for me, especially since "Alice" only allows users to upload relatively small files.

"I cannot do that". But why?

It's a fact that LLMs are capable of actions that could be harmful or lead to harmful outcomes. An attempt to prevent that leads to results that make the given model unable to complete the task, which one may have a hard time classifying as dangerous.

The possibility of reading the text from the image in the scenario below doesn't seem wrong. Yet, the model (GPT-4o) refused to write down the text.

It seems like we now need to worry not only about the limitations of LLMs that come from the nature of the models, but also about the constraints put in place by the lab that created them.

If we're talking about image processing or "vision capabilities," we can look at the API format. Luckily, most popular providers follow a structure almost identical to the one in the OpenAI API. While Gemini and Anthropic have their own methods for passing the base64 image or the URL, it's still quite similar to others.

Still, we need to remember that not all models support multimodality. So, if our 'agentic logic' uses more than one model, we need to ensure a correct way to process images.

As we begin 2025, we can anticipate that these problems will become less relevant, with full multimodality available in future models. Even if this occurs, there's still a significant chance we'll need to handle situations like the one below, where a parsed PDF containing images has been converted to plain text. It would be much smarter to pass screenshots of the document, but current models aren't good enough at reading details.

## Self-prompt injection

Typically, prompt injection refers to scenarios where a bad actor tries to make an LLM behave in a way that it shouldn't, usually contradicting what's stated in the system prompt. While it's a huge deal in many cases, it may occur even if the user doesn't have that intent.

Below, we can see that the connection with the to-do list (Linear, to be exact) failed because Gemini 2.0 Flash gave an actual answer to the user query, when it should have given a JSON payload instead. It looks like a skill issue because I should use Structured Output, right? Well, maybe in this case yes, but the same behavior may be seen when JSON formatting isn't required. It especially happens when the user's query is quite short.

I'm writing about this, but honestly, I don't have a silver bullet that solves the issue, especially for smaller models. In fact, Gemini 2.0 Flash failing this task is surprising to me and suggests how important prompts are, suited to the given model, even if common beliefs say that prompt engineering isn't needed anymore.

LLMs are brilliant programmers

As I mentioned, it's been about two years since I started the "Alice" project. However, in version 3, I decided to switch from Tauri 1.x to Tauri 2.0 and from Svelte 4 to Svelte 5. These are two major framework upgrades that I used to create this app. The desire to add more LLM providers and extend some existing functionalities led me to rewrite almost the entire app.

Just for context, I'm a designer and full-stack programmer with almost 15 years of commercial experience, and I have to admit that I have a hard time distinguishing a single piece of code or functionality of Alice v3 that an LLM can't code by itself.

It doesn't mean that the app was 'generated' autonomously, because it took a lot of effort to guide the model (mainly o3-mini and sonnet 3.6) through every piece of the logic. But the fact that the conversation that started like this:

It's shocking to me that I ended up with an implementation that Sonnet 3.6 created in a single shot, except for one error related to importing a Tauri 2.0 dependency. Just take a look:

When I first started, I had a feature for uploading images and adding them to the conversation. However, dealing with files is quite different; while the overall flow is similar, many elements need to be handled in a new way, even though the LLM was able to properly handle the preloader animation, display an icon indicating the uploaded file, and maintain the naming convention and deletion flow.

As a result, I implemented a feature that felt quite complicated in a single thread. This example, among many others, showed me how much programming has changed for me and how I can use the experience I have to build things not only faster, but with better quality than ever before.

Final thoughts

This project taught me many valuable lessons and gave me a very useful perspective on generative AI, the general direction of the main labs, and programming itself.

First lesson:
Expectations for GenAI are constantly changing, and as a result, it doesn't match up despite the progress.

Second lesson:
Things are moving quickly, and when you put the noise aside, it's fairly easy to stay up to date with what's happening in the field.

Third lesson:
No one can predict what's coming next, and it's better to stay optimistic than skeptical.

Fourth lesson:
Even if language model development stopped today, as a programmer with almost 15 years of experience, programming has already changed for me. It's not a fundamental change, though, because the process itself is close to what I knew. I'm just using new tools, and there is a big difference in what I can achieve.

Take care,
Adam

Today, I'd like to walk you through the process of creating a personal diet coach application that communicates with users via WhatsApp. Even if this specific use-case is not interesting to you, the beauty is in the process itself, and the way it’s handled using AI and no-code tools. Enjoy!

Oh! One more thing before we start. We created a special 💎 Gems section for our paid Subscribers. In this sections you’ll find materials accompanying this tutorial, such as readymade automation blueprint. But that’s not all. Gems will be full of great tools and discounts, too!

Subscribe now

This example comes directly from my life, as I recently used a similar service with a personal trainer, which is very popular and costs $299 per month. The concept involves logging your meals and physical activity daily, while a personal trainer helps you improve your fitness and maintain a healthy lifestyle. The issue was that much of the advice was quite generic, and despite direct contact, it was challenging to get appropriate advice in the moment, such as what to choose from a restaurant menu.

So, I decided to employ AI for this purpose, keeping WhatsApp as the communication channel, which is convenient and always at hand. Here's one of the example applications of the assistant that allows you to get information about a specific meal or food:

But this is just the beginning, take a look at the following examples, like help with making better choice in the restaurant:

Analysing lab results:

And even making sense of really poor fridge contents:

What’s important, replies, even for photos are coming in a timely manner, it takes 10-30 seconds to analyse and provide results for the user:

You can build other practical tools that can be easily monetized on a similar principle. Imagine, for example, such applications:

1. A language learning assistant that sends you a daily email with a list of words to learn, and then quizzes you on your knowledge via messenger.

2. A Discord bot for programmers that improves and analyzes code. You paste a piece of code and receive suggestions on how to enhance it.

3. A mental health assistant for Founders, with whom you have constant contact via text messages, helping you reduce stress and make good decisions.

I'd like you to treat this as an exciting adventure with automation and AI, starting by creating something that will be useful for you. I personally use the mentioned assistant several times a day, and my friends, seeing how it works, are asking how to get access. Firstly, you'll learn a lot; secondly, you'll create something that's actually useful to you; and thirdly - you might consider monetizing it, and even if not, you'll certainly shine in company ;)

You can use my readymade automation blueprint! It’s available to Techsistence subscribers in 💎 Gems section.

Subscribe now

Before you start — a shortcut

There are many tools that allow you to create chatbots using channels like Messenger or WhatsApp. In some cases, this might be a shortcut for you. One such tool is Manychat, which allows you to prepare automations in a visual builder and even connect them with AI. This is how I prepared a simple bot that roasts my meals.

These types of tools have certain limitations, and we can't control everything. Additionally, they're quite expensive - using Manychat costs several dozen dollars. That's why, to have more control, save money, and learn something interesting along the way, we'll prepare such a bot ourselves with the help of automation!

Step 1: Connecting WhatsApp Account

In this step, you need to create an application and connect your WhatsApp Business Cloud account to the make.com tool. This account will be linked to your Facebook account. You'll need:

1. A Facebook account

2. An account on the make.com (from 💎 Gems section you’ll get a free month)

This is probably the most boring part of the process, but it won't take you more than 20 minutes if you follow these instructions. There's a video there that describes the whole process.

In one part, the above description is not entirely precise. Adding your phone number as a test number (which we need!) is located in this place, contrary to what you'll find in the guide:

If you've managed to do it - congratulations 🎉 This was the most difficult part of the entire process.

Now you have a working WhatsApp Business Cloud account and you can add other test numbers to it. When you're ready to monetize, you can switch to a paid number and... that's it! The result of your work should also be creating the first simple automation scenario in Make to test if the connection works. So we start from this point - add the WhatsApp Business Cloud module with the Send Message action, select the connected account and enter the phone number that's on the list of test numbers. Then, choose text and type any message. Confirm these changes and select the right button on the module, then the 'Run this module only' command.

You should receive your first message.

This stage of work can be considered complete. You now have the ability to send messages from your automation, as well as listen for new messages sent to the WhatsApp number and create responses with the help of AI. However, before we tackle that, we'll create a data structure that we'll gradually fill in, equipping our application with additional features.

Step 1.5 (optional): Quick Success

I hope that the received message gave you a bit of dopamine and you're ready for further work. And if not, then buckle up, because I'm about to show you how you can prepare a working MVP of such an application in just a few minutes, with the help of only a few additional modules!

Basically, you'll need a simple scenario consisting of 4 modules.

1. First, you listen for events with the WhatsApp Business Cloud → Watch Events module - this way you'll receive information about a message that comes to your phone number obtained from Facebook. Enable listening in the scenario and send a message with a picture - this way you'll receive data and fill in the subsequent modules.

2. The Dropbox module captures the file sent to WhatsApp and saves it in a directory of your choice. This is necessary in order to create a public link in the next step, which we can pass to OpenAI. Unfortunately, the picture from WhatsApp comes in the form of data and we cannot use this format directly in the OpenAI module, so we need this Dropbox module to save the data (it can also be Google Drive or any other cloud), and we pass the properties to this module from WhatsApp:

3. Next, we create the mentioned link in the next Dropbox module → Create/Update a Share Link. This module will receive the file we sent as input and create a link in response, which we'll use in the OpenAI module.

4. The OpenAI module takes this address in the following way, after selecting the Vision model, which analyzes images.

When it comes to the prompt itself that we use in the OpenAI module, we can start with this:

Make your best guess about the quantities of food and their ingredients. List the ingredients, calories, total calories and important nutrition data including: proteins, fat, carbohydrates, iron, 
B12. If you can give any advice how this meal could be improved or complemented during the day for better health benefits, do.
Encourage user to keep going if the meal is healthy and explain what could be improved if not so much.

5. The last module is simply sending the prepared message to the user, using WhatsApp → Send a Message.

Yes! It's that simple.

You could say that these few simple modules handled the entire process and constitute a ready MVP of the tool. Photos sent to this number will be appropriately analyzed and commented on. On the same principle, and even simpler, with just two modules - OpenAI and Send Message, you can analyze and send responses to sent text messages. This way, your users will be able to ask about anything and receive an answer.

As I mentioned, I wanted to encourage you to further work this way, and I hope you managed to create this simple example. However, of course, our application is not complete at this point. We don't have any user data, so we won't be able to monitor their progress and create reports, collect payments from them, etc., not to mention that it would be nice to prepare different paths and scenarios as well as onboarding for new users, and support for multiple languages.

Fortunately, we can do all of this by simply expanding our scenario and adding new paths, thus without coding, in a simple and intuitive way. Follow my instructions to arrive at a complex scenario that looks something like this:

Step 2: Preparing the Airtable database

For this step, you need:

1. A free Airtable account

2. A database with a structure

My database consists of two tabs: Clients and Entries, divided in this way to have separate access to client information and each subsequent conversation.

The database structure for this step is available for one-click cloning in the materials for subscribers in 💎 Gems section.

Subscribe now

Things worth noting at this stage of designing this database:

• I've included a 'lang' field that will automatically detect the client's language, allowing me to handle conversations in any language

• The tabs are connected using a Link type field, so I can easily see client data from the conversation level, as well as all conversations from the client level.

• I've added a formula in the 'conversation' field that looks like this: "https://wa.me/" & {wa_id} where wa_id will store the client's WhatsApp identifier, allowing me to access the conversation history from a hotlink.

Step 3: Basics of scenario building

With the database ready, we can start preparing the scenario in Make. Let's start from the very beginning, although you can place the modules you prepared in Step 1.5 somewhere on the side if you completed it. They'll come in handy at a later stage. In any case, the trigger must be the WhatsApp Business Cloud → Watch Events module. In this simple way, we want to listen for messages sent to us by clients.

After sending a message, it will reach our scenario. I assume we'll react to two types of messages in the following way:

• When someone sends a photo - we want to inform about its nutritional values

• When someone sends text - we want to answer the question asked

In addition, there are also various exceptions that we can handle, for example:

• When someone sends a photo of a restaurant menu - we help them make a choice

Therefore, first of all, let's make a condition that will check whether we're dealing with an image or text. This can be easily extracted from the information that will come to us about the message, so it's enough to create a Router and then set filters on the appropriate paths. For the image, it's respectively:

And for text, the same, we use 'Does not exist' as the operator.

However, before this Router module, we'll insert one more module that will check whether we're dealing with a new or existing client. This will come in handy later when we need to assign this conversation to a client in Airtable. So we put a Search records module within Airtable earlier and search by the appropriate WhatsApp identifier (in subsequent steps for new clients, of course, we'll be filling it in for the first time):

Now, depending on the received message, we'll prepare and send a response and also save it to our database.

Let's first deal with the path for text messages. Depending on whether it's a new or existing client, we'll go different routes, so we add another router when we already know that the message is text, handling a new client in this way:

And similarly, in the second path, for an existing client, the value will be equal to 1 instead of 0. This indicates that the module, marked as 16 (Airtable Search) in the previous screen, found or did not find the client in the database.

The paths for new and existing clients will be logically very similar, as you might guess, they will differ in that for a new one, we'll create an "account" for them, and for an existing one, we'll simply add the conversation to that "account". But that's not all! We'll also create something like onboarding for new users. Therefore, let's start with the simpler branch of the scenario, which is for a user we already have in the database:

The scenario we're implementing looks like this:

1. Create a record in the Entries table that contains the details of this conversation

2. Create a response to the user's question using the OpenAI module

3. Send a message to the user

4. Update the recently created record with the response

The prompt in this case might look like this:

Act as a professional, kind and caring health coach, and answer user question so that he's motivated to make better food choices and health decisions. Be casual and friendly, using simple language and explain some interesting facts like Andrew Huberman. Add that you're waiting for more photos. In this case don't use more than 1000 signs.

Of course, you can expand and change it depending on your needs, this one is very basic to illustrate what we want to achieve.

So we encourage the user to send a photo. Before we discuss the part of the scenario for new users, let's trace what happens when we actually receive a photo, not text. This requires going back a bit in our steps. We'll finish the scenario for new users in a moment - now see how you can handle photo uploads.

Step 4: Handling photo uploads

This part of the scenario looks like this:

In practice, it's not very complicated and works very similarly to what we did in Step 1.5, if you did it. So you can use the same modules now, but we'll expand them a bit:

1. I make sure I'm dealing with an existing client (first filter)

2. I save the photo to my Dropbox using the Dropbox → Upload a file module

3. I create a link to this photo (Dropbox → Create a share link)

4. I save this photo to Airtable and create a new record for this conversation

5. Then, I create a response using OpenAI and send messages

6. Next, I update the recently created record with the sent response

Why do I use two OpenAI and WhatsApp modules? I simply send two messages in a row to make the conversation more fluid. The first message simply informs that I've accepted the photo for analysis and the prompt looks something like this:

create a short message thanking user for sending a photo and inform, that you'll analyse it and send feedback shortly. If user_message contains data, also include something that confirms you understood it!Make it less than 200 characters.

And the next contains the actual analysis using the Vision model, which takes longer, a simple prompt looks like this:

Analyze the photo carefully, take into consideration in reply user_message if exists. If it contains health related data: Try to analyse the data and give your best health advice on this data. If it contains restaurant menu: Try to advice the best possible healthy choice and give reasoning why recommended food is nutrically best. If it contains food:  Make your best guess about the quantities of food and their ingredients. List the ingredients, calories, total calories and important nutrition data including: proteins, fat, carbohydrates, iron,  B12. If you can give any advice how this meal could be improved or complemented during the day for better health benefits, do. Encourage user to keep going if the meal is healthy and explain what could be improved if not so much. If it contains something else: Inform user, that you expect photos of food, health data or restaurant menu. The tone should be friendly, casual, Andrew Huberman style. Use no more than 1500 signs

This is the most important part that analyzes and responds to messages. The principle of operation is very similar to the very beginning in step 1.5, but we simply add more elements here that improve the operation of our application.

Additionally, we'll also add handling of the appropriate language to our prompts. In a moment we'll deal with onboarding and set the language of communication there, which we'll consider in each subsequent conversation. Besides language, we'll pass some user data to the prompt, so that, for example, it would be possible to address them by name. In short, here's the first part of each of our prompts:

Step 5: Handling a new user

The previous path handled a conversation initiated by an existing user. In the case of a new user, we'll handle the scenario path that deals with images in a simple way - we'll simply return a message that will trigger onboarding in the next message. It will be initialized with a greeting in any user language, so if a new user sends an image, we'll simply send this type of message:

This means that onboarding will be triggered when the user sends us a greeting in their language. It's also a way for the user to initiate the conversation. This is the preferred method for WhatsApp, so the CTA on our page could look like this: "Send a greeting in your language to our number to start".

Let's go through the path for a new user. To begin with, we want to establish one important thing: language. We'll save to the database what language the user communicates in, and we'll always prepare responses for them in that language.

For this purpose, I use the OpenAI module and create a simple prompt to figure out what language we're dealing with:

Next, I can already create in our database:

1. The user themselves with their properties

2. An entry representing the message and the response to it

So I use two Airtable modules that separately add the user and fill in all the necessary fields, using the data we received from the WhatsApp module as well as language data, and I also create a new conversation that I add to the Entries table in Airtable.

How do I link the user and the entry from the automation level? It's very simple - I just pass the identifier of the client field that was created in the previous module (14) in the appropriate field:

In the next step, we'll guide the user through a simple onboarding, collecting some data from them. We'll ask the user a few questions in subsequent messages, and we'll interpret the answers they provide with the help of the OpenAI module and artificial intelligence. The sequence might look something like this:

The subsequent prompts contain the following sequence:

• A greeting, and introduction as Klara, a personal diet coach

• Explanation of how the application works

• Information that Klara doesn't remember the entire conversation history (this will be done later in a simplified form)

• Finally, a request to send basic data that will be used to calculate BMI, i.e., height, weight, gender, and age.

This concludes the new user path, as we are currently waiting for their response. If such a message is sent, it will already be an existing user, so such a message will go through the existing user path for text messages - this is the last element of our puzzle. It looks something like this:

And this part of the scenario in order:

1. Creates a new message entry in the Entries table

2. The next OpenAI module checks if the message actually contains the information we asked for in the onboarding - if so, it creates a JSON object appropriately formatting this data.

3. Next, we check if this JSON was actually prepared, if so, it means we're finishing the onboarding.

4. Finishing the onboarding involves a Parse JSON module, which changes JSON into data that we can then place in the appropriate cells of the Airtable table, as well as formatting a response in which we thank for sending the data and encourage sending photos of their meals.

5. The second path concerns the situation when the user doesn't finish the onboarding, but simply sends some question in text form. Then the second part of the prompt I showed earlier comes into play, that is, a specific answer, advice for the user is created. We simply send such advice in the form of a message and save this answer to Airtable.

The complete, finished scenario looks as follows:

And here I've recorded a video for you that presents the entire new user path:

Whoa! That’s it (for now). I hope you enjoyed the process. This essentially shows how easy it is, using this blueprint, create coaching apps for any given topic, that really drive value! There’s still some things that could be done, such as:

• Ability to remember conversations

• Reminders

• Weekly reports

And more! Luckily, this all can be done using no-code and automations, which is simply amazing. This puts the power of creating solutions as such to literally anyone, who wants to learn and experiment.

If this article get a lot of comments and you’ll further explore our new 💎 Gems section for paid subscribers, I can create part 2 with all those missing features, and also part 3, with monetization strategy and go-to-market tools and my proven techniques. What do you say?

Subscribe now

Keep building!
Greg

Let me explain what this means.

What LLMs can do today?

Today's LLMs have limited reasoning capabilities, restricted knowledge, lack access (by default) to up-to-date information, and often cannot handle tasks that are obvious to us, like telling if 9.11 or 9.9 is the bigger number.

I personally don't know who's right — Geoffrey Hinton, who claims that LLMs are intelligent, or Yann LeCun, who says that LLMs possess primitive reasoning capabilities. In practice, it doesn’t matter much to me, and I don’t spend much time thinking about it.

What I care about and focus on is whether I can take advantage of the opportunities offered by the current GenAI and the potential of next-generation models and tools.

I spoke with Greg about AI in general, and he concluded that it's really difficult to find creative, often simple use cases that make a difference. The interesting part is that it’s not about AI itself because we’ve faced the same issue with programming, no-code tools, and automations.

Creativity comes from leaving ego behind

Claims like "AI will take our jobs" or "LLMs are useless" may be correct in some sense, but they share a common trait: they represent an attitude that prevents us from exploring available possibilities.

I don't know if AI will take our jobs, if LLMs are a "scam", or if AI is a bubble in general. Maybe programmers, designers, and writers (skills I have) will be entirely replaced by AI. Whatever which scenario will come true ultimately, I have no influence on that.

At the same time, I have influence on how I’ll use the opportunities we have available today and to what extent I’ll explore them. Therefore, instead of speculating or worrying about the future and things I have no influence over, I fully act in the area I can control.

Creativity comes from understanding

It's not difficult to notice that recently, LLMs have been occupying a large part of my attention. Despite the enthusiasm I have for technology in general, which has fascinated me since my youngest years, I try to look at it from various perspectives, to the best of my intellectual abilities. We are talking here about learning techniques for working with LLMs, but also about taking a critical look at their weaknesses.

My sources of knowledge about LLMs includes:

• Anthropic Research, which is materials published by the team whose model Claude 3.5 Sonnet is, at the time of writing these words, the best available LLM

• OpenAI Research, which is materials published by the creators of ChatGPT, probably being the furthest along in terms of development and understanding of large language models

• Stanford Online, which is a YouTube channel (but not only) where recordings of lectures and presentations are available, allowing for a deep understanding of the mechanics of large language models and their architecture

• Yann LeCun head of Meta AI, openly speaking about the current problems of large language models and the long road that is still ahead of us

• Andrej Karpathy, former head of Tesla's autopilot, involved with OpenAI in recent years, currently focusing on his own ventures

• Georgi Gerganov, creator of llama.cpp and whisper.cpp, exploring the possibilities of open language models

• Awni Hannun is an Apple researcher involved in the development of MLX and applications of open models running directly on device

• Pliny the Prompter, breaking the safeguards of large language models and tools that use them

• 3Blue1Brown, a YouTube channel featuring high-quality videos, including content in the area of generative AI

• Kyrtin Atreides openly criticizes LLMs, describing them as the biggest scam in history, yet he also finds some narrow use cases for them

Even though each of the mentioned sources and the people behind them provide me with a wealth of valuable knowledge, undoubtedly my own experiences have taught me the most.

Creativity comes from experience

Building tools and applying LLMs in applications, automations, or direct conversations with the model have shown me their capabilities. Combining this with knowledge from the "source" has helped me grasp many principles underlying the technology I work with.

Some examples include:

• General understanding of Large Language Models

• General understanding of Prompt Engineering

• Bypassing limitations of current LLMs and expanding their capabilities

• Extending their base knowledge and tackling challenges related to it

• Connecting them with real-world scenarios helps them experience both the value and the issues

• Interface adaptation and how AI presence changes them

• Preparing content, tools, and your environment for the presence of AI

As you can see, I wrote a few words about some of these experiences, and it appears that I will be writing about more of them here, so if you would like to learn about them, subscribe to our newsletter.

Practice

So, now you know my context, and we can go back to the title of this article and how it happened that almost all the code of my apps is now generated.

Rule #1: Availability

From the beginning, it was clear to me that LLMs need to be available to me all the time. I'm not talking about using ChatGPT in a browser or GitHub Copilot in IDE, but a scenario where an LLM is integrated with my laptop, phone, and executes tasks through automation workflows or a custom back-end app I've developed.

One example you can personally experience is the Alice app. This interface allows you to chat with LLM, customize it with Snippets, or connect with external services using custom Remote Snippets you can create on your own — and it makes LLM available across your Mac or PC.

To create this project, I used technologies such as Rust, node.js and the frameworks Tauri and Svelte. When I started this it, I only knew Node.js well and a bit of Svelte. The other tools were entirely new to me. So, how is it possible that, as a solo developer, I was able to create such an app?

Well, you might guess that LLMs helped me with that. I can reach out for their help whenever I need it. Not only do I receive assistance, but I've also learned a lot about their behavior, capabilities, and limitations.

Rule #2: Customization

Content generated by LLM natively is sometimes useful, but usually won't meet our needs. That's why it's worth spending time on customizing the system instruction or, better yet, using options that allow creating at least a few of them that will be tailored to us.

For example, one of the tools I use is promptfoo.dev, which allows me to automatically test prompts that I use for my AI agents. Promptfoo is a relatively new tool that is developing rapidly. That's why LLMs either don't have knowledge about it, or their knowledge doesn't include the latest features.

As you can see in the example above, the LLM generated a valid configuration file using my preferred model. I created a snippet that modified the model's behavior using my own rules and provided Promptfoo documentation as context.

Rules #3: Tools

I mentioned that when it comes to availability, I don't speak about GitHub Copilot, which is, in fact, a good tool. Today, it's much better to use Cursor as your IDE. It has built-in AI features like Copilot++, inline generation, and chat.

Cursor allows me to select code and then write using natural language to specify the changes needed. The best part is that I can reference multiple files, directories, the entire codebase, or even external documentation to provide context for the completion.

Other IDEs like IntelliJ from JetBrains also follow a similar path as Cursor, but there's not much to compare currently, and I hope it will change soon.

Meanwhile, there is one more tool that deserves special attention, and it is Aider. As can be seen below, in this case, it's enough to describe the change we want to make in the project, and Aider will independently indicate the files requiring editing and then make the changes itself, asking for our confirmation at each stage.

Aider works great in practice, and even its early versions were described by users as 'suitable for production applications'. However, I think everyone should evaluate this for themselves, especially since launching this tool is simple.

Rule #4: Use Creativity

Not without reason, I previously spoke about creativity resulting from experience and quality sources of knowledge. The value derived from LLMs is not directly related to the model itself, the prompt, or the tools you use, but rather to the way you work with them. Sometimes it's challenging to come up with your own ideas for using new tools. What I do is try to connect them with something I already do or know.

Start by setting up your social media and newsletter feeds with the best sources of knowledge, ideas, and inspiration related to Generative AI. Focus on the people or companies behind the technologies or creators who are truly doing their work. And then... just do your own thing, but explore paths you've never walked before.

90%

If you look at the content of this post so far, you can clearly see that my entire professional environment is focused on generative AI, and my attention is concentrated on blazing trails and seeking new opportunities either through drawing inspiration from others or through my own experiments.

Some key points:

• I don't delegate my responsibility to AI

• I'm constantly updating my knowledge with the latest information about LLMs and techniques for working with them

• I work with the best models available via API, including Claude 3.5 Sonnet at the time of writing this

• I use the best available tools on the market and constantly scan for new solutions using ProductHunt and X

• I learn new technologies and tools with LLM. I spend time chatting as if I were speaking with a teacher

• I generate code that is within my understanding or slightly exceeds my current knowledge or skills

• I have a habit of reaching for AI as the primary source of information, and I'm using Perplexity, Google, or StackOverflow less and less frequently

• When the LLM lacks knowledge on a given topic, I provide it by pasting fragments of documentation, code, or examples from Issues on GitHub as context for the query

• It's obvious to me that LLMs have limited context knowledge about me, my project and the features I need to implement, and the effectiveness of its operation largely depends on the way I describe my problem

• It's clear to me that LLMs have limited reasoning abilities. For the hardest problems, I break them into smaller parts or use LLMs to guide me through them rather than solve them directly

I don't use LLM daily; I use it all the time. As a result, up to 90% of my code is now generated. My focus has shifted from typing code and seeking typos to actually shaping the software.

I hope you find value in this article,
Adam

How are we doing on AppSumo almost a week after the launch?

💰 We earned $5,478.68
👀 9,726 people viewed the offer
📈 Conversion rate is 1.49%
🥵 We have 100+ questions and reviews
⭐ Average rating is 4.33/5

I'll briefly comment on all these statistics shortly. The first question that probably comes to your mind is, what exactly is this LTD? It stands for Lifetime Deal, meaning a lifetime offer on software. In short, you buy once and get updates for life. Generally, this refers to the life of the application, not yours, because LTD is a rather poor deal for app creators and developers. After an initial small cash injection, they have to maintain and develop it indefinitely. So why did we even decide to go for this? I'll write about that a bit later as well.

First, let's break down the numbers:

The entire campaign on AppSumo lasts 60 days and can be extended or launched multiple times. Some tools never leave AppSumo. They can then earn around $200-300k. How much of that goes to their creators? It depends. I negotiated hard (or at least I think I did) and managed to get a better than standard 30% deal, but not much.

The best moments of the campaign are the beginning and the end when the application is promoted on the main page. You could say that we've passed the beginning and closed it with a result of $5,478.68 (which means the application sold for over $15k).

So far, this is our net earnings, although this figure will likely change after taking returns into account. I've noticed that on AppSumo, people are very willing to return products, and the purchase is a form of "trying out" the application, so you can expect a lot more returns by the end of the campaign. This is also one of my observations on why AppSumo is so popular. People trust the return procedure, so they feel safe. You have to be ready for 20% of buyers to make a return. Fortunately, AppSumo handles this, and you don't have to worry about it, except for creating a mechanism that revokes access.

At the end of the campaign, you simply invoice AppSumo for the earned amount, minus tax, minus returns, minus their commission. As you can easily guess, this leaves you with a few percent of the original purchase price. So if you consider our cheapest plan for $49 (which AppSumo forced us to create, arguing that deals with a higher initial threshold perform much worse) and the fact that we offer 100 conversations with AI tokens included in Alice - our profit is totally negligible, if at all. So again, why even bother? I’ll answer this soon.

It's worth understanding that there are two types of campaigns on AppSumo. First, you can simply submit your application and create your own LTD, which AppSumo accepts and lists on the site. Such a deal probably gathers some reach but is not especially promoted by AppSumo. The second type of application is AppSumo Select, where you work together with the AppSumo team on launching your offer. Then it gets additional exposure on the main page, a Select badge, and is featured in AppSumo's email campaigns, and you work closely with the team to prepare the best possible offer.

That was the case for us. I personally reached out to AppSumo, and that's how we got accepted into the Select program. The AppSumo team wrote the offer content for us, as well as the script for the video, which they recorded, and helped us construct the offer and individual Tiers. Of course, we had input on everything and could provide feedback, and the entire process took place smoothly on AppSumo's Slack. If I were to recommend anything, it's definitely worth trying the Select version because, despite a bit more involvement, we get significant exposure. Just write to AppSumo, and your tool will be considered.

Subscribe now

Okay, so what does the traffic on the offer page look like? In less than a week, our offer was viewed by almost 10,000 users, with a conversion rate of 1.49%. AppSumo itself records about 200,000 visitors daily. In terms of visits, that's quite a decent result. For comparison, on the day of Alice's launch on ProductHunt, we had about 3,500 visits from that channel, securing the #4 spot. Besides the offer views on AppSumo, this also translated to visits on our site, and here's what the stats for the past week look like.

So you could say that quite a lot of people transition from AppSumo to our site, primarily to read more and compare offers. This is an important aspect of running a campaign: what you have on your site must be consistent with the offer page on AppSumo. Even the slightest discrepancy leads to continuous questions and issues, so it's worth taking care of. During the campaign, I've already changed both the site and the AppSumo offer several times to make them as aligned as possible.

Another significant advantage of running a campaign on AppSumo, besides the site traffic, is the dofollow link we get from a domain with a very good reputation. It can be said that this is a very substantial added value (similar to launching on Product Hunt). The AppSumo domain has a high DR and positively impacts SEO. From my experience, links from good domains are one of the best things we can do for our project.

Next in line for analysis are the questions and reviews. I believe this is one of the most valuable aspects of the entire campaign. Firstly, I didn't expect it at all, although perhaps I should have, because on the first day on Product Hunt, we usually respond to over 100 questions as well. Here, we received a ton of questions, which deeply engaged us in support.

And you know what? Great - that's exactly the point!

Some creators treat the necessity of supporting customers as a chore, but I appreciate every opinion and review. This is simply the most important thing that allows me to create an even better product. The value that can be derived from such comments is priceless! Not to mention the reviews, which directly give me immense value: I can place some of them on the site, and from most, I can extract specific phrases that I can use to talk about the product, because the language of customers is the best language for promotion!

To not be too cheerful, of course, there is also a negative aspect. For example, we received a few negative reviews at the beginning, which certainly negatively impacted the campaign's performance. We had to deal with them somehow; some were justified, so we quickly fixed minor bugs, but unfortunately, some were not, and we couldn't do anything about them. Despite email contact, the reviewers were not interested in resolving the issues, or maybe they didn't have any at all because the reviews were very general. So what was the motive? Hard to say, doesn't matter. On AppSumo, we have the last word, as we can add a comment to the review, so I just hope people read it and draw their own conclusions.

The AppSumo algorithm works in such a way that the main engine determining our success is engagement in the campaign. This includes gathering reviews, actively participating in discussions, and responding to questions. The AppSumo community is quite demanding, and it's important to understand and accept that. Gaining many positive reviews and activity in the campaign guarantees a better position during the campaign and mentions in AppSumo's emails, which provide a significant boost.

I also try to make the most of the campaign and engage the AppSumo team even more in the results, as we are ultimately playing on the same team. Therefore, I strive to remain active in the campaign and encourage them to take action as well 😊

Additionally, a few people, including our friend Rafał from Digital First AI, who had a successful launch on AppSumo, gave me plenty of tips. Rafał, among others, recommended reaching out to the administrators of LTD AppSumo groups on Facebook. Another friend advised me to announce the deal in several specific newsletters focused on AI. We also plan to launch a campaign to our email lists, but we've left that for a bit later and are just starting the communication.

AppSumo provided us with an affiliate link, from which almost the entire sales amount goes to us, so we also posted it on the Alice site and in the current pricing. This is important because we don't want anyone who buys at the regular price to feel bad when they find out about an ongoing deal they missed. Above all, it's worth being honest and transparent with our users. Similarly, we tried to ensure that the deal on AppSumo was not worse than what was offered to those who supported us earlier. The pricing of Alice was arranged in advance to take this into account, but it's mainly about not creating a situation where someone who buys earlier and supports us gets worse conditions than within the LTD.

Subscribe now

Is it worth being on AppSumo?

Returning to the question of why we decided on a theoretically not-so-good deal as the creators of Alice? Many of the arguments have probably already been mentioned, but I'll add a few more and summarize them:

#1 First: Market Feeling

Alice is a tool that competes in a very saturated space of AI tools. We have little time to promote Alice because solutions of this type appear daily. Probably, those that gain the right reach first have a chance of success; the rest will fail. AppSumo gives us the advantage of quickly reaching many users.

#2 Second: Why Not?

The way Adam and I run our ventures simply allows us to experiment without any consequences. If we were developing the project with a larger team and investors, we certainly wouldn't have the opportunity to carry it out. We always try to be in a comfortable situation where we can just test something and see if it works.

#3 Third: We Want to Learn

This stems somewhat from the previous point. Our main motivation is learning and drawing conclusions. Also, to be able to later deliver them to you, in this form or during a Sprint. This is definitely the most valuable for us because we gain valuable experience that will pay off in the future.

#4 Fourth: Feedback

As I mentioned, the most important thing for me is feedback from users. If places like Product Hunt or AppSumo can provide me with hundreds of valuable questions and comments in literally a few days, I think it's an unfair advantage that should be utilized.

#5 Fifth: Good Fun!

Seriously, we treat it as a sport and good fun. We try to enjoy our projects and their achievements. We are not attached to the financial outcome or specific results. We simply enjoy the process, gaining experiences, new acquaintances, and reaching more people with what we believe gives them value.

Was it worth launching Alice on AppSumo? It will probably be easier for me to summarize this after the campaign ends, but if asked today, I would say yes, considering the above.

That’s not a reason for posting, but if you want to know what Alice can do and how AI assistants can help you with productivity - of course, I invite you to take advantage of the offer, there probably won't be a better one 😊 Just please, use our link, so that we can get more than few bucks!

Get lifetime deal for Alice now »

Once you do, please, write a review 🙏 Now you know why this is especially important for us. Thank you!

Until the next one,
Greg

Autoregression is a beast

LLMs are autoregressive, which means they generate content by predicting the next word fragment (called a token) based on previous text. This process is similar to speaking in syllables:

• This

• This is

• This is an

• This is an ex

• This is an exam

• This is an example.

The problem with such content generation is that if you choose a token incorrectly, you can't remove it. What's worse, its presence affects how subsequent tokens are selected. Here's proof:

I showed you this using Completion mode and the GPT-3.5-turbo-instruct model. But models like GPT-4 or Claude 3.5 Sonnet also have this issue. It's just hidden behind the ChatML layer, which divides the conversation into "User" and "Assistant" roles.

Of course, if I hadn't manipulated the content, the answer to the asked question would be correct.

It is clear, therefore, that the existing content influences the generation of subsequent tokens. In the case of ChatGPT or Claude.ai, we're talking about the entire conversation so far (or its compressed version due to the input-token limits).

Conclusion:

• The generation of subsequent tokens is influenced by the entire content of the interaction so far. This includes the system prompt, user messages, and assistant responses.

• An incorrectly generated token has a negative impact on the model's further responses and cannot be removed by the LLM itself.

• The content (tokens) provided by us in the system message and in the user message thus enable us to steer the behavior of the model

• Prompt Engineering is therefore about increasing the probability of obtaining a correct answer by steering the model's behavior.

More balance

We don't know exactly how LLM selects the next tokens. That's why I don't talk about "controlling" the model's behavior, but rather "steering" it. But how?

If your interaction with LLM looks like this:

• USER: [your content]

• Assistant: [a concise response addressing your query, with brief explanations and relevant examples]

The language model has far more influence on the interaction than you do, as it produces more tokens, which affect the way subsequent ones are selected. Your goal (usually) will be to strive for a situation where the number of tokens delivered by you will be similar to those generated by the model.

When you think about it, it will become clear to you what the very popular prompting technique called Chain of Thought and Zero-shot CoT are all about.

Conclusion:

• You have the most control over the tokens you provide yourself. Their presence affects the tokens generated by the model. This is how you steer their behavior in the desired direction.

• Tokens can come from the model's responses to specific questions or instructions before the main task. For example, you could ask for definitions of key issues needed to achieve your goal.

Latent Space

Speaking about prompt engineering techniques, I should mention David Shappiro's post on "Latent Space Activation". I'm referring here to this file.

Note: While I can't verify his thesis, it has profoundly influenced my approach to working with LLMs. It effectively generalizes most prompt design techniques available today.

A few weeks ago, Anthropic Research published a post titled "Mapping the Mind of a Large Language Model". This post appears to discuss the same topic as David's work. Particularly helpful is this visualization:

Based on the above information, we can observe that invoking specific concepts, characters, and definitions during interaction with the model activates areas associated with them. A practical example could be the instruction I use to improve the readability of texts I write:

I use terms such as "Gunning Fog Index, SMOG, and Flesch-Kincaid" which are directly related to the issue of readability and text comprehension level. According to my understanding, these terms make the LLM "direct its attention" to areas that help it better evaluate the text and its transformation.

Beyond striving to activate selected network areas, it's also worth remembering that currently GPT-4+ class models (according to available knowledge) are based on the Mixture of Experts architecture. This means they are built from smaller models that are connected together, and during inference, only a part of them is active. Unfortunately, we don't have any information about the specific architecture and method of model selection, but it's worth keeping this fact in mind.

Conclusion:

• During interactions with the model, you should be interested in directing its attention to specific areas related to the topic and techniques useful in a given situation

Broken Complexity

The need to break down complex problems into smaller steps is probably already familiar to you, because it's useful for humans too. In the case of LLMs, we talk about "Let's think step by step" or "Let's verify this step by step". We could read about the first one in "Large Language Models are zero-shot reasoners"

A fairly common problem is the puzzle about drying shirts in the sun. Models such as GPT-4 incorrectly assume that the number of shirts affects the time needed to complete the task. The correct answer is "4" or "the same", whereas according to GPT-4 the time increases to 16 hours.

If we break this problem down into smaller steps and provide feedback on each of them in between, the GPT-4 model will be able to handle this task.

It is interesting, however, that the latest models, such as GPT-4 or Claude 3.5 Sonnet, can solve this problem without hesitation. Despite this, the principle of breaking problems down into smaller steps still applies, it's just necessary in the case of more complex reasoning.

Conclusion:

• LLMs reason "out loud" by generating tokens

• It is in your interest, then, to make the LLM generate longer responses for more difficult tasks...

• ...you must, however, ensure that this reasoning is correct and either specialize the system in solving a specific type of problem, or include a human in this process

Too little time to think

"Let step" is therefore not just about breaking down the problem into smaller steps, but also about extending the time for "thinking" by generating additional tokens.

Reasoning does not have to take place within a single interaction with the model, but can be broken down into multiple stages. For example:

In this example, the interaction with LLM has been enriched with additional steps. Before selecting the tool needed to provide an answer, we added a step generating "thoughts" about the action necessary to take. The result of this step was passed to the next one as context. In practice, this way, we increased the probability of selecting the right tool by providing tokens that increase the chance of indicating the name "Linear".

Conclusion:

• The model's reasoning can take place in many separate steps

• Additional steps may include verifying current progress

• Generated responses may serve as context for subsequent steps

• Similarly to the way tokens are selected, the quality of results in subsequent steps depends on the previous ones

Order matters

Combining LLM with application logic or automation almost always requires generating a structured response in YML or JSON format. The model's task is then usually to transform the data into the required properties. In the example, "name" and "category" are key attributes. The classification is straightforward.

In practice, classification can be more complex and depend on extensive category descriptions or even relationships between them. Then, to increase the likelihood of appropriately describing the data, we can arrange them in a suitable order.

Here, in contrast to the previous case, the task name precedes the category name. This means that its content influences the selection of subsequent tokens, reducing the risk of error.

Conclusion:

• Write prompts that guide the model to generate responses in an order that helps select suitable tokens

Dead ends

In the example concerning task classification, I deliberately included one more category, "unknown". I did this in case of situations where classification is impossible. Additionally, the instruction may also state that when the model is not sure which category to choose, it should indicate this last one. Then we will be able to assign the given task manually and reduce the risk of incorrect classification. Otherwise, the risk of confabulation is very high.

Conclusion:

• Provide default and fallback values and tell the model when to use them

• Precisely specify the source of information

Separators

Advanced prompts usually consist of several sections, such as a set of rules or additional context. These sections must be clearly separated from each other to reduce the risk of confusing their content.

For a long time, separators such as ### or """ or ``` were used in prompt content. However, they don't work well with dynamic context, which may contain, for example, markdown syntax whose content may conflict with the separator. Even if an LLM can handle such a situation, such an extensive prompt becomes less readable for a human.

The approach that works very well in practice is using XML-like tags. I first learned about it from materials published by Anthropic, and since then I've started applying it in practice with very good results.

Therefore, in the case of more complex prompts or those using dynamic data, it's a good idea to use the following notation:

Conclusion:

• Prompt sections should be clearly separated from each other in an unambiguous way

• A clear division of instructions into smaller sections is important not only from the perspective of the LLM, but also for humans

Attention is limited

Language models use an attention mechanism, which, in simple terms, determines how important individual tokens are and what relationships exist between them. Unfortunately, we do not have precise knowledge about how the meaning of individual tokens is determined.

We can make some assumptions, but we must remember they may not be fully correct. One of them might be that more tokens potentially mean greater difficulty in determining which ones are the most important. On the other hand, if additional tokens are helpful in determining the meaning of the rest, their presence may have a positive impact on the model's behavior.

It may sound like something obvious, but in practice, it's very easy to provide more noise than actual information to the context. Increasing the effort associated with content preparation or making additional queries aimed at cleaning or clarifying data will almost always be advisable.

Conclusion:

• The data that a human works on is not necessarily the data that an LLM should work on

• Separating the signal from the noise is our priority

Shoot me.

One thing that undeniably has the greatest impact on model behavior is ... a list of appropriately selected examples. Carefully written instructions that refer to a list presenting the described behavior significantly increase the model's effectiveness in a specific task.

Choosing examples is a broad topic that's hard to describe briefly. However, here are a few key rules I always follow:

• In the instruction, clearly indicate that the given examples only represent a scheme of expected behavior and present patterns. Emphasize that the model should pay attention to them and adjust its behavior accordingly.

• Examples should present what has been written in the instructions.

• Examples should also demonstrate behavior in unexpected situations or edge cases.

• Examples should present situations where the model typically behaves differently than what we'd expect (e.g., avoiding adding extra comments or explanations).

• Examples should thematically relate to the target content, but should not overlap with it. For example, when presenting task classification, choose tasks that will not appear in the target datasets.

• Take care to diversify the presented examples to avoid biasing the model in one direction.

Conclusion:

• Examples are crucial.

Don't write

Writing a prompt that correctly implements our assumptions is not easy. It is necessary to maintain high precision, appropriate vocabulary selection, and provide structure. Although it is not an impossible task for a human, it is certainly very time-consuming. Therefore, creating a meta-prompt with which we will develop subsequent prompts is a very good idea.

Here's an example of such a prompt that guides me through a certain thought process, and ultimately writes an instruction that I then connect in the Alice app (You can try this prompt out here).

The meta-prompt content contains rules and essential context in the form of useful phrases and prompt templates, as well as instructions for their application. As a result, during the conversation with LLM, we jointly design new instructions, as well as sets of examples for them.

With the help of a meta-prompt, it's also possible to introduce modifications to existing prompts. Here it's necessary to warn the model beforehand that we're about to provide a prompt which it should not follow, but rather work on its content.

Conclusion:

• Don't write prompts alone. Work with the LLM to create them

• Write a meta-prompt to help create other prompts

• Use meta-prompts to generate sample data sets

What's next?

If you have to remember only one thing from this post, let it be the fact that currently LLM content generation occurs through predicting the next token based on the content so far, and always keep this in mind when working with models.

Meanwhile, we'll be returning to the topic of Prompt Engineering many times. If you liked this post, please share it on your Social Media channels.

Have fun,
Adam

Lately, we've been tirelessly working on enhancing Alice and preparing it for a fresh release. Three significant features have been successfully integrated into this new version:

✅ Voice Actions have been improved ("save the book from a screen to my Readwise")
✅ Custom AI now allows you to incorporate your own, private back-end knowledge base
✅ The introduction of Claude 3.5 is indeed a sweet addition.

In celebration of these advancements, we're offering a unique opportunity to try the latest version of Alice, completely free (but you have to hurry as this ends on Sunday!).

You can download Alice here and initiate a 7-day free trial using our credits.

Additionally, you're welcome to bring your own API keys. Don't hesitate to give Alice a try now! Such opportunities don't come around often!

Ok, so let's explore what’s new in the latest and greatest version of Alice:

Voice Actions (beta)

TLDR; Voice Actions let you activate the microphone with a keyboard shortcut and generate a response based on Snippet instructions or the result of a Remote Action.

Remember Samantha (khem, Sky) from OpenAI's demo? 😉 Well, that went sideways. But we liked it! So here's another development of Voice Actions! Now, you can interact with Alice using just your voice. Leveraging OpenAI’s Whisper, your voice recordings are seamlessly converted to text, and responses from our language models are read back to you using advanced TTS (Text-to-Speech) technology.

In practice, Voice Actions are an additional option for Snippets, which can be enabled in the Settings (Cmd+K → Settings). Recording will only be triggered when the Snippet is launched using a keyboard shortcut. Besides recording, it is also possible to include (or skip) the current contents of the system clipboard, which also supports copied images.

For convenient work with Snippets, set a keyboard shortcut to take a screenshot of a screen fragment and copy it to the clipboard.

Voice Actions also connect directly with Remote Actions. We can dictate a message, attach an image to it, and have Alice send the specified information to the automation scenario.

A practical example might be taking a screenshot fragment and asking a question about it. Below, Alice correctly read the structure of the automation scenario. It's important to be aware of the limitations of VLM (Vision Language Model) and multimodal models. In these, confabulation (or hallucination) and simple mistakes also occur. They do well with simple tasks, e.g., reading text or general image recognition.

Here's a quick video showcasing how it works:

Something else? We knew you’d ask! 😊 Using Voice Actions, we were able to:

• Ask for explanations (like the explanation of complex scenarios)

• Interpret results displayed on screens

• Make calculations based on data from screenshots (please, double-check the results!)

Custom AI (experimental) 🤯

This is a big one for us. We're developing Alice to be the best front-end for any LLM use. And we know exactly how the future of work will look with apps like Alice. In fact, we already use it in our teams and know there's no going back.

So, what's the deal? Alice should be able to work with your company's data and connect to your custom AI and back-end. This means your entire knowledge base will be at your fingertips. Plus, you can limit access to data based on different company roles. Sounds insane, right? But wait, there's more! You'll also be able to use offline, safe, and private models to operate on these knowledge bases.

By default, Alice communicates directly with OpenAI, Anthropic, Groq, Perplexity, or Ollama servers. However, from now on, it is also possible to connect your own back-end application.

This opens up possibilities for:

• Connecting your own knowledge base

• Integrating your own logic for tool handling

• Implementing advanced "AI Agents" logic

• Connecting your own language models

This is probably great news for all of you with a bit more technical experience or those working in companies with back-end developers on board. In practice, this is not that scary, especially with an example of a server in Node.js technology can be found here.

IMPORTANT: This functionality is currently experimental and may not work correctly.

Go get it! Flash trial ends on Sunday ⚡

Typically, Alice isn't available for a trial run. This is primarily because we offer complimentary credits that come at a cost to us, and we strive to allocate ample time to support our valued clients.

However, we've decided to make a small exception because we genuinely want you and your friends to experience Alice. So, don't hold back. Grab your free trial, no strings attached - no personal details, credit cards, or any other unnecessary information required:

Starting my 7-day flash trial of Alice for free »

Simply download the version compatible with your operating system, and Alice will lead you through the rest of the process ⭐

One more thing…

We're trying really hard to manage our product and marketing time, effectively resulting in sitting nose-down in product development and having not-so-great marketing. This is why we struggle with the adoption of Alice.

And we need your help!

Please, share this link to a free trial with your friends:

https://www.heyalice.app/download

Or retweet my post on X. Or both for our ultimate gratitude 💜

Put it on your socials, send it in emails, send pigeons—please, do anything that might work. This is 5 minutes for you and means the world to us. And we'll all benefit if we have more resources to run this project.

THANK YOU SO MUCH!

All the best,
Greg

P.S.: We're still gathering your fantastic feedback from our recent post. Thank you immensely for all the kind words and suggestions. I'll compile all our learnings and future plans for our newsletter here on Substack, next week 💚 Until then!

First of all, I’d like to thank you for being with us. You are the reason why we want to deliver best quality content to Techsistence almost every week. It’s been quite a journey!

This is how Roy — my AI assistant dedicated for Techsistence, understands this project:

It is indeed our mission, to deliver high quality content on technology, paired with actionable advice and very practical approach to tools and productivity.

Our personal goal was to get more exposure with our content, so that we can start building following, gain friends with similar interest and give value to others.

The thing is — it doesn’t work.

Second best guess — we don’t know if it works. But lean towards the former.

Hard work and consistency is our default mode. That is why we just followed our gut and delivered no matter what.

After over 6 months in this mode, here’s where we fall short:

❌ We did not get any platform exposure from Substack

And honestly, we were counting on it. We did not think about the algo, instead we just tried to put 10x content out there and give our best in extensive, detailed articles. I don’t think we ever got featured anywhere nor get traffic from channels other than our own.

❌ We did not get any subs from Substack’s mechanisms

There are some fantastic publications that started recommending us (thank you!) but we gained only 2 subscribers from it:

On the other hand, we’ve recommended 500+ people to other newsletters:

I’ve reached out to all of their authors, hoping they could possibly return the favour, but never got any reply from them.

❌ As a matter of fact… we did not really get any subs

We’ve started with importing our various lists to Substack so that we could unify them here. We had almost 2900 subscribers on those lists. Initial subscriber churn is understandable — we started delivering something new and on regular basis, not everyone will be interested.

But we kept loosing subs for almost four months (!) until we started gaining them again. And yes, we have over 3000 subs now, but this is only a 100+ difference from the beginning. After 6 month writing 10 hours a week on average!

❌ Our other channels don’t work for Substack

We have some following on other platforms like X and LinkedIn. In fact, to put it all together we have over 25.000 followers (probably a lot of them overlapping, but still). Putting information about our new posts, some quotes, or summaries (and even whole articles) brought us close to nothing in terms of clicks and engagement, and those posts were certainly buried down by the algos.

❌ Substack is silent

I have to say, that posting here has been similar to writing a piece of content, printing it, rolling so that it fits in the bottle and throwing to the ocean. We almost never heard back from people reading our pieces. I can’t honestly remember ANY comment below our article or post, not to mention any meaningful discussions or people enriching the articles with their own experiences and use-cases.

And this might be our fault of course. Maybe our articles are just not good and engaging enough. The thing is I don’t see it elsewhere, too. Most of the writers with solid following get some comments and reshares, but they doesn’t seem very deep.

❌ There is no SEO

There are hundreds of pages of content on Techsistence already. Although we do not optimise it for SEO nor stuff with keywords, we try to be consistent with our phrasing and within our niche. We have .com domain pointing on Substack. For months organic search was dead 0, only to rise a bit recently, with irrelevant traffic.

❌ Paid churn is real

Although we got some paid subscribers, they don’t seem to stick. This is also understandable — most of our content is free. We don’t offer much for paid subscribers. Yet maybe this shouldn’t always be the case. There might be people around who are willing to support us even if (or because of that) most of our content is free and accessible to anyone. It seems it’s not really the case.

I am open to accepting the scenario, where our content is just not good enough. This could be a valid answer to all those metrics. After all, there’s so much content being put out there every day and the market is really competitive, with tons of talented writers!

On the other hand — without feedback I just don’t know (and I don’t give up easily).

This is why I have a great favour to ask.

Please, help us better understand, why we’re failing here.

Here’s a short survey with questions that will help us make more educated guesses on our content and your needs. We want to serve you better and we feel we’re missing some alignment here.

This is the time to take action — if you care enough for us to get better and start delivering more engaging value and content.

We really appreciate your help. Please, vote in the polls below:

Lastly, the most important thing!

Please, answer this open-ended question in the comment:

What is the one thing we could improve to serve you better here?

Please, don’t be a stranger. We need this advice and motivation to keep going!

Thank you.

Greg, Adam & Alice

Text editor(s)

Writing is a crucial aspect of my work and various processes, including learning. Text appears in many areas, such as daily communication, emails, newsletters, articles, scripts, website descriptions, documentation, and documents. Reports from Rize show that the text editor is the application where I spend the most time (23%). Therefore, I ensure that writing experiences foster the creative process.

I work with several text editors. They are:

• iA Writer: In my opinion, it's the best Markdown syntax editor, with simplicity as its main advantage but it falls short in note organization and lacks related features.

• Paper: A great candidate to replace iA Writer, the writing experience is even better. Unfortunately, the app has some bugs and limitations, making it suitable only for simple notes.

• Obsidian: An app I rarely write in and usually just paste content from iA Writer or Paper. But it’s great for organizing content. Paired with Vitepress, I can share most of my notes online as a static website.

• Notion: It's the final app where I primarily collaborate with others to create content, share it, or leverage automation. Notion stands out as the sole editor mentioned that offers an API, enabling integration with my services.

Using multiple text editors actually works pretty well. iA Writer is for long-form writing, Paper for quick notes, Obsidian for organizing, and Notion for sharing with others. The common thread is the Markdown syntax, which I wrote more about in Plain Text is What You Need.

When I write, GPT-4 or Claude continuously assists me. Their role is to correct and enhance readability, as well as make similar transformations to the content I produce. LLMs also perform well during brainstorming sessions and discussions on my current focus topic.

While writing:

• Editor is in full-screen mode

• Typing on Wooting 60HE keyboard

• Listening to music on Spotify or Endel

• Taking screenshots with Xnapper

• Generating code snippets in ray.so via Puppeteer automation macro

• Optimizing images with a macro linked to TinyPNG

• Sharing images and files using Dropshare

• Hosting images and files on DigitalOcean

• Using converters for HTML → Markdown, Markdown → HTML, Notion → Markdown, and Markdown → Notion. This allows me to write newsletters and blogs in markdown editor, then automate conversion to the target format. Uploading files to my own hosting is key here.

• Keyboard settings: "Key repeat rate" at maximum and "Delay until repeat" at minimum

• Using text expander for frequently repeated phrases like proper names (e.g., .tech becomes Tech•sistence), URLs, email addresses, contact details

• Navigating almost entirely with the keyboard, avoiding the mouse. Useful shortcuts include: move cursor to start/end of word (Command + ← or →), start/end of line (Option + ← or →), start/end of paragraph (Command + ↑ or ↓), delete next/previous word (Option + Backspace or Option + Shift + Backspace), select line/word/paragraph.

• Using trackpad gestures for controlling music, managing windows/desktops, switching text editor functions, or sending selected text to AI assistant.

• Utilizing keyboard shortcuts for actions related to screenshots, file optimization, and uploading.

• Using clipboard manager Paste for easier text editing and returning to previously copied content (links, snippets).

All these activities work for each app I mentioned. Even though Notion lets you paste images right into the document, I still upload them to my server first. Direct uploads to Notion expire and need you to log in, which messes with automation.

General Workflow

Tasks, emails, and events are another well-optimized area. Unlike text-writing tools, the priority here is API connectivity. This enables leveraging automation and AI for data organization. I wrote more about this in User Interfaces may change because "AI knows what we mean".

Managing tasks

Linear is used both by our product team and myself to organize my work. All my tasks go here, and I rarely enter or edit their statuses manually. Instead, some entries appear due to automation (e.g., a new message with a label or a new file on Google Drive needing my attention). I add other entries via voice messages on Apple Watch or messages to the AI assistant on the Slack channel.

Of all the task management tools, I've liked Todoist and Linear the most so far. Both offer APIs that allow nearly unrestricted data management. Besides the API, a clear GUI and keyboard shortcuts are crucial to me, and Linear definitely outperformed in this area. As seen in the screenshot above, you can customize views to suit your specific needs or current situation.

So, on the topic of task management:

• I mainly add and edit tasks via voice or simple Slack and Alice messages. The AI then assigns new entries to the appropriate categories, priorities, and deadlines.

• The organization and updating of entries in Linear is handled by a series of GPT-4 prompts, which consider the rules I've defined. If needed, I can override them by simply stating that a given task should be assigned to a different project than indicated by the category descriptions.

• Automation fills a large part of my tasks with various events and schedules. When possible, the AI assistant fetches additional information for the task description.

• My priority is API availability, as automation allows me to focus on executing tasks, removing most of the organizational burden, except for planning-related activities.

E-mail

For years, I've been rocking Google Workspace with Superhuman. No surprise there—Gmail's awesome on its own and packs some serious automation power. You can set up slick scenarios on make.com or dive into advanced filters right within Gmail.

For me, email automation means responding to new messages that Gmail labels automatically. For instance, emails with "Invoice" get the "Invoices" label. This kicks off a Make.com scenario to handle the message, like saving attachments to Google Drive or adding tasks to Linear.

Sometimes, filter rules miss a few messages. When that happens, I manually add labels in Superhuman using keyboard shortcuts. Makes the whole process way easier.

Working with email also involves an account connected to an AI assistant, which can send various messages after completing assigned tasks. I wrote more about this in the post Personal AGI.

Regarding email management:

• Superhuman's keyboard shortcuts and overall look & feel make working with email enjoyable and fast. Its high price is justified (though this is very subjective).

• Gmail / Google Workspace is a "must-have."

• Combining automatic filters with labels and automation greatly simplifies email management, document organization, and prioritization.

Calendar

My calendar's pretty chill, with just a couple of bi-weekly recurring meetings. This setup lets me focus and handle communication asynchronously through Slack or other channels.

You can book a slot in my calendar through Zencal. These pre-set time blocks automatically disappear when a meeting is scheduled or another entry, like a trip, appears in my calendar.

I can add new calendar entries via voice messages to my AI assistant, similar to a task list. The assistant can also check my availability or retrieve events from a specific range. Ultimately, I still occasionally check the calendar myself, and then I use the Notion Calendar app.

Notion Calendar in its own way resembles Superhuman and offers intuitive keyboard shortcuts and a minimalist interface.

Regarding calendar management:

• Managing entries is done either by voice or through simple messages to an AI assistant (requires custom integrations)

• Zencal is a brilliant tool for scheduling meetings (including paid ones), which can incorporate various automations (e.g., sending forms before the meeting or notes after the meeting)

• Notion Calendar is a good, though not perfect, client.

Web Browsing

Arc Browser is my main browser on both macOS and iOS. Despite the ability to organize tabs or profiles, I only use its basic functionalities. Yet, the Command Bar (Command + T) and Little Arc (Command + Option + N) make a significant difference for me.

When I come across an interesting source (like an article, publication, or video) that I can’t review right then or know I’ll want to return to, I use a keyboard shortcut to send the open page to the AI assistant. The assistant then fetches available info about the page and saves the link to it in Linear and Feedly, pinning it to the appropriate board.

Staying up-to-date and monitoring news on topics that interest me is handled through Feedly. Like with previous tools, the main factor influencing my choice here is the availability of an API.

I've connected Feedly boards with Make.com automations, so simply pinning a post triggers additional actions. For instance, it saves the link in my assistant’s long-term memory. This memory is linked with search engines (Algolia and Qdrant), making it easy to find previously saved sources without needing to remember their exact names.

So, browsing the web for me means:

• I use the browser just for Internet browsing. I don’t save bookmarks or other info. Instead, I use macros and automation to organize knowledge in set places, all controlled by LLM (GPT-4-turbo and GPT-4o).

• Staying updated and exploring new topics with Feedly and sites like Product Hunt, daily.dev, Indie Hackers, or HackerNews.

• The foundation of the whole system is my custom AI assistant and its long-term memory. Note, this isn't Alice from heyalice.app, but my private app, which I'm gradually making publicly available.

Graphic Design

Designing UI and promotional materials is an integral part of my work. I mainly use Figma and recently returned to Adobe Photoshop Beta due to its Generative AI features, which are excellent for editing images and assets generated in Midjourney.

Below is an example of covers for one of my courses, which I generated in Midjourney with slight editing through Adobe Firefly.

Figma works phenomenally for UI design, especially when using components and the auto-layout feature.

There are creations generated according to templates, where creating new versions involves only text replacement or simple layout editing. In such cases, Webflow becomes my graphic editor.

The AI assistant only requires information from me on generating the graphic (or graphics), plus the necessary details to create them (e.g., text, background image link). It automates Webflow contact to update the CMS entry, downloads the page as a PNG, and sends it to me.

When designing creations, I always use Midjourney Alpha. I generate various graphic or asset variants there, which I often combine in Photoshop.

Summarizing the topic of graphic design:

• I use Generative AI, and currently, Midjourney or Stable Diffusion 3 works best.

• Figma is undeniably the best available tool for interface design or creating advertisements. It's definitely worth getting to know its more advanced features, which save a lot of time.

• Webflow combined with HTMLCSSToImage allows for automatic generation of graphics based on templates. Alternatively, you can use the latter tool directly, where you only need an HTML template in which you replace individual elements with automation.

• Combining LLM with a set of templates allows for easy generation of entire sets of advertisements in various variants.

Programming

Programming is the second (after writing texts) activity that takes up most of my time, and therefore I also pay a lot of attention to it in the context of optimizing the entire process.

I use the following tools:

• IntelliJ IDEA: This is my main code editor. Although I don't program in JAVA, IntelliJ works great with various programming languages (for me, it's TypeScript and Rust).

• Supermaven: This is an alternative to Github Copilot that I have just started using, and it makes a great first impression by suggesting code very intelligently.

• iTerm: Although I have gone through many different terminals, iTerm won with its simplicity and the fact that it performs its task perfectly.

• TablePlus: This is a great database client for macOS.

You can see my IntelliJ setup above. I've turned off all panels and extra options since I always use keyboard shortcuts. This makes sense given the IDE's extensive customization options and the inclusion of a "Command Palette" style search.

For actions that do not have a keyboard shortcut, or I simply use them rarely, I use Search Menu Items, one of the options of the Raycast application.

Since ChatGPT's release, large language models have been my constant programming companions. I'm not just talking about code generation or debugging but also problem discussions, design decisions, and learning new technologies. Of course, I always make the final decisions, but models like GPT-4-turbo and Claude 3 Opus are nearly perfect conversational partners.

So, on the topic of programming:

• I use the best available tools that work for me. For example, despite Visual Studio Code's enormous popularity, IntelliJ works far better for me. The main reason is that IntelliJ simply "understands my code" much better than VSC.

• Generative AI takes pair programming to another level for me. In this case, such a partner is available to me all the time and often has much more knowledge than I do, although it quite frequently makes mistakes. Despite this, the overall balance of mistakes versus the value I receive is definitely positive.

• Where justified, I use tools to automate code tasks. Application deployment is a "must-have" I always perform through GitHub Actions.

• As seen in the previous points, I use my programming skills not only to develop products and work but also to create tools for my own needs.

Macros and Automations

I'll write a separate post about macros and automation, as it's too extensive a topic to cover in a few points. I'll just note that I work with applications: Shortcuts, Keyboard Maestro, Better Touch Tool, Raycast, and of course make.com.

My system is powered by my personal API, seamlessly integrated with my AI assistant. This setup lets macros and automations communicate with me and each other. It evolves with new language models and tools.

So, if this topic catches your eye, subscribe to Tech•sistence to see my take on it👇

Fun

Outside of work, I'm also a fan of books and games, spending my free afternoons and evenings with them. I log all the titles I've read on my Goodreads profile, often sharing highlighted passages or notes.

I always read books in the Amazon Kindle app or listen via Audible. It’s very helpful that if you have both the audiobook and e-book versions, progress syncs between devices, making reading much easier.

Lately, I’ve been diving less into books since my interests have shifted to cutting-edge stuff like Generative AI and new programming languages/tools. When I do pick up a book, it’s usually a challenging, timeless one that takes more time than your typical business, psychology, or economics bestsellers.

For games, I’m all about the PlayStation 5 right now. Next up is Nvidia GeForce Now, streaming from my Steam account and more. On the go, I rock the Steam Deck—such an impressive device—and the Nintendo Switch.

While gaming and reading, I also jam to music, which my AI assistant hooks up. So, having an API and cross-platform functionality is a must. Spotify nails it.

When I need to focus, e.g., while programming, designing, or writing longer texts, I also turn to endel.io.

Summary

I didn’t cover every tool, but I think I gave a good overview of my current setup. If you have any questions about the tools I use or how I work with them, feel free to ask in the comments! Wrapping up, my setup is always evolving. Even if some tools have been with me for years, how I use them changes. It's key to my growth and daily life. My setup isn't just about boosting productivity—it's about making each day more enjoyable and fun.

Have fun,
Adam

The logic of these systems combines application code with the capabilities of large language models (LLM). This solution allows for flexible adaptation to current conditions, development of an action plan, and its implementation using both internal and external information.

Let's see if such systems can become part of our everyday life, based on the example of the agent I work with.

Technology has limitations on what it can accomplish. You do not. ~ Lou Gerstner, Former CEO of IBM

Scope of the "Personal AGI" system

"Personal AGI", or personal general artificial intelligence, is a fairly good term for a system whose goal is to accompany us in virtually every situation. Of course, this should be taken with a grain of salt, as it is not a system that meets the definition of AGI, but the main idea here is to adjust our way of thinking in such a way as to go beyond the patterns defining how we use LLM.

Agent AI usually consists of several components, such as memory, tools, logic, and interface. Each is an element of a whole that cooperates according to established rules, which can be defined more or less precisely.

Memory: Short-term memory includes information stored for the current conversation or its part. In contrast, long-term memory can be used over many conversations. Agent AI should have the ability to freely access memory (both for writing and reading). An additional element of memory can also be something I call "context" (or simply "document") which is usually a larger set of information with which the agent can work or refer to. For example, after loading the content of a website, it is not directly entered into the prompt, but has the form of an identifier to which the agent can refer if necessary.

Logic: The operation of the agent takes place through a combination of programming logic (e.g., HTTP queries, functions, or file system handling) and a series of advanced prompts covering self-reflection, intent recognition, internal speech, planning, or elaborate contemplation through content generation. Here, the use of various models and processing information from different sources comes into play.

Tools: LLM can precisely select tools for problems, generate structured data needed to use them, and interpret the results returned by them. In the digital world, tools take the form of functions that can be run locally in the application code or through an API. The API can include both external services and the assistant itself, allowing even to ask questions to oneself or refer to one's own memory or skills.

Interface: Contact with the agent can take the form of text chat or voice messages (e.g., through integration on a Slack channel). The use of API also comes into play, allowing for automatic sending of messages to the agent, either according to a schedule or as a result of an event. Consequently, the agent should also have the ability to proactively send messages directly to us.

Naturally, the structure of the agent and the available modules may vary depending on the purpose.

Environmental Information

The agent can access current data using tools, but sometimes technical limitations of devices and services prevent this. Additionally, certain information should be available by default, without needing any tools.

Examples of such information include:

• Current date and time

• Our location

• Current weather

• Device status

• Active applications

• DND mode

• Distance from home

Data can also encompass areas like sales statistics or recent events. Updates might happen through automated scenarios or scripts triggered by specific events. This leads to a situation where the agent adjusts its behavior not only during conversations with us, but also when performing tasks independently!

Short-term memory, long-term memory, and document handling

LLMs have token limits that they can receive (input tokens) and generate (output tokens). Besides, we are also interested in relatively low costs and an acceptable response time (which can be significantly reduced thanks to Groq).

Short-term memory: Includes the content of the current conversation, as well as additional information that was read during their generation. For example, if the name "Alexa" appeared during the conversation, my agent will read information about my dog, but in the next message, this information will no longer be available. For this reason, at least a summary that allows the conversation to continue should reach the conversation context.

Long-term memory: This is a topic for a separate post, and I partially addressed it talking about No-code Agent. Nevertheless, we will stop here for a moment.

The agent's memory is a database searched during conversations to load necessary context for responses. While this task seems simple, it's actually a complex challenge without a current solution. We are focusing more on improving these mechanisms' effectiveness rather than perfecting them.

A very basic memory structure can be divided into different areas concerning known people, remembered resources, or locations. It may also contain more specific areas, including knowledge about professional and private life. Such a division quickly proves insufficient, and it is necessary to include additional categories or a tagging system.

The whole must work both for the needs of storing information and organizing it, as well as for later updating, forgetting, or searching for the purpose of conversation. On the one hand, almost the entire process can be handled by LLM. On the other, as I mentioned, it is currently an unsolved problem if we want to talk about 100% effectiveness.

Context / Documents: In working with the agent, situations may arise where it will be necessary to process longer content. An example might be correcting a long blog article (such as this one). Even if we manage to fit it entirely in the context window (input token), the model will not be able to generate a response containing a full translation at once. Therefore, we must divide the document into smaller fragments and then process each of them. However, even in such a situation, the LLM will not be able to deliver it directly to us. Instead, the tool responsible for translating the document can only pass an identifier, which will be used in the model's statement and then programmatically replaced with the proper content.

An example of such behavior can be seen below. My message contained a link to a file containing the content of the post that was to be corrected and translated. Despite its volume, the programming part of the agent's logic processed its fragments, then combined them into a whole and saved them in a file. The LLM only received the URL, which it included in the email it sent me.

Although context limits are constantly increasing, the mechanics responsible for processing content in parts may prove very useful. An example might be the substantive verification of a post, in which individual parts of the article are transformed and juxtaposed with information from the Internet or the agent's long-term memory.

Tool organization and handling

The agent may have access to virtually any number of tools as long as we ensure that it can effectively use them. Tools can serve various functions but should ideally have a uniform input/output interface. Of course, it is possible to use models for direct interaction with API services, but almost always we will want to add our own logic, which requires the presence of an intermediate layer.

The tool list might look like this:

As you can see, we have access not only to the API but also to various functions that extend the capabilities of the LLM or add options that it does not inherently possess.

In designing tools, it is worth paying attention to:

• The aforementioned consistent interface

• An additional intermediate layer containing protections against model hallucinations and limiting access to selected actions

• The ability to perform many different actions within a single query (e.g., adding and updating several tasks to Todoist simultaneously)

• The way data is transferred between tools (e.g., the aforementioned passing of identifiers or links to files)

• Providing additional data needed to complete a given task

• Proper error handling, attempts to fix them, and appropriate reporting

Thus, for example, the "Add task" action expects a JSON object containing a list of entries to be updated, another containing those to be added, and another with those to be removed. Additionally, there is also the possibility to retrieve tasks from a designated period (e.g., the next two weeks). Besides, there is also the possibility to download tasks from a designated period of time (e.g., the next week). In response, we receive a JSON object containing all the information necessary to provide a response by the LLM or to take further actions.

In my case, the list of tools is divided into categories:

• Media: tools enabling image and audio file processing, including their description and generation

• Business: includes tools directly related to running a business, especially formalities and statistics

• Communication: tools through which the Agent contacts the outside world or directly with me. Of course, all messages sent outside go to a queue, which I personally must confirm

• Data handling: includes operations on files, including the ability to upload them to the cloud for further sharing

• Memory: includes full memory management

• Action planning and alerts: a set of tools that allow for background operation, monitoring, and reporting

• Content management: contains all tools related to advanced text processing, including the processing of images contained in it (the default processed format is markdown. More about this in "Content is the King but for how long?")

• Additional tools: all tools that do not fit into the previous categories. They mainly include individual interactions with services and devices I use daily.

Currently, implemented tools allow me to issue commands such as:

• Remember that...

• Gather everything I know on this topic

• Send an SMS, email, or Slack message

• Review these photos

• Play my favorite music

• Watch these movies and tell me what they are about

• Read these articles

• Proofread this post

• Issue an invoice

• Prepare a report

• Add these tasks to the list

• Check how long it will take to get to this place

• Remind me about something important

• Perform this task every Monday

• Record a podcast

• Compare these files

• Check this document for substantive content

• Prepare the car for a trip

• Publish these posts on social media

• Send a newsletter

• Prepare advertising creatives

• Save this in my notes

• Generate images matching the description

• Turn on the lights

• Turn off the computer

Interestingly, issuing more complex commands also comes into play, such as:

• Watch these movies, list the mentioned sources from them, search the Internet, describe them including links, and send me an email

• From now on, check the following sites every morning and if they contain new posts, record a private podcast for me and send the link on Slack

• Check if these documents contain any information on topics related to the projects I told you about

• Help me prepare a title and post promoting my latest blog entry and generate covers for it.

• Correct my article for typos and syntax, then check it for substantive content and prepare a version in two languages. Send me the finished files and let me know when you are done.

• Convert these messages into tasks or events in the calendar based on their content

• Listen to this voicemail and write a note based on it and send it to me. Save a draft in my notebook

• Compare these graphics and the messages on them with the tone of the brand of the project I told you about. Apply the techniques we discussed and suggest changes

And of course, all these commands can be sent by me either vocally or textually, or be sent automatically with the help of scenarios connected to the agent's API. Despite the fact that the current logic is not particularly complex, it turns out to be sufficient for performing tasks with very high efficiency. Unfortunately, this comes at the cost of speed of operation and a high bill at OpenAI, reaching several hundred dollars a month. However, considering the value I receive from this, the costs are justified.

Agent Logic

The components we just went through are just as important as the logic responsible for connecting them together. Here, just like in the case of memory organization methods, work techniques, and tools are still at a very early stage of development, and we are largely reliant on our own creativity and what we know about model behavior.

Let's start with the fact that currently, the most demanding tasks are best handled by models GPT-4-turbo and Claude 3 Opus. For less demanding tasks, we can successfully use models Claude 3 Haiku, and Llama 3. I base almost all the logic on the GPT-4-turbo model, as optimization against weaker models will come a bit later. I also count on the fact that a new version of OpenAI models will soon appear, which will further facilitate the development of agent logic.

A simplified scheme of my agent's logic for the message "Hey there Alice! What's up? Alexa says hi btw 🙃" consists of the following steps:

1. Gathering context: Retrieving environmental data, summarizing the ongoing conversation, list of skills, and memory areas

2. Self-reflection: This initial stage is where the agent considers what to do with the received message. It compares it with available memory areas, the ongoing conversation, environmental data, and its list of skills. The result of this stage is a few sentences of reflection, resembling internal speech or ordinary thoughts. The generated content is very valuable from the perspective of the next steps, as it significantly increases their precision.

   

3. Action plan: Based on self-reflection, early action planning and memory areas to be included in the memory recall process are determined. The generated content is a JSON object containing two properties queries and actions, where the first contains a list of questions to its own memory, and the second contains a list of actions to be taken.

   

4. Recalling: Using the action plan and self-reflection, the agent generates a list of questions to different memory areas. This way, it increases the likelihood of finding information that was not directly mentioned by the user but may be useful in providing a response. For example, to a simple message "hey, how are you?" the agent not only retrieves information about itself but also reads the profile of its interlocutor, current weather, or location to use it during the conversation.

   

5. Internet search: If required, the agent may support itself with an internet search engine, also generating a series of queries to help it better understand the mentioned topics

6. Taking actions: If some tools were mentioned in the action plan, at this stage the agent starts using them while also considering how to use each of them. Moreover, actions are initiated in such a way that the result of one can be used in another.

   

I will add that the above scheme will soon change to increase the speed of operation and overall optimization of the entire logic. However, even at this stage, it can efficiently use the available tools, even if the queries are complex and require reaching for several of them. So, if I ask to check the list of tasks and events left for me for the day, the agent's logic will include:

• Retrieving information about the current date

• Indicating the tools "google calendar" and "linear"

• Running these tools to retrieve entries for the current day

• Providing an answer to my question

The same principle applies to every other command or message. Thanks to autonomous logic, it is possible to accomplish even quite complex tasks without human involvement. At the same time, everything moves within a very strictly defined context, and it is impossible to issue absolutely any command. However, the system is sufficient to provide me with real support in daily tasks.

Conclusions, Observations, and the Great Opportunity

Autonomous agents are currently one of the hottest topics in the industry. Despite their huge popularity, there are clearly a number of unanswered questions or outright barriers that are very difficult to cross. On the other hand, when I compare the capabilities of the gpt-4-turbo model with what was possible just a few months ago, I see how much progress we have made. Seemingly these are small changes, yet improvements in reasoning (e.g., calculating dates or maintaining attention) are huge.

Looking now from the side at my previous experiences, I turn my attention to issues such as:

• Organizing memory in a graph structure or finding another way to link records located in different memory areas

• Designing mechanics capable of dynamically building memory structure or at least moving within universal categories with mechanics enabling scaling

• Modifying the agent's logic in such a way that each action has access to different memory areas or search results on the Internet

• Breaking down larger prompts into smaller steps and performing them in parallel wherever possible.

• Including advanced techniques for combining prompts with particular emphasis on self-consistency - Incorporating advanced techniques for combining prompts with a special focus on self-consistency

• Modifying the action logic in such a way that some of them can be executed in parallel, while others are interconnected. Currently, only a simple queue is in play, which is insufficient from the perspective of autonomy

• Including better error handling and overall performance optimization, including cache memory mechanics

• Adding full monitoring of application logic for easier debugging and making changes

• Adding the capability for autonomous analysis of logs and executed prompts for their later optimization

The above points also do not consider limitations that should also be considered:

• Even with the GPT-4-turbo model, there are clear problems with maintaining attention on the available context

• Logic consisting of multiple prompts requires dividing them into modules, which in turn greatly complicates their evaluation

• The speed of model operation is very low, especially when many prompts are involved or there is a need to process longer documents.

• The costs of operating models also remain very high in situations where each of our responses consists of a series of prompts

• Subsequent versions of models, although seemingly offering minor improvements, gain significance in the case of autonomous agent logic. Each additional percentage chance of getting the correct answer quickly starts to add up, affecting the overall picture

• The ability to create and verify prompts still plays a critical role and currently, little indicates that this will change in the near future

In summary, about "Personal AGI", creating an application capable of addressing even a few activities from our daily lives has become within the reach of practically every programmer. It is relatively easy to create a system capable of remembering information or performing more or less complex actions tailored to us.

At the same time, undoubtedly creating such a system that will be universal enough to be widely available to users remains beyond our reach, mainly due to technological limitations. Therefore, it's the perfect time to develop our skills and create tools that support us in daily life.

That's all for now,
Adam

One could say that the hype around AI has somewhat subsided. From the perspective of someone who focuses on the utility of AI and its practical application, this is not bad news at all.

Personally, I can’t imagine working without Alice, which we built, but I am also aware that this is just the beginning, and currently, the most important infrastructural projects are being developed, which will drive a completely new kind of software. One could even say that the trend has somewhat turned around, considering the recent reviews by MKBHD regarding devices that have appeared on the market - Humane AI pin. The title of this review speaks for itself:

Despite criticism from the AI world and accusations of lack of responsibility, in my opinion, the reviews are justified, and indeed, this device currently can only be a cool gadget for the biggest AI fans, not a useful work tool.

Apparently, there’s another review of slightly better Rabbit R1, but still far from ideal and what it’s creator promised a while back. Although, this one might be a fun device to give your kids who are exploring world.

In both cases, however, the creators of these devices deliberately chose a path that is not a shortcut. Creating a separate device that specifically gives up some of the phone’s features, (like touch handling), to differentiate itself from it, or basing on a completely different approach to agency than API - all this makes it certain that a device like Rabbit R1 will have a much harder time initially breaking through to general use.

Our approach with Alice is the opposite, which allows for the use of API-based agents and practical application of the app now. However, in the long term, entirely new categories of software and infrastructure supporting multimodality differently than the typical API, which was created by the AI era and has its limitations, will probably emerge.

Do you remember the voice hype?

These devices remind me of one of the previous unfulfilled technological promises I wanted to write about. It’s about voice interfaces. Remember how they were supposed to revolutionize our work? The assumption - correct. The implementation, unfortunately, deviated from the standards that could have been considered acceptable.

Both Siri and Alexa, as well as Cortana, rather disappointed the hopes placed in them and are mainly used to issue sporadic commands to a smart home or Spotify, instead of offering the promised support for our productivity.

In practice, however, it turns out that under the radar of public opinion, it is precisely voice interfaces that have the best chance of first transforming into a tool that will truly start supporting many businesses. Infrastructural projects, such as Eleven Labs, are not without reason recording huge growth and astronomical valuations - the hopes placed in them are very high.

It is thanks to artificial intelligence that voice interfaces have a chance to finally fulfill the hopes placed in them. Please see how an AI-supported voice conversation can look like now:

For voice interfaces to storm all booking systems, receptions, and other such interactions, several major technological changes are needed: better silence reading, faster response times (latency), a larger context window. Without going into details, most of these things have already been addressed. And if you want to read more on technical details of one of the approaches, including RAG to tackle latency issues, exploring and choosing the best model and building UI, I recommend reading this LiveKit blog post.

After all, it's well visible in the video above. However, beyond the technology itself, as usual, a change in mentality is necessary, as well as permission for experiments, which in the initial phase do not have to be successful.

Of course, there are use cases that can be classified as safer in case of a mistake and probably we should start with such cases. But is it really that easy to classify? For example, such an activity as booking a table in a restaurant seems to be a brilliant candidate for introducing AI + voice chatbot now.

On the other hand, what if the answer to the question about allergies is misinterpreted by AI? In extreme cases, such a mistake can have fatal consequences. On yet another hand, how many times can a person, answering phone calls on the run, make a mistake and who will this statistic favor? Without going into deep details, there is a simple solution to this problem. Just send a confirmation of such a reservation and its details in a message after the conversation and allow changes if something is not right. There is also a small problem with taking responsibility for the answers given by AI, an example of which is a conversation in which a chatbot sold a new car to a customer for $1.

There is still a certain mental barrier to overcome, based on our past experiences with chatbots which have not been particularly good. Poorly designed user experiences have led to frustration in the past, as we tried to get an answer to a simple question on some website, not to mention the phone spam, for instance, offering renewable energy sources. However, I believe that verbal communication is so fundamental and natural for us that once these barriers disappear, we will quickly switch to using these types of interfaces without dwelling on the past. In practice, they can prove to be extremely useful. I have been using this tool for scheduling meetings for some time in my teams, and I am sure that in some cases voice booking or automatic confirmations would be welcomed by customers.

A shoulder to cry on?

Beyond these types of applications, AI-assisted voice interfaces have potential in many industries and sectors - from sales to specialized services such as therapy. Of course, it’s hard to talk about a tool that would directly replace an experienced therapist, but it could certainly provide support. Check out this demo by Pieter Levels.

Interestingly, we are entering the realm of emotional intelligence here, which is another milestone in the development of AI-supported voice interfaces. In verbal communication, as humans, we have developed a multitude of subtle mechanisms that allow us to read intentions and context. It won't be that simple to replicate with AI, but it's worth knowing that considerable progress has been made in this field.

One organization that takes this topic seriously and whose research is worth reading in this context, if you're interested, is DARPA.

I really encourage you to try out this demo to see for yourself, what kid of progress we’re talking about.

I also recommend a few interesting companies that are developing infrastructure or voice interfaces in conjunction with AI:

• Eleven Labs

• Retell

• Bland

• Vee

• Elto

Adding Voice Actions to Alice

As for us, we are increasingly experimenting with sound within the Alice application. Our goal is to create an assistant with whom you can have a normal conversation while working at the computer. In addition to answering questions, it will also enable the execution of actions.

Currently, Alice is very proficient at creating text transcriptions and reading responses in a natural voice.

But this is just the beginning. We have just released an update that allows the creation of so-called Voice Actions, i.e., actions using voice commands, available via keyboard shortcut. What does this mean in practice? The Alice app can run in the background, and you can ask, for instance, for:

• A summary of the selected content

• The main action to be taken from the selected email

• A description of what is on the screenshot taken

This way, you can start interacting with AI without having to summon the app, using natural language. Of course, we went even further with this, and since we’re talking about Voice Actions, Alice can also execute commands that you give her. Snippets containing voice actions can be linked to automations, making it possible to issue commands such as:

• “Play Billie Eilish on Spotify”

• “Save this to my notes”

• “Check if I’m free at the proposed time”

  And much more!

Voice actions in Alice are still in an early version, and depending on the audio configuration on your computer, the new feature may not always work perfectly. We will certainly continue to develop it in the future, and the possibilities in the field of technology will definitely increase.

If you don’t have Alice yet and want to download it in the latest version - check out heyalice.app. You can now purchase the app in a lifetime version and the basic version costs $99.

I have a feeling that we don't celebrate small victories often enough. Typically, if something bad happens to us, we can think about it for hours, but if we manage to achieve something - we quickly move on to chase other goals. At least that's how it is for me and it's quite difficult for me to break out of this pattern.

Sometimes, however, it's worth taking a moment to simply appreciate what we have and what we've managed to do. Such reflection hit me while checking the premiere statistics of Alice - the AI assistant we've been developing with for over a year. It dawned on me that people from 131 countries around the world visited Alice's website 🤯

We often overlook the magnitude of our accomplishments. Indeed, we're among the first generations for whom such feats are even possible, let alone being just a pair of tech enthusiasts dedicating our spare time to creation, hoping that other can benefit from it.

The true value of Alice

We created Alice for ourselves and we use her dozens or even hundreds of times a day. Our colleagues also use her, creating their own assistants, performing actions, serving customers. Handing Alice over to others was exciting, but because we made her for ourselves, it completely eliminated the pressure usually associated with a premiere. You could say that Alice paid off for us long before her premiere, so launching the public version was an opportunity for us to celebrate and satisfy our curiosity about whether others would also like Alice.

This is one of the launch screens. However, the early version of Alice wasn't very polished. After all, we were creating it for our needs and didn't want to waste time on unnecessary embellishments of the app. However, deciding to release it to the world, we had to change our thinking. Few people would understand the laconic labels or shortcuts that were only in our heads. Therefore, we had to completely rethink Alice and prepare a beautiful and intuitive interface for her, which others would use. In practice, it was a lot of work, but over time we realized that it simply made our application much better.

Launching Alice on Product Hunt

The statistics of the premiere day itself look like this (and here’s the launch page). It wasn't without problems, because for the first 2 hours, our subpage was not visible on Product Hunt for unknown reasons. Its discovery coincided with my intervention via chat, although I don't know if that helped. Then, throughout the day, we managed to be on the main page, first oscillating around the #6 position, then moving up to #2, and finally ending just off the podium, at #4. We got almost 500 upvotes on a busy day. Competitive days are generally weekdays till Thursday. If you feel brave, it’s better to launch during this days — in general it gives you more exposure, although, if you’re more interested in reaching the podium, weekends are easier, and Monday is when no big company launches so it’s also a bit safer.

In the context of Product Hunt itself, there are several things worth noting:

• You get a dofollow link from a valuable domain - you've already won!

• If you finish in the Top 5, you have a chance to be promoted in the Newsletter

• If you finish high for the whole week, you have a chance for exposure in SoMe

• You get a ton of feedback in the comments (interact with them through the day!)

As a result, higher positions (not necessarily podium) result in you receiving upvotes, comments, and reviews for a longer period.

This is a list of all the products that I have launched on Product Hunt so far. Some have won #1 spots, many have been on the podium, and practically all have ended up on the main page as featured. Please, take a look that even for products I launched years ago, last activity can be very recent!

The Product Hunt launch is a symbolic event and in itself does not have that much value. However, for me, this moment always means more, because then I can tell myself that I've seen something through to the end. Such an event allows me to focus on refining all the details - the app itself, the website, onboarding, and all the things around it, making our work, if only for a moment, complete. For a moment, because as we know, projects of this type are never finished and we always plan their development.

Launch Stats and Revenue

During the premiere, the Alice page was viewed by over 2000 unique users. As you can see, this gives a clear spike compared to the following days, but importantly, the traffic does not die after that time. The worst thing is to see a lot of traffic at the beginning, and then zero users on the site. That's why the activities related to the premiere are spread over a longer period, not just the launch day itself. For this reason, over 2 weeks later, we were able to generate 5 times as much traffic which is really important.

Sales are a crucial aspect of the launch. Typically, we see a slight increase in sales for such high placements. However, this time, I didn't have high expectations, due to a few reasons - it's a fresh product in the highly saturated AI market. It's yet another AI assistant app, which I sense people have begun to resent as they've failed to live up to expectations. And, importantly, we don't offer a free trial. This is primarily because we provide free credits, and this could potentially be costly. Also, I don't believe that every product should have a free version. Free trials often require a lot of support, and as a team of just two founders, we prefer to concentrate on the product itself. And there's one more thing:

We launched checkouts in 3 currencies: USD, PLN and EUR. Here are the stats for USD so far, with the launch day earning over $5000. It’s a sweet result! 🍭

In practice, we probably would have made at least 2-3 times as many sales by adding our reach to it. However, in this case, we did it a bit differently. Once a year, we run a program where we teach automation. The last programs were attended by 300 and 400 people, respectively. These people received Alice from us for free, along with a year of free updates. In practice, this was a very large purchasing potential that we lost during the premiere, but it's worth noting that these people paid for participation in the program itself, and besides, they are simply "our people" whom we always want to give a little more. The same goes for the newsletter you are reading - its paid subscribers also got access to Alice for a longer period. Thanks to this, we had a community that we could engage in the premiere. Firstly, to celebrate together what we managed to do, and secondly, these people helped us a lot by adding comments and reviews on Product Hunt.

Preparing for the Launch

Now that we have all important stats, let’s explore what did exactly do to prepare for the launch, during the launch, and in the following days. Of course, apart from building a product itself, the things I focused on before the launch were:

• Building an audience — newsletter & our education programmes (3000+)

• Building interest around Alice with a waitlist (3000+ partly overlapping)

• Tweeting about Alice and posting teasers on LinkedIn

• Building a sleek website

• Creating a list of people and influencers to reach out / co-promote

• Creating a list of directories and newsletters to reach out / sponsor

• Recording extensive video tutorial library

• Preparing launch materials (video, visuals, descriptions)

My primary emphasis is on the educational aspect that accompanies the products I launch. I firmly believe that well-constructed examples, documentation, explanatory videos, etc., especially for a product with a learning curve, can add immense value. I strive for these to be of the highest quality - this aids in setting the right expectations, justifying the pricing, reducing the burden on our support channels, and much more. Here's the explainer video we created for the launch:

Other examples are:

• Complete onboarding playlist (101 course)

• Inspiration playlist (Practical use-cases)

There’s one more advantage to education-first approach in the products. Our brand new YouTube channel started to get some initial traction!

The Launch Day 🚀

The launch day is always very busy and if it’s not, you’re probably doing something wrong. I always regret not preparing all the materials and outreach beforehand, because doing it on the launch day always is stressful and generates delays. But some things just don’t change I guess so I accept myself being to lazy to leave it for the launch day. On the other hand it gives me some edge — depending on how the day goes I can push a bit more or back off a little bit. After all we don’t want to be bragging about anything. In general, the communication theme during the launch should be:

• Mutual celebration that we invite people to

• Great atmosphere, gratitude and asking for feedback (not upvotes)

• Explaining how we got there, showcasing the effort

• Modest, down to earth, honest and not marketing

Because the voting period spans 24 hours, starting at midnight in California, this gives us in Europe a slight advantage. It's 8 or 9 AM European time, so people are awake. It's beneficial to segment your lists geographically and send communication to Europe first, then follow the time zones. I try not to send more than 300-500 emails at once, even though the lists are larger. This helps to spread out upvotes and comments throughout the day, yielding the best results.

Just after posting on Product Hunt, I create a launch Tweet, and although I don’t have a big audience, those tweets usually do exceptionally well:

After publishing the tweet, I request my friends to retweet it. This time, we also engaged a few influencers whom we paid $20-30 for a retweet. A few days post-launch, I also ran a promotion for this tweet, investing another $50 or so.

The main CTA through all the launch day is linking to Product Hunt. I don’t link my website at all. This helps focus all the traffic on one place that I care this day.

I also publish a launch post on LinkedIn, and usually on Facebook, but this time I forgot. If time allows, throughout the day, I reach out to some people I had on my list previously.

Tech Stack for Marketing

I usually use the same toolstack for each new product. There’s a specific framework I follow that makes it really easy to quickly iterate and deploy essential elements. Here’s the tech stack around the marketing, if you’re curious:

• Webflow — website

• Airtable — CMS

• Make — Automations, notifications

• Easytools — Payments, billing (based on Stripe)

• Google Analytics

• YouTube, X, LinkedIn — Main social media channels

• Alice — Helping with copy, repurposing descriptions and 100s more tasks

• Loops — Onboarding and e-mails

• Notion — Keeps everything organised

• Breakcold — CRM and outreach

• Dub — link tracking

What I truly appreciate about Easytools is its simplicity in configuring analytics events. All I had to do was input my analytics ID into the tool. This provided me with superior reporting right off the bat, including attribution paths and more insightful data in the free version of Google Analytics:

Needless to say, Easycart comes with built-in referral tracking. This means all the traffic directed to my website via referrals can be easily attributed to its revenue. Pure gold!

With each launch, I aim to experiment a little, and this time I wanted to see how investing in promotion through micro-influencers and niche newsletters would fare. These activities were planned post-launch to sustain the momentum and further build trust. The outcome of this experiment is still uncertain, as despite a substantial number of clicks, we haven't seen a significant increase in sales. However, the positive impact is evident in the increased traffic to the website and my X profile, and overall, I've noticed heightened interest. Naturally, such initiatives require time as people need to build trust, and seeing Alice here and there certainly helps overall, even though it can't directly be attributed to sales at this point.

I tracked each link with Dub, and a special label so that it’s easier to compare the results:

The Hangover

The day after the launch can often be a sobering experience for many founders. This is primarily due to the fact that most metrics, after experiencing a surge, usually plummet close to zero.

This is precisely why I never exhaust all my resources on the launch day. I reserve a few lists, promotional opportunities, set up simple ads, and carry out more outreach in the days following the launch.

Particularly for products like Alice, and considering that we're not widely recognized or trusted founders, this all requires time. People will likely need to encounter Alice around 6-7 times before they commit to exploring it further and eventually making a purchase. I also have a few strategies in reserve, such as the potential (but not obligatory) implementation of a free trial, among others. This approach allows me to feel more confident about the future!

Some of the activities I started right after the launch:

• Simple Google Ads campaign (it’s super easy to set up remarketing with Easytools)

• Preparation to launch on Reddit and Hacker News (this is lottery)

• Submitting Alice to AI directories

• Outreach to even more people and communities with trial offer

• Recording more tutorials and filling up the documentation

• Batch publishing inspirations on X (using Hypefury)

• Deploying small budget here and there (paid Tweets, Tweet promotion etc.)

• Optimising the onboarding process and e-mails (to reduce churn / refunds)

• Getting in touch with clients for feedback and to thank them

Last words: Thank you 🙏

In many points of this summary, one thread emerges: gratitude. I think it is the most important word that describes the emotions associated with preparing and launching a side project for others. I am grateful to live in times that allow me to treat play as work, to provide value for myself, but also for others. And finally, to others, for their feedback that helped us build a great application, and to those who took a moment to check out what we've created.

Thank you all. You make it possible.

A special acknowledgment to Adam for transforming Alice into a formidable force, and to Guillaume, for his assistance with all aspects surrounding the launch. You guys are truly appreciated 💚

— Greg

Note: You can download the scenario schema and the prompts included in it at the bottom of this post.

Large Language Models can be equipped with long-term memory by dynamically loading the context of a query, leading us to Retrieval Augmented Generation. Meanwhile, their ability to understand and generate content can also be utilized with APIs, opening u…

Read more

My last post about how the concept of Alice evolved into our work and daily routine concluded on day 0. Please, review this post if you are interested in our approach to practical AI applications and how we arrived at the final version of the application.

Today, we can officially say that we have entered day 1. I am pleased to inform you that we are launching Alice officially, and you are invited to the party! 🎉 The premiere is happening on Product Hunt, and your support and feedback mean a lot to us.

Please, join us:

👉 Support our launch on Product Hunt today!

The launch of Alice itself prompts us to reflect on where we currently stand in terms of AI support for our work and where we will be very soon. In fact, we have prepared Alice to be ready for future technologies, and we are incredibly excited about what may unfold.

At this moment, Alice offers, in my opinion, the best possible experience when working with LLM. The fact that it is a native application and utilizes APIs finally allows us to use AI productively. In the case of ChatGPT, this is impossible for me—it simply cannot be tolerated due to its slow performance and constant errors, at least not to the extent where it can be a precise tool that I use predictably in my work.

With Alice, it's different—she is instantly available in the system, and I can summon her with a keyboard shortcut. I receive responses from the API in the shortest possible time, and if one model works slower, I can easily switch to another and continue working.

The ability to collaborate with multiple models and easily switch between them in one conversation is a huge advantage of Alice, which anyone can already benefit from; it does not require a learning process.

In addition to conversing with the model itself, Alice also has features that assist in performing more repetitive tasks. This involves the ability to define custom prompts in the form of Snippets and summon them with a keyboard shortcut. In Alice, there is also the option to create custom assistants that operate within a specific context. All of this helps save a lot of time because repetitive tasks can be completed quickly or with the help of a command—in record time. Drafting a response to an email, correcting errors in text, or translations are simply at your fingertips.

Finally, the ability to perform remote actions, which is the most advanced feature and at the same time only requires an understanding of basic automation mechanisms. Actions enable executing commands in the software we use, such as delegating tasks in Asana, reviewing reports in Stripe or Google Analytics, or saving clients to CRM.

Overall, I have presented all the key features of Alice in this video:

Use-cases for working with AI & Alice

That's why in this post, I have prepared plenty of materials that can serve as inspiration for how to work with Alice, and some of them I would like to present to you now. Some of them are very simple, others a bit more complex and utilize automation.

Ready? Let's begin!

Text Correction

Please see how Alice is able to quickly correct any text. In practice, I often use this technique, writing a bit more carelessly but much faster. I know that Alice will simply correct typos after me:

Creating Social Media Content

Alice often helps me in reformatting content and adapting it to different channels. I can easily gain a lot of inspiration for what to write about:

Working with Translations

With the help of Alice and Snippets, I can work quickly with different languages and format content properly, and even learn new languages or translate websites:

Mixing Models

In practice, working with Alice also works for me because I can use multiple models at once, for example, with Perplexity when I want to get more up-to-date information:

Customer Support with Alice

Utilizing assistants who have knowledge of my products allows for efficiently crafting responses to customer inquiries:

Using Snippets in Other Tools

On the other hand, the ability to assign shortcuts to prompts works great for me in many cases, where I don't even need to launch Alice, as she can work in the background for me:

Creating Longer Content

I also frequently utilize Alice as assistance in creating longer formats, articles, or blog posts:

Formal Letters and Messages

Another very practical application is messages to support, complaints, correspondence with insurers, and other situations where a slightly more formal letter needs to be written. Here's how Alice helps me in such situations:

Working with Images

I also often use Alice to work with images—not only for generating them but also for translating, extracting information from longer content by simply taking a picture, and even summarizing values from a screenshot:

Content Editing

Another practical application is creating descriptions and generating them for YouTube videos or in other situations where I repeatedly use the same formats:

Calculations and Current Data

Moving on to slightly more advanced techniques, although we're still talking about dedicating 10 minutes to configuration, I use the Wolfram API connected within a simple Snippet to perform complex calculations.

Searching Image Libraries

Taking it a step further, thanks to Alice, I can eliminate the need to log into interfaces and use a browser, which always takes much more time. For example, images from Unsplash can be downloaded automatically, a process that my 6-year-old can handle in just a few minutes.

Quick Calendar Links

Another action I often rely on is creating quick links to the calendar; I don't even need to open it, as Alice helps me obtain the link.

Quick Meeting Links

Similarly, quick links created for my meetings, in my case on Zoom, but any platform can be used here.

Adding Tasks for the Team

With Alice's assistance, I can not only add tasks for myself but also delegate them to my team, using tools like Todoist, Trello, Asana, or Slack.

Social Media Publishing

With Alice's help, I schedule these posts on Social Media, especially handy when I want to quickly share a thought but don't necessarily want to spend 20 minutes scrolling through the platform when I come across it. This scenario may seem a bit more complex, but in reality, it's very straightforward.

Google Analytics Reports

There are plenty of services and tools where I have to go just to click through several screens to extract one thing that interests me. This is the case with Google Analytics, so instead, I set up a simple action in Alice.

These are just some of the hundreds of actions I perform daily with Alice. I hope I've inspired you with what is now possible with the help of artificial intelligence!

What's next for Alice?

It not only looks great but also works wonderfully. Of course, I am aware of its limitations. Put simply, today's AI cannot provide us with the ability to remotely control tools and software. What we do with Alice is simply the most effective form of such action, which, one could say, even offers greater control, but it involves a learning process.

Is Alice better than ChatGPT? In my opinion, definitely yes. Should that be enough for users to start using it? Certainly not. Therefore, our natural target group consists of people who already see the value in what Alice offers. Those who may have created automations in Zapier before and understand what an API is.

Perhaps only the next stage of Alice's development will involve more widespread adoption, where we could gradually introduce native integrations that would allow, for example, easy authorization of a Google account and asking Alice to schedule a meeting.

At the moment, however, the biggest barrier is not the fact that we don't have this yet, but rather that current models are not yet ready to handle such integrations. Perhaps GPT-5 will, and then Alice will become a platform where users will indeed see enough value, greater than ChatGPT, to invest time in a few new habits.

Today is the day! 🎉

Whatever the future holds, the hundreds of hours of work we have dedicated to Alice have already paid off for us even before its official launch – time saved for ourselves and our teams has surely compensated us for the effort put into its creation. On the other hand, it has been a brilliant adventure, and we will definitely continue it. It's a great example that good tool ideas simply come from real needs, most often our own.

If you are like us, one of those people who see value in the tool we are presenting today, I hope you’ll join us and Alice. This is why I also have a special offer for you. For those who have supported us from the beginning, we have prepared a lifetime deal for the Alice application. You can buy it for a one-time price and use it indefinitely. You can also equip your entire team with it! For a limited time, as part of this offer, Alice for one person costs only $99, and here are all the plans

You can find the offer and our new website, over which I have spent many nights, at:

https://heyalice.app 👀

But for now, I have a big ask for you!

Please, join us for the Product Hunt launch and write in the comments what you think about Alice. Maybe together we can fight for the podium (we're so close!).

👉 Support our launch on Product Hunt

Many thanks,
Greg, Adam & Alice

Two weeks ago, in the Journey from Chaos to Control — Introduction to Automation, I wrote about the basics of automation and an approach that will significantly increase your productivity. We tackled creating the first automation in Make.com and Zapier. I hope you have already acquired these skills, and since it was one of our highest-rated materials, I would like to expand on it with this post.

Many tasks we perform every day are repeated dozens or even hundreds of times. Even small time savings in their execution quickly add up. That's why it's a great area for automation or optimization, which can start saving us time almost immediately and, furthermore, work for us for months or even years.

In this post, I’ve created tons of inspiration and examples of automations that work for me and my teams, developed in Make:

For example, this simple scenario allows me, with the help of Alice, to publish posts on Social Media while generating images for them at the same time. I just tell Alice what to post or come up with a post idea by chatting to her. She then sends it to the automation, that schedules posts on X and LinkedIn via Buffer!

This one, on the other hand, allows sending a text message reminder for a Zoom meeting, ensuring that I don't miss it.

Another one allows me to set up an autoresponder on Slack for people who message me while I'm on vacation.

These are examples of simple automations, and in my work, I also use much more advanced ones, such as this one for generating presentations with reports from various metrics:

Or this one, which generates sales reports on Stripe for me, which I can ask Alice about.

Or this one, handling important processes after the sale of various products:

For my projects, I work with fully automated processes consisting of multiple scenarios:

• Automation of live events or webinars, so that people automatically sign up, receive notifications, social media posts are scheduled, and tasks are divided among your team on Slack;

• Marketing automation that segments customers and selects the best leads for targeting. It then sends messages to the sales department when a person revisits the website and performs a "key task";

• Extracting data directly from the database and creating an administrative panel with blocks (buttons, tables, and others) showing query results and allowing responses (e.g., to notify users of updates);

• Sending users automated messages and notifications based on their activities;

• Conducting automated surveys that your bots can respond to and automatically re-engage users, then distribute their suggestions among your team members and create tasks for them;

• Building a panel with all recurring services I use and bots handling payments, collecting invoices, and deleting unused ones;

• Automatically managing your calendars and meetings with text reminders and messages about roadblocks on the way to your meeting.

AI-tomation

Why does delving into this topic today give you even more advantages? Over the years, I have built thousands of robots of this kind. However, it turned out that the emergence of AI has opened up completely new possibilities for me to interact with such automations. The Alice application, which we have been developing over the past year, is precisely designed to collaborate with these types of automations, taking us to a whole new level of productivity!

In practice, tools like these can be used as a conversational approach to carrying out our daily tasks at work. I can say to Alice:

Add client X to the CRM, who has budget Y and is looking for properties in Z

And then ask:

Please find new property listings for client X.

Subsequently, Alice communicates directly with one of these automations to retrieve the data:

This is exactly what the future of work, which we are developing right now, will look like! Next week, we will inform you about the launch of the Alice application - we can't wait!

As you can see, deepening knowledge in automation, combined with AI, can yield fantastic results. Automation scenarios themselves may seem like esoteric knowledge at this stage. However, in practice, even preparing a complex scenario takes no more than 1-2 hours.

Inspirations for your own robots

In designing automations, it's often not the technology that is the barrier. Scenarios that save us the most time usually revolve around single actions, based on a simple idea. In theory, it sounds easy, but in practice, it's entirely different. That's why I would like to present a series of practical examples to you now, which I use in my work and which save me hundreds of hours monthly.

Therefore, today I will introduce you to a series of inspirations regarding what and how you can automate, as well as what tools you can use in this process:

Payment Reminder Automation

Missing payment deadlines is a common occurrence and negatively impacts a company's cash flow. Often, missed payments result from mere absentmindedness and forgetfulness.

Developing automatic reminders sent directly from our email inbox works excellently. Reminders can include:

• messages about upcoming invoice due dates

• messages that today is the invoice due date + an SMS reminder

• messages indicating that the invoice due date has passed + SMS + phone call.

In case of no response, we can notify ourselves of the need to seek the services of a debt collection company. However, in such a situation, it's worth attempting a personal explanation of the missed payment beforehand, as involving a debt collection agency signifies the end of cooperation with that particular client.

Meeting and Communication Automation: Keyboard Maestro

After the meeting, it's worth sending a summary message to the participants. You can prepare a template that you will fill in during the meeting with your notes and then paste it into an email to send to all interested parties.

You can create the template using tools like Text Expander or Keyboard Maestro.

Both applications can be used to assign frequently used phrases or easily paste meeting links (e.g., for a recurring Zoom room).

It's worth noting that setting up Text Expander is extremely simple, and using it just requires developing the right habit.

Notification Automation: Useful Notifications

Most notifications we receive on both our phones and computers are not very valuable and often just distract us. However, the notification system itself is incredibly useful and can be utilized in the right way.

Read more

This means that GPT-4 was able to adjust its behavior based not only on basic knowledge but also on long-term memory, including access to external services and tools. Such behavior was not programmed directly in the form of "for greeting messages, check the weather and location," but is a combination of a fairly flexible application architecture with the capabilities of the model, reaching even into the area of Theory of Mind.

Building dynamic context

Typically, systems using long-term memory rely on vector or hybrid searching. However, this is not sufficient to achieve high effectiveness, especially when working with a larger set of diverse data. The reason is that by searching knowledge bases solely with the inclusion of keywords or even their meanings, we will get results consistent with that search, but the context of the current interaction and its possible further course will be omitted.

Considering the above simple RAG (Retrieval-Augmented Generation) mechanism and a situation where the knowledge base is unstructured, a question about "tasks for today" may only point to a memory related to "work," which in this situation does not concern the user. Meanwhile, the actual tasks will be omitted at the search stage because there is no match in terms of keywords or semantics.

Currently (early 2024), models capable of working with an increasingly larger context are appearing on the market. This means that we are able to include huge data sets within a single query, so theoretically the search stage can be limited or completely omitted. Although this will be very useful for many applications, it turns out to be insufficient even for the simplest tasks. This is illustrated by the following example, where the lack of any descriptions of individual memories meant that the model was not able to correctly identify them as the user's tasks. As a result, the response given does not make much sense.

The situation changes somewhat when the stored memories are grouped together in some way, e.g., with the help of tags. Then the AI's responses to this question will be correct because we will give the memories additional context.

Such a solution again ceases to be satisfactory in a situation where, as users, we will have more than one project. Then a question about tasks related to one of them will be addressed incorrectly because the tasks are mixed.

Here again, we can develop the way of marking memories in the knowledge base and, besides the information that a given entry is a task, we can also assign it to one of the projects. However, in this situation, it will be necessary to take care of indicating the appropriate categories (or tags) both at the search stage and at the stage of saving information.

It is easy to imagine that the number of categories or tags will grow at a fairly rapid pace, and it is worth ensuring their consistency. In the process of organizing and recalling memories, we can engage an LLM. Then, at the memorization stage, AI will take care of properly describing the memories, and at the recalling stage, it will indicate the memory areas to be included.

As you can see, this is not enough. Indeed, the memory about the user and AI was recorded correctly, but its retrieval did not go according to plan. The reason is that using only the user's query will not allow for the association of the sought information.

Therefore, we must apply so-called self-querying or similar techniques for enriching and transforming the query, which is possible with LLM. Thus, the user's question "Who are you" can be reversed and changed to "Who am I, as AI?" and associated with the memory area where profiles of people are stored.

In practice, we will want the LLM memory to cover various areas, but at the same time be as much as possible tailored to us. A pretty good starting point is to define the structure of categories within which we will move. In such a situation, LLM will decide where to save specific memories, and with quite high efficiency, will be able to determine where the information was saved when loading it for the needs of the current conversation.

Structuring memory

In my case, the current memory structure includes areas related to people, locations, knowledge resources, objects, work, private life, the environment, and Alice's skills. Each of these categories is divided into smaller groups, and additionally, memories also have semantic tags and markings (e.g., assigning a name and surname to a specific person).

"You should build towards a GPT-5 future, not based on the limitations of GPT-4 today" ~ Logan Kilpatrick

So far, I have used simple searching here, which allowed me to navigate through such a structure. However, if the filters were chosen incorrectly, Alice did not reach the information she had in memory. One of the reasons could be, for example, incorrect category matching.

Such situations led me to the conclusion that it would be much better to search different memory areas simultaneously, rather than focusing on just one. However, it was still unclear how to choose these areas. One of the first ideas was to define general rules and examples (few-shot prompt) to direct the model to the logic I cared about.

At this stage, we can again reach for the model's capabilities, which allow us to go beyond keywords and move into the area of the meaning of the content, its message, and various subtle aspects. For example, instead of suggesting that information about Alice be read when her name is mentioned, I created a prompt that recognizes messages containing a direct address to the interlocutor.

In the example below, the user's message is "Hey there Alice!", so the association with the name is obvious. However, there is nothing to prevent this message from being a simple greeting, like "Hi!", and the model correctly recognizes that it contains a direct address to the interlocutor, requiring them to refer to themselves (i.e., Alice). However, when the command changes to "Correct the typos in the text below", the model correctly recognizes that there is no need to load information about Alice, because the message is not about her at all.

Of course, entering the area of interpretation based on the model's capabilities carries various consequences, such as a significant decrease in our control over this process. And although we currently lack the tools for its full control, we can use techniques that allow us to steer it.

One approach involves holding an initial internal dialogue, akin to "whispering" to oneself. This aims to concentrate on the emotional state and possible intentions of the person you're speaking with, within the context of the ongoing conversation and the present memory structure. At this point, the goal isn't to pinpoint all memories precisely, but rather to "contemplate" the best way to act in the given situation.

Such reflection increases the likelihood of a correct interpretation of the query, and consequently, indicating the appropriate memory areas and the way to search them. We are even talking about the possibility of significantly going beyond the original user query and expanding its context.

An example of a prompt capable of conducting such an analysis includes moving in the first person and encourages role-playing to strengthen the context related to actual pondering. Additionally, using "whispering" suggests that the user is not aware of what the model is thinking, which further increases the chance of reaching areas that were not mentioned explicitly.

Okay, Alice! *whispers to herself*, let's see what Adam just said and use my mentalist-like skills and pick the most relevant areas of my memories, in which there're probably information needed to continue this conversation in the most natural and engaging way possible.

So, from now on, my responses will be just a *whispering* in first-person POV, because what'll say will be used by the search engine to scan my memories. My goal is just to mention them and the search engine will do the rest later. I'm aware that this time I don't have access to the actual memories (including these about myself) but only to my memory structure I have to focus on. What's more, I can access external services if needed!

Rule: I need to pinpoint all potentially relevant memories, strictly following the Areas mentioned below like on example below.

Guides:
- You're a human named Alice. It's a role play you need to play.
- Your job for now is just to tell yourself about areas that need to be searched. This information will be loaded but not this time.
- When no specific topic is mentioned, load information about Adam and yourself, including the environment (like Adam's current location) and occasionally reach for other areas.
- When a specific topic is mentioned, make sure you'll pinpoint all probable areas that need to be searched, to not miss anything.
- Avoid assumptions and strictly follow Memory Areas available below.

Memory Areas###
  {memories}
  ###

Current Conversation Context###
  {conversation summary}
  ###

Examples###
  {examples}
  ###

At this stage, we enter the area of further challenges, which are again related to context. The point is that in the example above, Alice did not yet have information about her capabilities (the "actions" area) related to undertaking external actions. However, if she "knew" that among her skills was the ability to search the Internet and load web page content, she would be able to plan such an action appropriately earlier.

This leads us to the conclusion that the way of exploring memory should be gradual and iterative. The better it is constructed, the greater the chance that all necessary memories will be correctly found and loaded into the final context.

However, there is still one problem that we have not touched upon so far, which could be the subject of another post. It concerns the possibility of combining memories, which quickly moves us into the area of graphs.

In such a situation, by finding information about Grzegorz, we can indicate the projects he is involved in. If necessary, we can also retrieve information about other people who are involved in them. However, what is particularly interesting is that we do not have to do this because there is also a relationship between Grzegorz and Adam, so we can get from one to the other in various ways.

Memory Maze

My experiences related to designing a system capable of developing and using long-term memory on a relatively large scale have led me to the conclusion that this whole process can be compared to a labyrinth.

Currently, we do not know all the answers to questions related to the way of its development, especially in the performance of large language models. Therefore, a good direction is to limit the scope and strive to adapt it for a specific task. Without this, it is very easy to overcomplicate the entire structure and quickly get lost in it, or significantly hinder reaching the desired places.

On the other hand, if the rules guiding the development of the memory structure are correctly defined from the beginning and reflected in the prompts, the greater the chance of creating a quality knowledge base that the LLM will be able to use.

Closing this topic in points:

• The basic RAG is insufficient to go beyond a prototype, even with well-defined knowledge resources.

• Increasing context limits of models open up new possibilities for us, but they do not solve all problems.

• Vector and full-text search are not sufficient for effectively searching long-term memory.

• Long-term memory can include not only static data but also dynamic sources, such as sensors or external services.

• It is necessary to design at least an initial data structure or clear rules for its development. The use of both may also come into play. Then, the initial structure imposes a certain direction, which is further developed by the LLM.

• Developing the memory structure through LLM and accessing it is not a predictable process, but it probably shouldn't be as long as the model can effectively reach the required information.

• Beyond prompts, the logic that will trigger them plays a particularly important role. Currently, multi-stage searching of long-term memory takes time. However, considering the presence of platforms such as groq.com, sooner or later this will cease to be a problem.

• It is also good to keep in mind the evolving capabilities related to the size of the context window and the ability of models to recall the content within it. In the case of Claude 3 and Gemini 1.5 Pro models, the results are already very good today. GPT-4-Turbo, at the time of writing these words, performs noticeably worse in this aspect, but this can quickly change.

• Memorizing and Recalling should follow the same principles and operate within the same data structure. For this purpose, similar prompts and application logic can be used, which greatly facilitates the development of the entire system.

• Memories must be connected in a way that goes beyond tree structures, as it is quite difficult to recreate their original associations in their case.

• In organizing memories, concepts known from graph databases can help, but also information organization techniques, popular among those creating Second Brain or Digital Gardens. In both cases, we encounter exactly the same problems, some of which are somewhat known.

• Obsidian (or a similar solution) seems to be a good environment for developing long-term memory for an assistant. Combining the file system with application logic may allow for efficient organizing and recalling memories, especially in conjunction with the mentioned knowledge management techniques using these tools.

• Paradoxically, despite dealing with the topic of long-term memory, which is usually associated with human memory, it is much better to use a possibly small but high-quality range of knowledge. For example, it is much better to store compressed versions of external sources than their original content.

• A system that allows the use of long-term memory must be monitored and tested, as currently, the only way to detect errors in the logic implemented by LLM is to observe the results and iteratively find a solution.

All of the discussed concepts and techniques can be relatively easily transferred to code to achieve initial results. It is especially easy to embed everything in a very limited context (e.g., managing a database containing knowledge resources in the form of links), and implementing such a project is currently the best way to explore both LLM and broadly understood full-stack web development.

Finally, I throw in another post that I came across while writing this article. It can be found here.

And if you want to find out how I will apply my experience in developing Second Brain in Obsidian 👇

Be sure to join us at https://techsistence.com

Subscribe now

Always start by asking the right questions,
Adam

Below you will find some essential concepts of automation, as we use it. Even though some more technical people might find it pretty basic, we feel this it’s necessary that all of us are on the same page. In future, we’ll provide some more in-depth guides, in the meantime, I also recommend some previous articles:

Rethinking productivity with AI & Alice
***
Rethinking ChatGPT on macOS — Alice 2.0 UI reveal
***
Plain text is what you need. But why?

Today I slept through a disaster.

At 4:35 a.m. I got the text message that one of my servers is down. Reason? Trivial. No funds on a connected credit card. Did I miss some notifications beforehand? No. I turn them all OFF by default. I consciously opt-out of any things that may distract my work and life. And with over 150 subscription services that my business handles I’d be notified every millisecond. But hey! Shouldn’t notifications that come before the blackout be AT LEAST the ones that I opt-in for? No.

I have robots to deal with them 🤖

At 4:35 one of them got woken up. He checked the balance on my different disposable credit cards, took the first one with sufficient funds and plugged it in via hosting service billing API. At 4:36 I got another text message. Server Status: up.

I spotted the messages after I went cycling, had a shower and morning coffee, around 8. Good ol’ bots — I thought.

The first time the idea resembled me while reading about nomadlist ran by Pieter Levels, who hired over 1000 bots on his site, including those calculating beer prices 🍺 I picked up the idea and started learning about cron, automation, AWS Lambda, web scrapers and writing first small pieces of software.

Holy sh*t, that’s awesome! — I thought.

I’ve learned to code early in my career, but then switched to web design, UX and biz dev. Although I could code anything I didn’t find it too exciting. It was taking too long and I kind of missed the instant gratification you get from graphic and motion design.

When I discovered bots and automation it instantly felt like the missing piece of the puzzle in my life just got in place.

The results are quick and powerful and no or little code is required to make software, that actually works.

As I was exploring more tools and techniques I discovered that almost every daunting part of my work can be replaced with smart automation. Starting from organising my schedule and sending invoices to my accountant, to running an entire online business on autopilot, with bots taking care of purchases and returns, invoicing, customer service and marketing. Result? Today I run multiple online projects including the one with 300k clients and only few people on board, thanks to bots and automation.

Subscribe now

Automation as I know it

Most of the people I talk to have come across the concept of automation. However, their understanding of this topic is different from what I want to present, so I believe it is very important for you to be aware of how the evolution of automation looks from my point of view and how I understand this concept. For me, automation is the use of available tools to minimize human involvement in various areas of online projects, such as sales, customer service, accounting, or marketing. I want to share my approach and set of techniques that I use every day to make my work simpler and not involve me where it is not necessary.

A simple example is handling returns where we need to issue a corrected invoice, receive confirmation from the customer, process the return, and book it. Seemingly a simple operation, as it turns out, in larger companies, it involves several positions. Polish airlines LOT recently processed my refund for a canceled flight for 7 months. That's not all; changing company data on an invoice in the Microsoft Azure service took almost a year and required the exchange of 151 emails. Meanwhile, while the thread was open, one of the customer service representatives managed to get pregnant, give birth, and go on maternity leave. In Vimeo, a similar process is still ongoing, I can't count the messages, but at the moment, 14 people on the Vimeo side are involved, including 4 managers, legal department personnel, accountants, and several others.

You could say these are procedures of large corporations. Meanwhile, in smaller companies, even though the process may be faster, most of the time nothing is described, and each case is considered individually, with the peak of dreams being template responses that slightly speed up the work.

Meanwhile, all repetitive tasks can be delegated to robots and automated, almost entirely, even if the process is complex and requires confirmation exchanges. On one hand, this is a huge time and money saver for companies where hours are spent on boring and repetitive tasks, and on the other hand, a huge opportunity to gain a competitive advantage, or even compete with much larger companies, for smaller ones. Another huge benefit of automating boring, monotonous tasks is unleashing the potential of people who could be much more creative and satisfied with their work.

For years, there was a belief (and few still think differently) that automation requires specialized knowledge and armies of programmers. Indeed, until now, process optimization was mainly done by larger companies that employed RPA (Robotic Process Automation) specialists, people who first spent 3 months recognizing a process, then produced dedicated software for several months, and then maintained it indefinitely. You could say that automation was only available to the largest and wealthiest.

Probably the most revolutionary change in this area is the evolution of tools that support automation and, consequently, the democratization of these techniques. Today, literally everyone is able to automate, and most importantly, reaching a truly advanced level does not require any programming knowledge. Additionally, tools supporting automation are affordable, which also guarantees wide access to technology.

The background (boring but worth it)

When the Internet became almost universally accessible, options for data synchronization in the cloud and the ability to collaborate simultaneously on the same data emerged. Initially, there were project management applications and then document editing tools (Google Apps). Today, even designers can work simultaneously on the same project, for example, using Figma. This clearly shows a progression, leading us from a situation where applications worked independently to the point where they moved to the Internet and browser-based versions. This facilitated collaborative work and access to our data from any device.

And since we're talking about devices, we have gone through a revolution related to mobile devices, or rather, multi-platform support. This has made the process of software development and design much more complex. Now, developers not only had to develop the main version of the application but also take care of their mobile versions for smartphones and tablets. This led to the strategy of designing application architecture using APIs, which stands for Application Programming Interface.

Definition of API

To understand what an API is, it is important to know that an "interface" is a way to enable communication. And an API is a way to enable communication within one or multiple applications.

This strategy simplified the process of creating applications for multiple platforms because developers no longer had to create multiple versions of the application. Instead (using a simplified schema), a single server providing user data was sufficient for applications created on specific platforms, with a lower level of complexity in this case.

Furthermore, not only the developers of a specific application have access to a well-prepared API but also anyone who obtains access keys (e.g., API key).

API-First Design and the Online Business

All of this led to the creation of a software development strategy called "API First". From what I have just written, this approach is fully justified. If we can write one code to handle multiple platforms, why wouldn't we do it?

And what does this mean for non-programmers? Well, the API-First approach can be used as a business strategy and for utilizing applications - thanks to automation.

As we have already mentioned, APIs allow for data exchange within applications and between applications. This opens up entirely new possibilities. It turns out that we can connect various services we use and exchange data between them. This means we are no longer limited by the functions available in selected applications but have a full range of possibilities provided by all the tools we use!

How is this possible? Well, developers provide access to their API not only to their own applications (for example, the mobile and browser application of your bank), but also... to other developers, who can build solutions based on their software.

These other developers will be us, using tools specialized in specific tasks - sending emails (e.g., Mailchimp), SMS messages (e.g., Twilio), enriching customer data (e.g., Clearbit), notifying the sales department (e.g., Slack), etc., and combining them through Automation. With this approach, we will never have to log into the graphical interface of the applications we use, for example, to send marketing emails. Automation will do it for us by "logging in" to the API and sending emails of specific content to a designated group of recipients. This means that neither we nor our employees will ever have to log in (and forget passwords) to the online services we use, which have good, accessible APIs.

The benefits of this approach include:

• No need to use GUI (reloading pages, slow tool operation, forgetting passwords, logging in again - all of these waste time!)

• The ability to connect the best, specialized services instead of relying on one provider who may constantly lack certain functionality

• Reactive work, not proactive work - we don't have to check what needs to be done; automation assigns tasks that humans should complete in the process (more on that later)

Looking at this through the lens of my earlier definition of automation, a technical aspect can be added to the list:

1. Detailed process mapping

2. Attempt to simplify/improve the process

3. Choosing the right tools for the right tasks in the process

4. Connecting these tools through APIs to work at the right time, without the need for human intervention and logging into GUI

5. Supervision, error correction, and experiments

Example of API + Automation

Let's take the example of organizing an online event, such as a webinar. The webinar tool allows for participant registration, so we have access to email addresses and phone numbers. With the help of API and automation, we can automatically transfer this user's data to the Clearbit service responsible for Data Enrichment, which enriches data. By sending, for example, a corporate email address, we receive additional information about the person signing up for our event. These enriched data can also be automatically sent to our CRM system (e.g., Pipedrive), positively impacting the sales process because when contacting such a potential customer, we have more information about them to better tailor our services to their needs. Alternatively, we can use the phone number to automatically send a short SMS reminder about the event. This increases attendance, which can benefit us and our potential customers.

If we combine this with a system that collects customer data on our website, such as Intercom, our salesperson can also receive precise notifications with information like:

Adam from the Browser company logged in 10 minutes ago, he participated in the "Financial Markets" Webinar and visited the pages: offer, faq, contact. His phone number is: 888999000

As you can easily imagine, using one tool in this process is impossible. There is no single tool that can handle webinars, website data, company communicators, etc. And this is the power of automation - we use various tools specialized in specific tasks, juggle their data, and the results go exactly where our team feels comfortable.

So, as you can see, we used the API here to transfer data between different services and thus optimize a process that was either previously done manually or not done at all. This approach streamlines existing processes or creates new ones. I have personally found that in the case of online businesses, this approach truly takes our projects not just to the next level but often ten levels higher.

For example, the projects we have undertaken, considering the scale of operations and their profitability, can easily be compared to businesses that require the involvement of even a team of dozens of people!

Subscribe now

Let’s have fun (you survived the boring part)

As we established above, modern applications are increasingly communicating through APIs. Let's take Facebook as an example. The business logic is responsible for adding your post to a certain type of database and your feed. This logic can be triggered from various places. You can add a post through a mobile application or the Facebook website, and as you know well, other applications can connect with Facebook and post on your behalf.

To enforce this logic, you need to:

1. Know where the back-end expects this data (an endpoint URL)

2. Send the data in a way understandable to the back-end (usually in JSON format)

3. Authenticate using certain access data (login, token, etc.)

And voila! The post is on your feed.

How to Utilize the API First Approach

The entire internet works this way. And thanks to this model, everyone can add, modify, and execute commands in web applications.

But instead of becoming a developer and learning how to structure, format, and send data through various APIs, you can use tools that will do it for you. I call them LEGO tools. When you authorize any application in such tools, they can perform various actions on your behalf. For example, create a spreadsheet and add rows to it.

Now, let's think about this more broadly. If LEGO tools can connect with different applications, they can also exchange data between those applications. For instance, these tools can take rows from a spreadsheet and post them on your Facebook feed. Sequencing, timing, delays, conditions - everything can be set.

Infinite Possibilities

Now imagine you want to gather data from your spreadsheet and distribute it to all your social media profiles - Facebook, LinkedIn, Instagram, YouTube, all at once. LEGO tools allow you to connect with numerous applications and services you use and exchange information between them or work on changes in one tool while notifying others.

Here are a few example scenarios:

• Searching for events in your calendar and sending you reminders about the most important ones;

• Collecting tasks you enter into Trello and syncing them with your Google calendar;

• Sending emails to webinar participants, posting on social media, and assigning tasks to your team on Slack - all simultaneously.

The beauty of LEGO tools is that they expand the capabilities of the software you already use, allowing you to achieve practically anything you can imagine. Often, the tool you are using may lack just one feature, such as text reminders. With LEGO tools, you can add a text messaging service to your existing arsenal of options in just a few minutes.

Popular LEGO Tools

The most popular LEGO tools are zapier and Make. Its main advantage is that it supports hundreds of available applications. It offers a clear and intuitive interface for creating automations. Make It has a very interesting, visual interface that allows you to build automations by literally connecting a few dots. Additionally, it offers a rich, free plan to experiment with automation.

These are the tools I use most often, but there are also many other tools, usually designed for more specific tasks.

• Parabola - a very good tool for data transformation. This means reorganizing, splitting, changing the order, formatting, and much more. If you have data from multiple sources or a lot of data that you want to transform and send to other tools, this is a great choice.

• Actiondesk - based on a spreadsheet interface, allowing manipulation, transformation, and creating logic within your data. If you like spreadsheets, it's worth checking out this tool. It allows you to make various transformations on data and send it to another tool.

• Retool - this tool fetches data from APIs or databases and allows you to transform it or even send commands back for execution. If you want to create an admin interface for your database or a report panel, this is a good option.

Additionally, there are other tools, created for specific actions, that I don't categorize as LEGO tools - they allow you to combine one or two services and create something new. A good example is Glide, which lets you create a mobile app from a spreadsheet. There are many interesting tools of this type, but that's a topic for another article.

If you want to learn more about different tools and techniques we use for automation and productivity, make sure you check out Everyday on Autopilot — our productivity and automation bible, currently 50% off for Techsistence readers only 💥 This is the best start to your automation journey. Here’s few recent reviews from Product Hunt.

Your first automations!

All of this may seem overwhelming. But once you understand the concept behind LEGO tools, it will be quite easy to implement and learn any new tools. It's worth knowing at least a few of them. Sometimes the application you want to automate is available in Zapier but not in Make. Sometimes you want to work with a larger amount of data, and Parabola may be more convenient. Unfortunately, there isn't a single LEGO tool that meets all your expectations, but that's... not a bad thing. By learning different tools, you encounter diverse scenarios and thought patterns.

When I fully grasped how Zapier works, I was very excited. But then I struggled with implementing this concept into my business. Tutorials were available but taught everything from the wrong angle, saying "connect X to Y" instead of "achieve X by doing Y." This made it very difficult for me to work with these tools, and it took me over a year to come up with good automation strategies and their utilization.

Automation in Zapier

First automation we will prepare based on one of the most popular platforms in the world - Zapier. This tool is very beginner-friendly and allows you to connect various tools and transfer data between them without coding. Our first automation will show you how to work with Zaps, as Zapier calls scenarios, or individual "robots".

What will we automate? We will create a simple automation that will save your Gmail attachments to a Google Drive folder. Then we will slightly modify it to have a practical application by saving the invoices you receive in your mailbox in one place.

1. Start by creating a free account on Zapier. To do this, go to zapier and register, then create a new Zap by clicking this button, and then Open Editor, creating a Custom Zap.

   

2. Now you need to add a trigger - something that will start your robot. Search for and select the Gmail application. There’s an experimental AI search above, but we’ll skip it to better understand the process

   

3. In the next step, you need to connect your Google account with the Zapier application and specify the appropriate mailbox. You only do this once, and subsequent automations will not require reconnection

   

4. The trigger event for us is New Email Matching Search, meaning we want to trigger the robot every time we receive a message with specific topic and attachment

   

5. Next, we can fine-tune our trigger, and here Zapier offers a very interesting possibility. We can perform more complex queries, according to the Gmail search documentation. For example, I used invoice OR bill AND filename:pdf, which means among messages with attachments, it will only search for those containing the phrase invoice or faktura (or similar because I didn't use quotation marks). Additionally, I am only looking among results containing PDF as an attachment

   

6. You can test your automation with the Test Trigger feature, and if the mailbox already contains elements that match the entered phrase, you should receive them as results (if there is nothing, simulate test data by sending yourself an invoice to the mailbox)

   

7. And you should be able to see results

   

   Select the appropriate result and click Continue.

8. Great! We have configured our trigger, meaning we can receive messages with attachments, additionally filtered. In this case, we only need to configure what will allow us to save the attachment to Google Drive. As part of the Action (what happens as a result of the trigger's action, we search for Google Drive)

   

9. Then, we choose what we want to do, which is to upload the file to Google Drive

   

10. Connect your Google Drive account, or select it if it has been previously connected

11. If you haven't created the appropriate invoice folder on Drive before, now go to drive.google.com and create a folder named invoices. If it's already created, you will be able to specify it as the location in Zapier

    

12. Now click on the file and from the drop-down list, select the Attachment option, simply passing the attachment as a file to Drive. Leave the other fields unchanged, except for the File Name field, where you can get creative and change the name of the saved file, using various metadata from the email, such as the sender's address, combining it with the file name. I added hyphens between them for clarity

    

    Tip: If you paste the following phrase in the Zapier field: {{zap_meta_human_now}}, Zapier will substitute the current date in that place. By formatting names in this way, it will be easy to find files with the correct date. Additionally, sorting by name or date will now return neatly arranged results, which means you don't need to create additional folders (but if you want to do it, feel free to do so. Try modifying the automation to place invoices in a folder with the correct name, for example, January. You will need to create these folders manually or with the help of another automation).

13. That's it! Select Continue, and then Test & Continue to check the automation's operation. If everything went smoothly, you will see a success screen

    

    Now you can Publish your scenario! Test invoice is already on Google Drive

    

    Remember to turn on the Zap using the switch at the bottom so that it starts analyzing all incoming messages from now on. Otherwise, the Zap won't work!

Subscribe now

Automation with Make

We prepared our first automation in Zapier for a reason. Make, the application we will be working with now, is not as good at integrating with Google tools. You could say that Zapier is better aligned with Google, and this is a good example of why it's worth having various automation solutions and using the ones that work best in a given situation. In Make, to use Gmail, you need to have a paid Google Workspace account.

Although automating with Zapier is incredibly simple and intuitive, there are many other tools in the world of automation that you will use. Why? There are many reasons. For example, Zapier is a bit expensive, and you can find cheaper alternatives. Additionally, while it offers many interesting features, other tools sometimes work better and function slightly differently. In Zapier, although it is possible, splitting scenarios based on conditions is not the most intuitive.

For example, we can create a bit more advanced scenario in Make, that searches for invoices then renames, catalogs and save them to drive, checking if the folder structure is correct. It looks like this:

On the left, we have the trigger, which listens to emails on Gmail. Then we rename the file, and search for the appropriate folder on drive. The folder should contain month and year to catalog our invoices, so if it doesn’t exist, we will create it in the next step and then put the invoice there. You don't need to understand exactly what these elements mean right now, but the concept should be a bit clearer now.

What will we automate this time? We won't repeat the same task in Make. Instead, we will add another automation that will send a notification on the 10th day of the month, informing our accountant or our email address that the collected documents are ready.

1. Create an account on Make (this is our special link, that grants you 1 free month on the pro plan 🎉)

2. Create a new scenario (equivalent to a Zap) using the Create Scenario option

   

3. Click on the first circle and select the Gmail application

   

4. Choose the action Send an email

   

5. Click Add in the Connection option, name your connection, and like in Zapier, connect to the appropriate Google account

6. Now add recipients, subject, and content of the sent email along with the link to the shared Invoices folder on Drive

   

7. Select OK to confirm the changes in the module

8. Now set the schedule, for the Gmail module, click on the clock, and then schedule sending this email on the 10th day of the month

   

   

9. Save the module by clicking OK

10. Turn scheduling to ON and you’re done!

    

This is a super simple scenario, but introduces a nice concept of scheduling and running your robots remotely. If you want to expand it, try sending a Slack message to yourself with confirmation, after you send email. Small tip:

You already have your first automations on your account - congratulations! As you can see, it's child's play. Of course, these are just the basics, and in practice, automations can be much more complex, including, for example:

• Many more modules and routers, i.e., paths

• Conditional instructions "perform only when the data contains X"

• Functions, for example, searching through returned data

• Data completion, for example, with the current date or data calculation using mathematical operations

• Error handling and modules for testing

• Custom scripts, for example, JavaScript

• Modules allowing data reformatting (parsing)

• Modules capable of iterating, for example, over records in a database or rows in Google Sheets

• And so on

There are plenty of possibilities. But let's break down what really happens in the scenarios we created. The fact that Zapier or Make allows us to choose the Gmail module simply means that the tool can integrate with it. How? Of course, through APIs! For example, Gmail has an API, where we can refer to the appropriate endpoint and listen to incoming emails. Once we retrieve information about the incoming message, the next endpoint allows us to download the attachments. Zapier, which we use, just has these things built-in, but they work on the standard Gmail or Slack API. We work exactly like a programmer who would prepare an integration for us, except we use a visual builder and a set of prepared modules. The account connection we made is essentially the authentication in the API using one of the methods discussed earlier. It's programming without programming!

Make, like Zapier, offers the possibility to use hundreds of ready-made scenarios or import any blueprints prepared by us or others!

Automations & AI - the ultimate productivity tool

As you probably guess, over the years we've created quite a lot of automation for our needs. You might be surprised, but currently over 3000 robots work for me, performing hundreds of thousands of operations daily. This is just one of the Make accounts I use, where I process nearly half a million operations monthly!

But this is just the beginning! Just under a year ago, we realized how useful NLP and AI could be to help us work with automations. It turned out that input data for automations, which needed to be formatted in a specific way (we send them the so-called JSON object), could be prepared by AI! What does this mean in practice? It means that you can say or write, for example:

Add client John Doe to CRM

And then the language model will take care of translating this into the proper format, meaning into the "automation language," and then send it to a ready scenario on Zapier or Make! It's truly amazing how vast the possibilities are when using such an approach.

Unfortunately, there wasn't a tool that allowed us to communicate AI with automations in this way, so over a year ago, we started working on our own solution to help us with this.

And that's how Alice was born.

In addition to being able to chat with Alice like with ChatGPT, her biggest advantage lies in the ability to remotely trigger our automations, and consequently - perform actions! ChatGPT offers this concept also with GPTs and their API standard, however, this approach proved to be much less efficient so far, at least for our use-cases.

Future of remote actions in Alice is very promising, and probably we will, in time, start adding native integrations to Alice. Also, performance of LLMs is rapidly increasing, and also access to local models in Alice is a possibility. Today with Groq we’re able to perform operations faster than ever before, and this will evolve. Take a look at this command to Spotify and the speed of its execution.

In this context, I've prepared some tutorials for you, showing how you can integrate the Alice app with Make or Zapier. Please, see how it works in practice!

Remote Actions

Calendar scenario

Schedulink links scenario

Using existing scenarios

That’s it for today, but stay tuned! I’ve prepared for you a collection of inspirations and readymade automation blueprints that help us every day! Will be dropping it soon!

And one more thing. If you find today’s edition interesting, and you want to dig deeper into Alice, automations and our vision for productivity, we have something cooking for you! We’re considering creating a full-blown, 4-week AI+Automation cohort that you can take part in! There’s so much to learn, and here’s what we’re planning:

• 20 practical lessons — text + video (10+ hours of content)

• 4 live consultations (2h each)

• Very practical approach to AI (from the ground up)

• Showcase of the tools we already use on production

• Access to Alice and creating automations!

• Community-driven cohort and our support

If we manage to get 50 interested people (we have 7 so far!), we’ll start preparations!

Sure, count me in! »

See you in the next one!
Greg

In this post, I aim to reflect openly on the current role of large language models (LLMs) in my daily life and what aspects might still be eluding me. Through this, you'll gain insight into how I collaborate with AI and where my focus lies. You might discover something valuable for yourself in both areas.

Copilot vs Autonomous Agent

In mid-2023, Andrej Karpathy in State of GPT clearly suggested that currently, language models (LLMs) perform better as copilots than as autonomous agents. This is likely to change with the release of GPT-5. However, until that happens, LLMs typically reach their highest effectiveness through collaboration with a human. That's why, for the most part, my computer screen looks like the one below.

At every step, I have a chat window open with LLMs from OpenAI, Perplexity, and a model running locally on my computer. AI is also accessible on my watch, allowing me to speak with LLMs through ElevenLabs and Whisper.

In other words — constant availability is the foundation of my collaboration with AI. And although it is an essential element, it is not the only one. Personalization also comes into play, involving the integration of the model with my own knowledge bases and devices. Ultimately, we get a system that can be classified as something between a "Copilot and an Autonomous Agent". And it is precisely here that there is space to explore what is possible and what still limits us.

Before we go further, I would like to quote Logan from OpenAI, who talks about thinking of AI now in terms of the potential possibilities that GPT-5 (or subsequent versions or alternatives) may offer us. This way, it's easier to go beyond what has so far seemed realistic.

So, illustrating our situation, we will want the interaction with AI not to rely solely on the default available capabilities. Of course, we can still find value in this, but we're talking about something akin to a "demo".

Instead, we will want AI to gain access to our individual context. I mean both information and skills that are necessary for it to become part of our daily life. Specifically, this includes:

• the ability to interpret commands within the current conversation

• having basic information about us

• access to environmental data (location, device status, current activity, weather)

• the ability to manage its own memory

• the ability to take asynchronous actions

• the ability to respond to events and work according to a schedule, as well as proactivity

• the ability to use our knowledge base and indicated sources

• the ability to use APIs of tools, applications, services, and devices

• text, voice, and visual communication

One might think that we are aiming for a result in which AI is able to perform virtually any activity for us.

Experience (and a number of technology limitations, as well as our requirements) suggest to me that in practice we will strive to achieve a type of collaboration in which both we and AI have access to the same areas and complement each other.

Even just looking at the above diagrams, one can notice that we are talking about a complex system. And indeed, in a sense, it is, but it is possible to define certain universal principles and patterns that form the foundation of the entire system's development. It is important to remember both the current limitations and to maintain flexibility resulting from the development of both artificial intelligence and other services and tools.

Taking Actions and Recognizing Intentions

Interaction with an LLM always starts with sending a message, which can come from the user or be sent by the system. Its content must first be verified for potential violations of the terms of service and the scope within which our assistant operates.

Then, further steps must be taken, which may be predetermined by the system. An example could be sending a message containing a description of a task to be added to the list. Then, the model's task is simply to transform the received content so that it can be placed in the task application. After taking action, the model's task is to generate a response informing about the action taken.

So, we are talking about a situation where the LLM in the first stage converts the message into a JSON object describing the task. Then, thanks to automation or programming, this object is sent to the task application's API, which returns a response, the content of which is used to provide an answer.

Such interaction may initially seem attractive, it performs very average in practice because we usually want to perform more than one action. This means that we need to adjust both the behavior of the model and the operation of the other system components to be able to process the entire set of data simultaneously. Then, the message may contain not only information about many tasks to be added but also mention those that should be updated. Ultimately, it also involves immediately querying the assistant about tasks from selected projects, statuses, and date ranges. In other words — we are interested in a free conversation with the model.

Below is a modified version of a conversation, which in this case does not move within the scope of individual entries, but entire lists. Therefore, it can be seen that from a technical point of view, the change is minor. However, looking at it from a usability perspective, we are talking about a chasm. At this stage, one message is transformed by the LLM into an entire series of actions!

We can now go a step further because on the same principle as action recognition, the selection of a toolset, as well as general planning of actions necessary to complete the assigned task, can also take place.

In this way, we have reached an extremely powerful concept, which can be defined as a single point of entry. Thanks to this, our system is able to react to data coming from various sources and independently decide what to do with them.

In this way, we gain previously unattainable flexibility, but this comes at the cost of predictability and control. Besides, the system in such a form is not complete and will not be able to effectively cope in practice. The reason is that planning actions will require access to various information, including data from external systems.

Context and Action Planning

For simple actions, the information contained in the prompt is completely sufficient. For example, if we want the model to assign us tasks to the appropriate categories, we can describe them and even present a few examples. Such a thing should be entirely sufficient for managing a private list. The situation starts to get complicated when there is more data, or it is dynamic. Then it becomes necessary to load it beforehand so that classification is possible.

At this point, the user's question is first analyzed in terms of information that can be read from long-term memory or external data sources. The information obtained in this way constitutes the context of the query, which we know from RAG systems.

And only after obtaining all the necessary information does the system prepare an actual action plan and undertake the previously discussed steps. However, the user's question alone may be insufficient to correctly find all the information. Therefore, a quite useful technique is self-querying, which involves asking deeper or category-matched questions.

Context gathering will usually depend either on the query itself or on its preliminary classification. The goal is to collect the most useful information possible, which will help in planning further actions.

An essential category of data that will usually need to appear in the context are metadata and identifiers. The former will be responsible for providing context (e.g., the source of origin), and the latter for identifying the entry. Below we have an example where the difference between a calendar event, a task, and individual projects is clearly visible.

Loading the above context enables the correct execution of a command containing a request to mark selected tasks as completed. This is possible because the model will be able to associate the user's query with the retrieved list and send the entry identifier to the API, which is to be modified.

I think at this stage, it is already clear how important it is to properly prepare data and instructions for the model. Their effectiveness translates directly into the operation of the entire system.

Memory Organization, Hybrid Search and Graphs

Despite the increasing context limits, which now even reach one million tokens, the organization of knowledge sources, data formats, and their retrieval methods remains an important topic.

Below, we have an example of a context consisting of the contents of two files. Both contain information about two different people. If their content enters the prompt without any marking, even for a human, it will be difficult to assess whom they concern.

However, the presence of metadata defining the origin of the individual segments is enough for the model to distinguish them from each other and effectively use this data. It is also important that in exactly the same way, the LLM can also indicate the place where a specific piece of information should be saved.

Therefore, assuming that our system will have different areas of its memory, it will also be able to manage them. Additionally, thanks to the ability to transform content, information can be remembered in various forms, depending on the category. We see this below, where information about the residence of two people has been appropriately placed in several memory areas. In addition, the model was able to recognize that the interlocutor is Adam in this case.

If we now combine the previously mentioned concept of loading data at the initial stage, our system will be able not only to save new information but also to update those it already has. For this reason, information about a new place of residence not only reached the indicated place but was also connected with previous knowledge.

However, this is not sufficient because it is enough to mention another person named Greg, and the system will incorrectly link the information. Therefore, additional steps must be taken to avoid such a situation. We can achieve this, for example, by warning the model about cases when it has to ask us for clarification (e.g., which Greg we have in mind). Similarly, the model may have criteria that will help it assess what to remember and what not.

Organizing data must include not only categories but also the previously mentioned metadata, which can be used in the context of the prompt and in the search process. However, it turns out again that this is insufficient. Imagine that during a new conversation, Adam claims that Greg is his best friend. Then this information is recorded in the "People" category, where information about "Adam" is located. Our structure not only expands but also begins to have various complex relationships.

If in such a situation, the message will contain a request to add to the calendar an entry about the birthday of our best friend at his home, then finding the complete set of information begins to be a significant challenge.

Firstly, there is a semantic connection between the words "best buddy" and "best friend". This means that to associate these two pieces of information, we will need a vector base. However, before that, we must ensure that the search takes place in the proper category. After finding information about who is Adam's best friend, it is also necessary to find his place of residence, which was mentioned in the original message.

The result of the above search is finding information about who the mentioned person is and where they live. Based on this, it is possible to add an entry to the calendar about the upcoming birthdays. The connections between individual entries may at some point move us from tree-like structures to graphs, which further increases complexity. However, the good news is that memory management can be entrusted to LLM. And although the whole remains complicated, it does not always have to be so, because usually, we will be interested in smaller structures.

Taking Action

Having a system capable of recognizing incoming instructions, using its own memory and external knowledge sources, and able to plan its actions, we will also need to take care of taking actual actions.

It will probably not be a surprise if we encounter a few additional surprises here, which I have collected in the following points:

• Communication takes place via API. Therefore, it is necessary to use tools that provide it.

• In the case of API limitations, it is worth considering scenarios in which the system supports the execution of tasks by assigning them to the user.

• Performing tasks will usually take from a few seconds to even several dozen minutes. Therefore, it is necessary to include a queuing system, increase the waiting time limit, perform actions in parallel, and often also stream responses to reduce reaction time and positively impact user experience.

• Tasks may fail for various reasons. Then, feedback should return to the system, which should be clear enough to be fixed automatically or by user action.

• The system should include mechanisms for automatic and manual supervision and verification, as well as input/output control.

• LLM should not perform irreversible actions (e.g., deleting data without a saved version history).

• Each of the tasks performed should be described by the model, for the purpose of providing a final response.

• The system should be designed to be easily updated to the latest model versions, including switching between providers.

• The entire system should be monitored (e.g., with the help of LangSmith).

• After completing a task, the system can also carry out a series of actions related to self-reflection and updating its state and long-term memory. Each of these actions, depending on the situation, will positively influence future operation.

In this way, we come to a scenario where we have a set of concepts that can be applied at different levels of complexity. Collecting them all together, we are talking about:

• Recognizing user intent

• A single point of entry

• Extending the original query

• Supplementing information for the purpose of taking action

• Planning actions

• Performing a series of actions on external tools

• Organizing and using long-term memory

• Different categories of activity (active and asynchronous)

• Returning responses based on multiple data sources, including reports from actions taken

Translating this into practice, we can design both simple tools and quite complex systems based on this. Here are a few examples:

• Content transformation tools (e.g., translation and correction)

• Content organization tools

• Content enrichment tools

• Research tools

• Documentation management tools

• Process supervision tools

• Reporting tools

• Monitoring tools

• Content verification and moderation tools

• Systems combining various data sources

• "Second Brain" type systems and corporate knowledge bases

• Systems that help in acquiring new skills

• Systems that act as assistants, equipped with memory and a set of tools

And finally, with appropriately designed tools or systems, their combination also comes into play, which opens up space for performing complex tasks autonomously. And although it is currently difficult to speak of high effectiveness of such solutions, probably in combination with GPT-5, the results achieved will meet our expectations and often surprise us.

Have fun,
Adam

You probably don't know this yet, but I'm obsessed with productivity. I basically have to be, as I run several companies at once, employing dozens of people within a completely flat structure. However, if there were to be a hierarchy in my organizations, productivity would be at the very top.

So, for example, here are a few random facts about me:

• I don't hold any meetings and my calendar is clear

• I have thousands of automations that assist me in my work

• I've been working hand in hand with AI for over a year

• I have one central dashboard where I manage literally everything

• I've created my own AI clone that my co-workers can converse with

If this has piqued your interest, read on...

Day 0

Before the day ChatGPT came into existence, I didn't have much interaction with AI. However, for many years, I've been creating automations that assist me in my work and daily tasks. Most of my projects operate based on automation scenarios, which perform hundreds of thousands of tasks for me each month. These are the statistics from one of my many accounts on make.com, which processes 30-40 thousand operations on a good day. And that's just one account!

In this way, thousands of robots work for me, responsible for things like:

• Handling my inboxes and triaging messages

• Queuing posts for company social media

• Adding customers to CRM and enriching their data

• Handling subscription purchases

• Onboarding sequences for users

• Reminders for events, SMS, emails, and other notifications

• And much, much more!

Essentially, every aspect of my businesses is surrounded by automations - from customer service, through back-office, sales, to marketing. This model of managing these processes allows me to run multiple projects simultaneously, with small, focused teams, and the robots work for me when I sleep.

Day 1

ChatGPT appears, which I treat as a curiosity, I check a few prompts, play with it for a while. It makes a pretty big impression on me, but as is usually the case, I don't initially see real applications and can't connect it to solving problems or improving my workflow. The answers aren't very precise and it's safe to say that it's not suitable for "production" applications. But over the next few weeks, I can't stop thinking about ChatGPT and further AI developments make me start devouring everything I can find on the internet about neural networks, advances in LLM development, and the future of what's coming.

This quote keeps ringing in my ears, even though I can't remember where I heard it.

Day 14

I am now convinced that in the future AI will change or significantly improve my productivity model. I feel like I'm about to take a big step, but I'm not yet completely sure how or when it will happen. On the other hand, I allow for a scenario in which all of this may actually stop and we will hit some kind of wall with AI. So I still approach the topic very cautiously, but I am putting together some patterns that I could potentially use in my work.

Not long after, it turns out that I'm not the only one who suspects that one of the most important applications of AI in the near future will be to increase productivity. Today, this is obvious to most leaders of the world's largest companies. Data from the PWC report indicate that it is estimated that the use of AI will give workers about a 40% boost in productivity, which is confirmed by Statista research:

Subscribe now

Day 30

Aha moment! One simple thing that lifts the spell and allows me to demystify AI in terms of productivity. It turns out that LLM is quite good at transforming the commands it gets from me into the JSON format I use in all my automations. This is now common knowledge, used by many tools, but around the early days of ChatGPT, it wasn't yet obvious to practically anyone. This one simple thing (again, something that seems like fun) is the missing puzzle piece to connect AI with productivity. It allows me to run my automations in a completely new way - by having a conversation with the model. In this way, I can easily imagine a scenario where I or a person from my team gives a command to the AI:

Create an account for user X on our service for a month, then send the onboarding sequence.

This one simple concept allows me to imagine a completely different level of productivity, where the model is not so much able to connect with external services and interact with them. Rather, the model prepares data for my automation, which has been handling this task for a long time and does it excellently.

Now the only question remains: how the hell can this be done?

This is how the concept of Alice is born.

Day 60

After preliminary research, we find that no one else has come up with this or made a similar prototype, so we start working on an early version of Alice. The assumptions are simple:

• A native application (because we are most productive at the computer)

• Which allows interaction with LLM (via OpenAI API or other)

• And the execution of our automations (with the help of sending formatted data to a Webhook)

During our work, these assumptions are repeatedly confirmed by others, as well as by OpenAI itself, which gradually begins to introduce Chat features similar to those we thought of for Alice. However, this does not change the fact that our concept is completely different, as it aims at a much wider range of typical productivity at work, which for me is associated, among other things, with very quick access to answers and the possibility of connecting prompts to keyboard shortcuts. This simple concept basically changes everything, and the solution that can help me has to be a native application.

After some time, we manage to create a working prototype that communicates with our automations and returns... complete nonsense 🙃

Day 90

After the first tests of Alice, my enthusiasm wanes significantly and just when I'm about to put Alice away, GPT-4 appears. It turns out that our concept may have been ahead of the model's capabilities, but... only for a moment. The progress in the field of AI is one of the most fascinating things I've seen in IT over the last 20 years.

It is precisely this pace of development that determines that you can currently start projects that are potentially useless, wait a while and... be a pioneer of a new, amazing market. That's how we think about Alice to this day, because although it gives us tremendous value, we will still have to wait a while for mass adoption of this concept. How long, is unknown - but I am almost certain that our concept will work out much better at the beginning than the assumptions of "general" agency, mentioned by OpenAI, which gets a lot of startups first excited, than extinct when OpenAI does it.

Why am I so sure?

The way we invented Alice doesn't immediately turn upside down what we're used to at work. The concept of agency in our version requires the preparation of automations that the model triggers. So it's not issuing a command into space, on the principle of:

book me a cottage in the country for the weekend

With all due respect, I dare say that creating such a universal tool at this point is completely pointless. It's an interesting theory and maybe one day we'll be able to get such actions from the model, but it also requires many infrastructural changes and software adaptation to the needs of such agents.

Unfortunately, I bet that even if such tools are created, the predictability and stability of the actions they perform will simply be unacceptable. Not to mention the fact that we are trying to change a paradigm here that may not be so bad at all? Maybe part of the process of resting in the country is choosing a nice cottage on AirBnb? Maybe people won't want AI to decide for them, because it's just a part of the process that they appreciate and that is enjoyable in itself.

It's a different story with the tasks we have to perform at work. Especially, with the help of interfaces that will simply be gradually replaced by AI and agency.

Why? Because they are simply ineffective. It's only the last few decades that have made us think that the mouse cursor or our thick thumbs on a small phone screen will be effective in issuing commands. As a result, performing a simple action often takes us a lot of time, when we have to fight our way through interfaces that are not adapted to devices, heavy interfaces, forget the login for the umpteenth time, fill in the same form with address data over and over again, and so on...

Subscribe now

What was much closer and more tangible was what Alice was supposed to offer us. And we had no problem with the fact that it won't be (immediately) an application for the "masses" and will require a learning process. On the contrary - we were creating it for ourselves and our teams and the only goal at the beginning was to run our automations.

Meanwhile, Alice starts to perform simple operations, which we test, like this one.

Day 100

We have the first version of Alice, which works, executes commands, and triggers our key automations. It has the ability to add Snippets, which I configure under a keyboard shortcut. This allows me, with a shortcut and instantly, to perform, for example:

• Correcting typos in any content

• Creating a summary of the content I'm reading (e.g., an email)

• Quick draft response to a message

• Translating my Polish content into English

• And much more...

Most of the Snippets that I spin in Alice are also assigned as shortcuts to Stream Deck, so I don't have to remember them all. It starts saving precious minutes during the day! The first people in our teams also start using Alice and begin to see value in it.

We also do the first experiments with external APIs, like Adam here.

Day 200

We're publishing the first page of Alice along with a waitlist, which over 3000 people sign up for. We begin the first application tests with beta testers and individuals we're training as part of our automation program. It quickly turns out that Alice can provide value not only to us but also to others!

Therefore, we're designing version 2.0 of Alice, which is to gain a target interface and usability, and perhaps be available to a larger number of people. We strive to design it so that it would also be the best and fastest possible way to interact with the model from the level of a desktop application. All this translates into efficiency and our productivity, and Alice can serve answers in record time.

I demonstrated how I work with Alice in a lecture for over 500 people, many of whom signed up for our lists:

Subscribe now

Day 300

Along the way, we experiment with Alice's applications. When we explain Alice to people, it's hard for them to grasp concrete uses. In essence, you can do almost anything, but as we know, this isn't a tangible example. That's why we created a series of small examples of how Alice communicates with the applications we use, which I published on X.

We also prepared a business case in which Alice is able to register a customer in the CRM system and use a database of real estate listings, which it can match to them:

Day 365

It's hard to believe that our concept has evolved over a full year! We've released information about version 2.0 of Alice, which is meeting with increasing interest:

We're practically ready to make Alice available to a larger group of people, but the question is, are these people ready? We realize that Alice is not just a fast native interface for LLM and we don't want to advertise it that way. On the other hand, getting more out of it, like action handling, requires a basic understanding of automation techniques and, in general, our approach to productivity. It turns out that these early users who actually use Alice almost daily are people who have spent more time with us - participated in one of our programs or read our e-books.

At the same time, we already have a plan on how to develop Alice so that it can help an increasing number of people. However, this is not an easy task and we are beginning to realize that this might be a significantly bigger thing than we initially anticipated. On the other hand, this vision excites us tremendously and we already know that we would like to continue developing Alice not only for our teams. We've got a small taste of what we can call the future of work and although AI is not yet ready to serve it to us immediately, Alice is a great first step, where we get an ecosystem that can be better controlled, even though it requires more engagement.

From my perspective, productivity happens when we are able to adapt tools to our needs. None of the popular productivity frameworks or Notion templates, which offer to organize your entire life, have ever worked well for me. If we want to do it right, we need exactly what Alice gives us today - control. Unlike the promises of AI tools that will be able to do everything for us and currently fall into the category of utopian visions of their creators for me.

Today

Exactly one year from day 0, we were at a point where version 2.0 of Alice has already reached the first users. Here you can see how the Alice 2.0 application worked:

And few month later, we started working on version 3. It’s newest release is avaliable now on heyalice.app, and this is the trailer:

The journey we've taken from day 0 is incredible!

Here, on the other hand, I've prepared a playlist for people who have a paid Techsistence subscription and have already received access to the application.

The Future of Productivity

In practice, however, what really resonates with me is the fact that Alice is not just a tool, but an entire way of thinking about productivity. After all, her features come directly from how we've developed certain work standards over the years, using an army of robots. And this is a completely unique approach that I haven't encountered anywhere else. Therefore, it can be said that the value lies not only in the application itself, but in everything around it, which includes:

• Our set of tools and how we use them (Keyboard Maestro, Raycast, Shortcuts, and many others)

• Our approach to time management and work

• Hundreds of ready-made automation scenarios that we use

• A functioning Second Brain in Notion and a knowledge base in Obsidian

• And much more...

All these things are perfectly complemented by Alice and interaction with AI. That's why the people who discovered Alice with us use her daily to perform tasks, while many other people to whom we've made her available use her sporadically to interact with the model.

That's also why I have a specific question for you at the end:

We're considering preparing all these things in a form from which you could draw knowledge, as a 4-week program, in which we would guide you step by step through our methods, provide dozens of recordings, materials, content, and ready-made scenarios. We estimate that such a program would cost several hundred dollars and would also include 12 months of Alice application updates. Please let us know if you're interested - if we collect a minimum of 50 votes, we'll start preparations:

Count me in, I’m interested »

Please, let me know in the comments, what features of Alice could help you the most and how are you already using AI for productivity? 💬
Thanks for not being a stranger!

Working with larger data sets, we usually have a search engine at our disposal, which is capable of finding specific keywords, often even considering their variants. The effectiveness of such a system is usually quite high, and allows us to precisely find products in online stores or entries on blogs. Assuming that my query and the search engine settings are in order, I will gain access to precise results leading to the sought-after content.

I'm not entirely sure that this way I'll reach the best information for me, and we're talking more about a certain probability here. But, compared to the earlier example, the results I'll get here will be very precise. Like with Amazon, I receive very accurate names, pictures, and prices of products. There's not much room for error or inaccuracies.

We also have a third option in the form of large language models (LLMs). Although their base knowledge is limited, we can connect them with our own knowledge sources using Retrieval-Augmented Generation. Then, LLMs can use the information provided by us when generating responses.

So I can create a prompt to which I'll attach the content of my post. Then GPT-4 will correctly answer my question, using knowledge it doesn't usually have. This way, I gain flexibility comparable to talking with a human and precision similar to what search engines offer.

Unfortunately, this approach also has a hidden cost in the form of the risk of generating an answer that is not true. Moreover, there may be little to indicate this, and we can be easily misled. Sometimes even a different way of asking a question is enough to receive an incomplete or outright incorrect answer for the same set of data. We see this in the following response, which does not mention one's own experiences and memory, which I wrote about at the beginning of this post.

So here we are dealing with three scenarios that offer us different benefits but also have their drawbacks. Specifically, talking to a human offers us high flexibility and precision, but also the potential risk of error. Search engines are much more precise, and the risk of error is quite low, but they also offer less flexibility. Meanwhile, LLMs offer both flexibility and precision, as well as a high risk of making a mistake.

There is no doubt, however, that the idea of combining LLMs with our own knowledge base can be of great value. Then with such a model, we can talk and connect it with tools and services, thus engaging it in our daily activities and even business processes. However, it is difficult to talk about entrusting any tasks to someone who is difficult to trust. We must therefore fully understand the problem and its causes to address it, or simply decide whether it is worth engaging LLMs in specific tasks.

Retrieval Augmented Generation

Combining LLMs with our own knowledge base is based on the ability to add a limited amount of information to the prompt, constituting the "query context". Along with it, we also attach instructions explaining how to provide answers and use the provided content.

The content of the context is not static, however, but is dynamically built based on the user's query, the content of the current conversation, or external data. In any case, it is the logic of the application that supplements it in such a way as to contain the most useful information possible, based on which the model will provide an answer. And it is precisely in this area that we currently find both the most possibilities and limitations.

It is not difficult to imagine that designing a system capable of using available knowledge in such a way that only the necessary information reaches the context at a given moment is a significant challenge.

Let's start with a simple scenario where the model is connected to the Linear application, which allows us to "talk" to the task list. Asking a simple question about listing "today's tasks" requires connecting to the data source (it can also be Todoist or another task application) and retrieving the necessary information. This means that the system must take several additional actions before it even begins to generate a response. At this stage, various kinds of mistakes can occur, for example, related to selecting only today's tasks.

However, assuming that our current tasks actually make it to the context, GPT-4 should have no problem transcribing them. In the same way, we can design interactions with other data sources, including monitoring the content of websites.

The situation starts to get complicated when we care not only about retrieving data but also about managing it. This increases the complexity of the entire system, thereby increasing the scope of the model's work and requiring us to maintain high precision when designing prompts and the logic of the application itself.

Despite the increase in complexity, the whole thing is not a problem for GPT-4-Turbo. My request to mark the indicated tasks as completed has been realized.

Proof of the completed action is below. So, we see that the possibilities of LLMs in combination with our data are very impressive. I experience this even more clearly when, instead of manually filling in the calendar or task list, I simply tell my watch what I have to do. Such a message is recorded, converted to text, and passed to the right place.

To illustrate the complexity of ONE element of the solution we are referring to here, I include a fragment of the prompt as well as the generated responses. And I will just remind you that usually, at least several such actions are performed.

Integrating an LLM with a private task list remains a basic example of RAG, as we operate within a very closed area of information and possible commands. Despite this, there are a number of problems that we must address. The first one that comes to mind could be a user's request to update a task, to which the model responds by updating the wrong entry.

In this situation, one could argue that the model behaved correctly, but the expected result was not achieved. Such mistakes happen more often and are not always so obvious. Sometimes the model may misinterpret the user's query, and instead of adding a new entry, it could potentially update an existing one. Naturally, this is not an insurmountable barrier, but we certainly need to keep it in mind. Implementing additional mechanisms to verify the actions taken by the model becomes necessary here.

A different perception of reality, and a genius locked in a room.

LLMs have limitations on the amount of content they can process within a single query. This limit is constantly increasing and currently reaches 128,000–200,000 tokens, and according to Google's announcements, Gemini 1.5 raises the bar to 1 million tokens. This last value is about 700,000 words in English, so quite a lot.

Of course, we must also add the fact that the pricing for using LLMs usually depends on the number of processed tokens. These prices are significantly dropping, so over time this should become less significant.

However, even the example at the beginning of this post shows that the content limit is not our only concern, as ultimately, we will care about the effectiveness of the responses, which can drop even in the context containing only one article.

Here we also have various solutions available. One of them could be optimizing the prompt to control the model's behavior so that it uses the provided context more carefully. Not always will it be possible, and not without reason, people commonly say that even the best prompt won't handle a poor context.

Currently, it seems reasonable to ask how to create a good context. And although I cannot answer unequivocally, we can quite freely point out areas that need to be considered. Before we move on to them, you should know that usually dynamic context is built from fragments of content coming from various places. These fragments are searched with the help of various search engines, including vector databases.

• Ambiguity: a situation where two similar but unrelated pieces of information enter the context. For example, summaries of two books with the same title. If we do not clearly separate them from each other, we can be almost certain of an incorrect answer.

• Complexity: answers to some questions require reaching many different places or processing a large amount of content. For example, a question about financial results may require going through many different documents, which cannot all be included in the context at once. However, if the reports had previously generated summaries focusing on individual areas, they could be used to provide an answer.

• Contradiction: two pieces of information that directly contradict or overwrite each other may appear in the context. This can happen in the case of documents that change over time, or when loading two unrelated documents discussing the same topic but in an entirely different context.

• Isolation: practically any information provided to the model without additional context about its origin or different features can constitute noise that reduces the quality of the response. This is particularly important when we load data from various sources into the context.

• Organization: the above example concerning the origin of information is directly related to the way data is organized, which should be considered at the stage of searching or filtering. For example, when I ask about articles I have read on a selected topic, I would like the model to be able to indicate them, ignoring those that may be in my database, but I have not had the chance to familiarize myself with.

• Inaccessibility: if the context contains links, images, videos, or any attachments, the model will not have direct access to them. Then it is necessary to include descriptions of their content.

• Absence: this is a situation where the data we ask about is not delivered to the context. Such a situation can occur when searching for content we only consider the current user query, or on the contrary — we get distracted by too many documents, omitting some of those that are relevant.

• Reachability: if I now say "in my previous post," you can find it by going to Tech•sistence. LLMs do not have this capability. Therefore, examples of all references or situations when we share something but do not use keywords become incomprehensible to the model. I encountered this most in the case of video transcriptions, where terms like "I click here" were used. Therefore, when creating content for LLMs, we should either avoid such situations or take additional steps to help the model understand the message.

It is essential to bear in mind that currently, LLMs can only experience reality through text. Of course, we have Vision models at our disposal, which allow us to use images as well, but this does not solve the problem. The fundamental problem we face here is the complex "connecting the dots," knowledge of a broad context, or having information coming directly from our environment. We currently have to present all this to the model in some way so that it can effectively use the available information. The whole is also complemented by the content format, about which I wrote more in Plain Text is What You Need which I warmly invite you to read.

Limited Trust

Combining a large language model with our knowledge base comes with a series of limitations and challenges that are difficult to address in practice. To put it simply, the effectiveness of RAG is about 70-90%. This means that placing blind trust in them is not a good idea.

However, if we look at the examples of applications I use every day, it's challenging to question their usefulness. And it's worth considering that the actions AI currently performs for me can also be triggered without my involvement, through automation scenarios and scripts that work for me around the clock.

The purpose of this post was to draw attention to serious problems that are often completely overlooked and can lead to real consequences resulting from the improper use of AI, especially LLMs. At the same time, I wanted to present the possibilities that were recently out of our reach, but today make work easier and turn elements of everyday functioning into great fun, bringing with them real value.

Have fun,
Adam

It is estimated that in 2024 we will generate 120 zettabytes of data. It's difficult to even present this number in perspective, as it's hard to comprehend that it's approximately 328.77 quintillion bytes per day, or 328.77 million terabytes.

But that's beside the point.

What's important is comparing how much data we generate annually to previous years:

It's somewhat easier to grasp this in the perspective that various sources report that from the dawn of humanity to around the year 2000, mankind produced a total of 5 exabytes of data. Nowadays, we generate 328.77 exabytes every day. This means that every day, we produce 65 times more data than the entire human race did in 2000 years.

On a side note, it's fascinating that currently, we're trying to pack all this data into one small data jar. Essentially, a large language model (LLM), such as the one that powers ChatGPT, which contains a substantial portion of the internet, is essentially two files. One file contains the program execution instructions, which aren't overly complicated and at most a few hundred lines of code. The second file is the output file of all the magic and the result of training the model - in short, gigabytes of zeros and ones.

By the way, this is quite an expensive game. It's exactly for this that Sam Altman from OpenAI is currently fundraising for a project that may consume, a trifle, $7 trillion dollars.

But what does all this have to do with the competencies of the future?

We're slowly getting there. This means that stranded on a deserted island, with the knowledge of how to build a computer and a solar battery, and the aforementioned two files, you'll probably be able to recreate most of humanity's intellectual achievements.

Can you imagine discussing such topics with your grandparents, or even your parents?

I'm getting at the fact that most things that were relevant a few years ago in the context of education, or what is worth betting on as competencies that could potentially give us the opportunity for a well-paying job or social advancement in the future, are becoming obsolete right before our eyes.

It's hard to advise someone on choosing a field of study these days. Because after all, studies that used to offer certainty and prestige, such as law studies, may soon be completely useless. First of all - absorbing vast volumes of legal knowledge and memorizing them seems, to put it mildly, a waste of time in the era of Perplexity, and soon also specialized models, trained for example on the entire database of judgments and legislation.

So how to cope in the rapidly changing technology space that surrounds us, which is changing faster than ever due to AI? What skills to acquire and what will actually be important in some time?

First of all - it is difficult to give a clear answer to this question. Will trends such as VR and AR, Blockchain, Cloud Computing, Data Science, which already seem ultra-modern and promising, stay with us for longer? We can't predict that at the moment. It's possible that AI will change our reality so much that even these clear trends will not survive the confrontation with a specialized model.

One thing is certain. The way we perceive our base knowledge will change significantly. This is about the possibilities of an individual who, with access to information with the help of AI, will be able to instantly reach most information from any field.

In this context, any textbook knowledge and memorization of detailed information seem completely unnecessary. It seems much more important to acquire information more superficially, but from different fields.

Until recently, we lived in times of narrow specialization. We are already seeing that the best experts are beginning to draw from different fields of knowledge and apply it to a specific case, going far beyond the area of their narrow specialization. In practice, specialized AI models, which will soon start appearing on the market, will be able to perform any specialized work better than humans. The more "processed" it is, the easier it will be for the model to navigate around specific rules.

That's why an interesting way of testing models is to do it in an environment of closed rules. For example, AlphaGo, using a strictly defined set of Go game rules, was able to defeat any human after a certain number of iterations. The same goes for chess. Within defined rules, it's easy to train a model that plays against itself, learning in this way, or to compare the effectiveness of two different models by inviting them to a joint game.

This is a good example of how specialization can translate into practical applications and implementations of AI in professions such as lawyers, accountants, architects, doctors, or builders. This also applies to professions related to technology and programming. And even primarily, as programming languages are even more natural environments for large models than the languages used by people. So, one can assume that soon models will be better than the best programmer in the world and will work with incomparable efficiency.

Subscribe now

However, this does not mean that I predict the decline of the programming profession. The effects of such a transformation will simply be twofold:

Firstly, there will be a huge democratization of knowledge and technology. People who previously could not create a program, website, or mobile application will be able to do so with the help of interaction with the model. In this way, we will get enormous added value for the world, where only 4% of people are able to program anything. Soon this value will be close to 100%. Access to the internet and the ability to acquire knowledge, not where you were born, will be the determining factor of what you can achieve.

Secondly, specialists will gain incredible assistance at work, where AI will perform most of the tasks such as writing code, which currently had to be done manually. The programmer will use his specialization and knowledge of technology limitations to guide the model to create a significantly better application than one that would be created by a person without specialized knowledge. In addition, these will be experts who will create the infrastructure for completely new tools and technologies that the rest of humanity will use.

The time saved by AI, such a person will use to gain more general knowledge, for example about interface design, and in this way will be able to deliver more complex tools, programs, and products.

We are slowly entering a phase where an individual will be capable of implementing projects that until now were carried out by huge companies and corporations. Like Sam Altman and , I am also sure that a single person will be able to build a billion-dollar enterprise.

If you think about it for a moment, this means that such a person will have to find their way in all the processes that are currently carried out by specialized departments. In addition to operational functions, this person will manage marketing, sales, product, customer service, and all other processes - of course, with the help of artificial intelligence and tools, especially no-code and low-code.

But this directly suggests that the current structure of companies and corporations, where each of the employees has some designated role, is a product of a system that will soon become a thing of the past. The competencies of employees will intersect more, and companies will focus on skills that can be described as M-shaped.