In any sane managed notifications service, this task would not come with any surprises. Yet “sane” and Azure rarely coexist in the same sentence. And so began a dark descent into pricing madness.

Setting up the resources

I’ve worked with AWS SNS and Twilio before. However, this was the first time working with Azure Notification Hub. As such, I felt like a “clickops” run to set up the hub for a development environment would be a good first step.

When opening the menu, there are a handful of options:

Perfect, it defaults to the Free Tier! I’m sure the rest of the defaults are fine, too. “Additional costs apply” next to a checkbox... that’s probably just a CYA clause for usage beyond the free limits. No big deal.

And with a couple clicks, boom, a new Notification Hub was born, ready to spam your phone with weekly deals – err, I mean totally valuable personalized content that transforms your shopping experience. Plugging in the Apple and Google keys was a breeze, and we quickly moved on to other infrastructure tasks for the project.

Maybe Azure isn’t so bad after all, we thought 🤔

Then the bill came.

Your monthly statement is ready

A few weeks later, Anthony, who monitors our cloud spend as part of a weekly cashflow review, noticed something unusual: Our Microsoft invoice went from $2.99 in July to almost $250 in August!

Our first thought was, “Oh, maybe this is a Microsoft 365 subscription trial we forgot to cancel or something.” But alas, opening the bill revealed a more grim truth: our Azure costs had skyrocketed. What the hell was going on?

Fortunately, Azure does have a fairly good cost explorer, which we were able to use to narrow down the culprit pretty quickly:

But… I thought we were on the free tier? Why was Notification Hub costing $240?

Finding the root cause

One more plug for Azure’s Cost Explorer. It seems the most pleasant thing about using Azure is how easy it is to find where your money’s being wasted. Anyway, in this case, we hopped into a different view that breaks down charges by service and product:

Availability Zones? For Notification Hub? What does that even mean?

To understand where we went wrong, we revisited the clickops process we followed, and sure enough, there it was, hiding in plain sight:

Hmm, yes, at this point, I would like to learn more 🧐.

Opening the link, you are taken to a page that helpfully documents the Availability Zones feature for Notification Hub. Great, I can confirm we don’t need it for sending weekly deals out to a few hundred regional users. If messages temporarily can’t be sent due to an Azure outage, we can easily redeliver them from the backend later by triggering the automated workflow built for that purpose.

Now, let’s try to find this pricing info… Oh wait, the price isn’t on this page. It looks like I need to go to the Notification Hubs pricing page to find out. And finally, lo and behold, we’ve found the costs!

So, uh, yeah, we did screw up, I guess. Whoops.

Did we say free? Our bad!

For me, this didn’t sit right. Why was that checkbox enabled by default on the Free Tier? Furthermore, what kind of user doesn’t care about an SLA, but is concerned about automatic failover of their push notifications, enough to to spend $350 / month for that peace of mind?

It felt like a pretty big oversight at best, and perhaps an intentional, dark pattern at worst.

After a little bit of research, I am leaning towards the latter. Apparently, in some regions, this feature is not only enabled by default; it can’t be disabled either. And why is the default via Azure’s REST API set to “Disabled”, but the checkbox is enabled by default on the Azure Portal?

Regardless of intentionality, I felt wronged. I opened a refund request, which, like cost analysis, was unusually pleasant compared to Azure’s actual cloud services. Later that afternoon, I got a response that our refund was being processed.

Hooray! Money saved! Time wasted! At least it’s over.

Not even IaC is safe

Surely, avoiding this would be easy the next time we set it up as IaC, right? In reality, it was just more transparent. Pulumi’s example code for creating an Azure Notification Hub enables the availability zone cost, once again segmented away from the Free/Basic/Standard price tiering.

Combined with the default SKU cost, a hasty developer copy-pasting Pulumi’s template code will opt in to $550/month. If we hadn’t already been snagged by the clickops implementation, it’s easy to see how this could have slipped through, too.

What’s more is the inconsistency between tools. When we eventually moved the project to Terraform, we noticed Terraform takes a more reasonable ‘opt-in’ approach to the added availability zone cost. Microsoft even leaves it disabled in their own Terraform tutorial. Still, we’ve learned not to let our guard down.

How you can avoid large bills with Azure

Don’t use it. Plain and simple.

I’ve spent nearly $2000 in the last year on development resources that were misconfigured. Not obscure niche services – these were marquee, bread-and-butter infrastructure offerings, like managed SQL databases and file storage mounts.

Every time, the expense came down to some dark UX nonsense like the single checkbox on Notification Hub, or an incorrect setting in Terraform/Bicep.

Indeed, refunds were sometimes issued, but reactively playing whack-a-mole with hidden costs creates unplanned work for my team to deal with, which is not free. In this case, aside from investigating the bill, we had to completely delete and recreate the Notification Hub Namespace, then update the backend services with new settings, all to avoid spending $3600 / year on something that has zero value to our client.

If you are in a position where you can make a decision on a cloud vendor, stay away from Azure. You or your developers will 100% misconfigure it. You will waste money reworking resources, contacting support for refunds, and hiring consultants to help clean up the inevitable mess that gets created.

If you’re already using Azure, good luck. I hope you know what you’re doing. Definitely set up some cost alerts, and perhaps learn some meditation.

This is a sad but necessary post:

We’ve halted current development and user testing with Limo. There are a number of reasons why it didn’t work out. I’ll dive into each one in this post, and touch on what’s next for the team.

Product Market Fit

We ran into a few classic product pitfalls. A lot of these stemmed from relying too much on my own experience and intuition. As a flimsy defense, there probably is at least some value in those. Limo was born originally to help automate repetitive, error-prone tasks like setting up and maintaining infrastructure in AWS, ones that I found myself doing over and over again.

In reality, the particular pain points of DevOps seem to vary greatly because of its broad scope and definition. For example, this is how GitLab defines DevOps:

DevOps is a combination of software development (dev) and operations (ops). It is defined as a software engineering methodology which aims to integrate the work of development teams and operations teams by facilitating a culture of collaboration and shared responsibility.

What is a development team? An operations team? What is the difference between dev and ops? Which one better describes the process of writing Terraform or other types of Infrastructure as Code?

The answers to these questions are not categorical. There’s no single standard, or even 2-3 competing ones, that can be applied easily to describe all DevOps practices. Naively, this lack of standardization seemed like the problem to solve. I’ve seen so much time wasted at nearly every job I’ve had dealing with bespoke CI/CD pipelines, infrastructure as code, ways of searching logs, and handling on-call duties.

Unfortunately, we should have looked for a problem to solve, tied to real potential customers or users. Instead, we built working prototypes covering way too much ground: for infrastructure setup. Deployments. Connecting to AWS, Netlify, and Github (and connecting them to each other). We had user-authorable “routines” to easily automate repetitive tasks. An optimizer workflow to discover savings opportunities. A troubleshooting one to analyze and correlate log searches. A Slack plugin and full Web UI. An MCP integration.

It was a mess. Even with four full-time, relatively experienced developers, and AI tools to help accelerate development (or perhaps because of them), the features’ demands exceeded our ability to support them.

Generative AI opens new possibilities, but it doesn’t change the need to focus on solving one problem really, really well, instead of solving many problems poorly.

A smarter approach would have been to narrow in on a more particular problem by engaging with several potential customers early, identifying common pain points, and seeing what solving these pain points was worth to them. An outreach-driven approach would have given us a starting point to find one thing to focus on and excel at. We had a list of over 100 friends and former co-workers to start with, many of whom had already expressed interest in learning more.

We waited to talk to them, which I deeply regret. I’ve only recently realized, irrationally, I thought that having demos in hand would be a prerequisite to being taken seriously, or getting good feedback.

This is just not true in hindsight. There are a decent amount of investors who will hear an early pitch without a product existing at all. The bar for talking to former colleagues is even lower than that.

Regardless, by the time we did ramp up outreach, and open up access, we had trouble getting more than a handful of users to try it! The feedback we got from developers was that they weren’t sure what to do with the product, or if it even solved a problem for them.

These combined factors meant we needed several more iterations to reach an MVP. With a few more months of runway, it’s possible we could have started tuning Limo toward serving a particular development community or industry. But our funding dried up before we were able to find out.

Lessons Learned

1. Solve a specific problem well, not a broad problem poorly

2. Outreach leads to problems worth solving

3. Don’t wait for a demo to be ready to engage

Funding Issues

If you ignore sweat equity, Limo was completely bootstrapped from the cashflow of Liminal Labs, a boutique consulting agency I started in 2023. Liminal Labs was/is niche and, realistically, not a huge cash cow. I don’t value our time very well, and we’re bad at marketing.

Anyway, to make Limo happen, we offboarded most of our clients. This crippled our revenue but freed up engineers.

Anthony, Liminal’s longtime employee, prepares a cashflow report we’ve reviewed weekly for the last two years. This meant we knew what we were getting into to some degree. We braced for impact, while simultaneously plotting a careful escape route toward a seed round.

Pretty much immediately, everyone faced reduced or no wages. We had to let go of our junior engineer (which I still feel terrible about). Those of us left were motivated and full of adrenaline. But the reality was sobering: We had started a countdown timer that couldn’t easily be stopped. We had to find some kind of revenue or funding by the end of August or it was over.

To us, that meant quickly making something we thought people would want to use, getting feedback knowing we’d be wrong, and iterating to close the gap. Using all these awesome AI-integrated tools like Devin and Windsurf would surely speed up development. Maybe along the way, we’d get lucky and meet some investors.

Indeed we did have some contact with a few investors. We also received deep, meaningful feedback on our pitch and demo from former friends and colleagues experienced in startups, venture capital, technical leadership, and more. These people were more than generous with their time. If you’re reading, thanks again for your help!

Everyone’s concerns were consistent and clear: competition from bigger players, no existing traction, a lack of a track record or founder with a previous exit, no particular expertise in AI or accolades in DevOps.

Undermining things further was our own lack of confidence that what we had was ready for customers. It’s hard to market or pitch something you’re not confident in yet. I can be confident, but it has to be authentic. I’d have been lying if I endorsed Limo in the state it was in back then. Early on, Devin had created a lot of slop for us to clean up. Quality sucked. It wasn’t worth paying for.

Improvements had to be made before it was ready. To compensate, we worked tirelessly on features and bugfixes. We sharpened our pitch deck and demo. We applied to startup accelerators over and over, occasionally receiving a polite rejection back. We eventually started getting help from Mort, an actual, really good product owner, which re-energized things for a bit.

But there’s a cold and cruel reality to cash and once we ran frighteningly low on it, we had to lean back into consulting. We attempted to rebuild a sales pipeline, got some small deals, and made it close to some bigger ones that never panned out. It felt wrong, it was wrong, but I wasn’t sure what else to do. We couldn’t skip paychecks forever.

All these side quests ultimately ended up being counterproductive. I spent critical time away from Limo when we needed more thought put into features, quality, and business development.

We didn’t meet our dream investors quickly enough, nor did we generate enough sales to buoy things. In the midst of our first batch of user testing, we ran out of cash to continue working on Limo while keeping Liminal Labs solvent. Game over 👾

Lessons Learned

1. To get investment, you need relevance, credibility, or traction

2. AI development tools bring down costs less than anticipated

3. Don’t get distracted with side quests

Learning Curve

As a software engineer, I consider myself to be more of a generalist. I’ve built apps on the web, mobile, and desktop; on touchscreen kiosks, room-sized imaging equipment, and Kinect-powered basketball courts deployed to shopping malls in China. I’ve worked on C# monoliths and Python microservices. Maintained Kubernetes clusters and multi-tenant cloud platforms. My specialties tend to follow the technologies and domains of the projects I’m working on, over any specific languages and tools I prefer working with.

While some had rougher learning curves than others, in the end, I’ve been able to become proficient enough to succeed in most types of software development. Troubleshooting and reverse engineering in particular are useful muscles one can build up to solve many kinds of technical problems. This is because in traditional software, you can find explanations for why something works (or doesn’t work). When stuck, you can dig through code, ask someone, or recreate a conceptual model of what’s going on to analyze it further. It may be incredibly frustrating and time-consuming to track down the reasons things work the way they do. It’s frequently irrational, but it’s at least deterministic. It’s possible.

When building Limo, which ultimately relied on GPT-4, techniques like reverse engineering and troubleshooting proved fairly useless. There were sometimes no explanations to be found, no one to ask, no concepts to latch onto. Models are inherently unpredictable by nature. More often than not, we were able to get the model working reliably for a narrow use case, such as queries similar to “list my repos”, which listed GitHub repositories by making a tool call using Octokit. The hype is not completely baseless. We found LLMs do perform well under certain conditions.

The “certain conditions” were the problem, however. In our development process, we were never really sure what those conditions were, as they were a moving target. Consequently, we lacked insight into how well Limo worked at any given moment. The UI and backend remained relatively stable thanks to early investments in a lightweight CI/CD pipeline and low-hanging fruit like unit tests and linting.

But when we’d integrate and test each other’s features, instead of merge conflicts, we encountered new types of collisions we’d never considered, something better described as “context conflicts”.

For example, the “list my repos” example before – under the hood, it was tied to a tool called list_github_repositories. Now imagine that in addition to GitHub, we also added support for AWS’ Elastic Container Registry through exposing a new tool named list_aws_ecr_repostories. What repos should Limo list now: your ECR ones, or your GitHub ones? Or do you mean a different private Docker registry, like Azure Container Registry (ACR)? Or perhaps a different source control provider, like Gitlab?

Well, all of that kind of depends on context. Like most chat-based interfaces, Limo’s natural language interface mostly puts context control in the hands of end users. They are the ones driving the conversation topic, not agents. Users’ inputs are often less predictable than LLMs’ outputs.

We couldn’t anticipate everything a user might do in a system prompt, or how to exclude context from some situations, but not others. Or when context changes were necessary at all. Or understand the degree to which any given context impacted results. Context management, we found, is both an art and a science, one that requires finer attention to detail and deeper domain understanding than traditional software features.

Prompt engineering partially mitigated context problems, as did techniques others have written about like tool loadouts. But we didn’t have a streamlined way of measuring how much any of these improvements actually helped, which massively slowed down progress.

In reaction to this pain, throughout development, we tried using various LLM observability tools to measure reliability and improve troubleshooting. Eventually we landed on Comet’s Opik after some trial and error.

Unfortunately, the learning curve on evals was tough for our team, who is more rooted in traditional forms of automated testing. Evals are not like those types of tests at all. Passing and failing is less clear – it’s more like an experiment in which you have to design, execute, interpret results, and draw your own conclusions about how the software performed.

We all agreed that tracing and evals were useful and important. They were worth investing in if we wanted to build a DevOps agent, where reliability and trust are implicit. And now that we’ve had plenty of hands on-time with LLM observability, it feels like an indispensable component of future AI projects.

But with few resources and little experience, these technologies kicked our ass. Despite repeated effort and attempts, we weren’t able to utilize them to improve Limo before running out of money.

Lessons Learned

1. Serious AI-native development has a significant learning curve

2. Traditional automated tests are insufficient to ensure quality for AI applications

3. LLM observability is crucial for predictable development cycles

False Confidence

Last but not least, we got bitten by something I’m still bitter about: AI development tools. They really fucked us, and we paid a lot of money for them.

I’m not talking about GitHub Copilot or ChatGPT. Those were fine for what they were: chatbots that occasionally wrote useful code snippets or helped work through an idea. At $10-30 / month, they served their purpose well.

I’m mostly talking about Devin, the AI “junior engineer” we hired for $500 / month. You can ask anyone who I met with back then – I was singing the praises of agentic coding, how every developer was doomed within a year. With a PR merge rate of ~75%, I was blitzing through features while still being able to handle clients. It seemed all it would take was a well-written Linear issue, and maybe a comment or two, for Devin to create working software.

It turns out, the progress was mostly a delusion. AI psychosis. Whatever you want to call it.

My weakness was blind faith. The code looked fine. Especially within the context of a single review. I barely noticed the mounting inconsistencies, hallucinations, and bugs that resulted from what I was approving and merging in small chunks. I was focused on tweaking the end result, not following the exhausting, boring details of implementation.

After a few months, we cancelled Devin in favor of Windsurf. By then, it was clear that trusting our autonomous engineer to go off and work autonomously was hurting more than helping. Windsurf was quite a bit better, as we all were able to “steer” the conversation toward actual useful code changes and command executions by watching carefully. Almost like switching to adaptive cruise control from Maximum Overdrive-style self-driving.

But even then, it was often too easy when stumped, impatient, or simply burned out, to phone things in. To vibe a prompt loosely resembling requirements into Windsurf’s agent mode and create a Pull Request. Reviewed by busy and unsuspecting teammates, AI slop often looked just fine, getting approved, merged, and deployed with subtle issues.

Another unsettling concern, both for Limo and AI usage in general: when problems were noticed by another developer, Pull Request authors often disowned their work, explaining how they had assumed Windsurf’s output to be correct without fully understanding why. With the end of our runway looming closer daily, how does one shrug off surprise and disappointment in a moment like that? Let alone discuss how to move forward, knowing the real work of critical thinking hasn’t started! Echoes of a recent MIT study stir uncomfortably in my head.

The end result was that we chased our tail a lot more than we had on any previous project. It was incredibly easy to get a false sense of confidence that the product was further along than it was. That the features would take less time to complete than they did. That the codebase was healthy and not full of hallucinated booby traps that would ensnare us later.

If we had relied less on AI tools, especially on the backend, I’m almost certain a version of Limo would be public and generally available today.

Lessons Learned

1. Trusting AI without verifying is a bad idea

2. AI tools provide negative value when used incorrectly

3. Human-in-the-loop is still critical for effective AI use

What’s Next

In the end, the Limo we built with the time and resources we had wasn’t compelling enough to raise money or land customers.

Now we are left to contemplate what to do with what’s left: the code itself, and the skills and experience we gained from trying to launch a product.

Short-term, as Liminal Labs, we’ll be posting more technical guides and content around developing reliable agents and AI apps, starting with evals. We hope to share some practical lessons for software engineers looking to take the plunge into their own AI engineering journey.

Personally, I’ve taken steps towards full-time employment in life sciences and shifted my interim consulting focus accordingly. I still believe in the long-term potential of generative AI, but I’m no longer willing to bet my entire livelihood on it.

To be honest, I’m relieved. I can go back to a relatively normal work-life balance. Aside from myself, Liminal Labs is taking on normal business again, with a team back from a fresh tour in the AI development warzone. Not everyone made it (no one is dead to be clear – just seeking employment). But the folks still around are eager to take on the next challenging and exciting project.

Let us know if you’ve got something in mind!

— Austin, Founder + CEO @ Liminal Labs

Time to come clean.

I’ve been vibe coding.1 I’ve been vibe coding…a lot.

It’s actually a problem. I have a bunch of house projects to do before the weather turns here in Oregon. Only have a couple of weeks of sun and yet here I am, spending too much of my time obsessively thinking about my side project. Sometimes you have to go where the energy takes you.

I figure I’m not the first to come to vibe coding but I also know that I won’t be the last. So I want to just drop a few simple(?) nuggets of wisdom I’ve learned in the process.

Note that I have intentionally not looked at anyone else’s vibe coding tips and tricks. I’ve gotten a few recommendations from a friend and probably gleaned a bit here and there from Twitter but I’ve mostly avoided looking for advice because I wanted to learn the process organically, from the ground up, tabula rasa. Which means I might have the exact same advice as everyone else and that advice is super elementary, or I may have come to radically different conclusions from others. Perhaps it’s just right! I’ll let you be the judge.

Lesson 1: From the 10k foot view, vibe coding really is a kind of magic.

My side project is a bit silly. I’m building a word game, the details of which I won’t go into but suffice to say it’s vaguely Wordle-like (in that you’re guessing a word) but more complex and difficult.

I’ve been thinking about building this game for years. It’s a pen and paper game that I’ve been playing with my father-in-law for at least a decade and for most of that time I’ve said to myself “it would be cool to turn this into an app!”.

But I also knew how much of a pain in the ass that would be. I’m a Product Manager by trade. I have a CS degree2 but I’ve never really coded professionally and I’m pretty rusty. I was pretty solid back in college but these days? I figured it would take me at least a month to learn javascript well enough to build the basic prototype-level app that I could play with my FIL.

With Claude doing the heavy lifting, I was able to build that prototype in two days. Not two 8-hour days –3 more like two 2-hour days.

I’m obviously not the first to say “hey, capable Product Managers can prototype with AI now”. At some workplaces, that’s become the standard. But it’s still worth remarking upon because it is remarkable!

Lesson 2: it’s … fun?

I won’t evangelize too much here. I’ll just say that I find, after I’ve spent some time working a feature and I finally nail it, that it feels good. It feels like the old days, back when I was a CS student and I cracked a hard problem. It feels like when someone on my team demos their work and it looks great, working just the way we want it to.

Outcomes over outputs: you don’t have to be the one coding to find the joy in the code. If you’re doing it right you’ll find yourself energized by the experience.

Lesson 3: It’s not just for Prototypes but you have to know enough to be dangerous.

Last week, I decided to:

• Restructure my app into logical concerns

• Start using a build process (Vite)

• Convert all of the code from Javascript to TypeScript

• Add in a test Framework (Jest)

• Build coherent tests for all the features I had thus far.

• Build a significant new feature, using a Test Driven Development approach4

For the engineers among you, you’ll probably recognize that list as something a little different than “prototype”.

To be sure, there’s work left to do. I need to clean up the CSS (I’m thinking Tailwind?) I need to integrate analytics (I’ve settled on Posthog, at least for now). Proper authentication will introduce a lot more complexity. But what was previously a prototype is now something a bit more “real”.5

Thinking back to the prototype, I doubt my father-in-law could do it, even with AI. “Computers” aren’t his strong suit. But my wife? She almost certainly could, despite having neither formal nor informal coding experience.

Yet, I don’t think she’d ever get past the prototype phase. Enough iterations and she’d have something that looked cool but was complete spaghetti underneath.6

Our paths diverge because I know enough to be dangerous. Enough time in the industry and you learn a lot through osmosis. I’d never heard of Vite before but I knew enough to ask the right questions and determine Vite passed the smell test.

The space is dynamic. Perhaps in six months, a year, two, a true neophyte could build production-ready code. Today, though, real domain expertise is still required. The domain is less about where to put the curly braces now but it‘s still about architectures and frameworks and trade-offs and best practices

Put another way: effective vibe coding is still the domain of developers, of PMs, of Engineering Managers, of people with sufficient exposure to coding practices to have a sense of how software development actually works.

Lesson 4: Embrace the inelegance

Still though, I’m not going to pretend what I have is bug-free, or perfectly maintainable, or anything like that. But to my eyes, it’s Good Enough. A real developer could come in, look through the codebase, understand what’s going on and make incremental changes, with or without AI assistance.

Note that this isn’t good. It’s Good Enough.

Does it solve the user problem? Have we tested all the branching logic? No linting errors? Does it meet our performance, accessibility, and security requirements. It may just be Good Enough.

Think about every enterprise codebase you’ve ever worked in? They pretty much all have some warts. Something was rushed, some area is in need of a refactor. That’s the nature of the game.

Let’s be honest with ourselves, we’re still making trade-offs here, of speed over elegance.

Lesson 5: Seriously though, it’s dumb a lot

LLMs have a bunch of failure modes. Many of them aren’t dramatically different than an overeager intern with too much time on its hands. The AI tends to lose the forest for the trees. It’s often overconfident. It occasionally rabbit holes into weird places. It’s often sycophantic. (That last one is new to me but the first three sound like 80% of the engineers I’ve ever worked with.)

Navigating these failure modes is the tricky part. It’s not always even clear you’re experiencing a failure mode. My heuristic is, “if it feels harder than it should be, it probably is”. So far this all seems more art than science to me. But perhaps I’m just too new to it to know better?

Lesson 6: Zoom in, and then out, and then zoom in again

One of the best ways I’ve found to handle both the forest-for-trees problem, and the overconfidence? Force the LLM to change its perspective. Or give its work to another model, and see what that model says.

I find myself frequently asking the LLM something like:

Ok, now step back. Consider <this aspect>. Have we built this feature in the best way possible? What trade-offs have we made, and what are the pros and cons of that approach?

Or I find myself asking a different LLM from the one I’m coding with:

Hey, I have <this code>. Read through and tell me what it does. Then, suppose I want to add features X, Y, and Z in the future. What architectural changes will I need to make, and when?

Repeatedly digging down to the line or the test level and then back, to the feature level and then back again, to the architecture level is incredibly helpful — for both me and for the LLM. It will frequently find fault with its previous work or reveal a trade-off I wasn’t aware of from its choice. (Here again, having a sense of modern coding practices is extremely helpful.)

Lesson 7: Context is king

The LLM has a tendency to be anchored to certain ideas, and it can be hard to break it out of once it’s settled there. As well, the longer you spend on a particular task with the LLM, the quality of output degrades, sometimes dramatically. Context rot, they call it, and I can assure you it’s real.

I don’t have great solutions besides “start a new conversation” and “choose your words/context wisely, especially when you open conversation” but the difference between “one-shotting” a problem and slogging through it for several hours can often come down to word choice and to knowing when to step back and start afresh.

What I will say is that a structured input, where you explain the purpose of your code snippet, the broader context, and the relevant roadmap right at the start of the conversation helps avoid architectural pitfalls.

Lesson 8: Ask questions first; shoot later

Whenever I’m planning a feature I never regret telling the LLM something to the effect of:

Explain how you intend to tackle this feature. Ask follow-up questions before you start making any code changes.

We iron out edge cases and avoid entire circuitous paths this way.

Lesson 9: Smaller problems are smaller

The last lesson is the easiest, but also, maybe the hardest? It won’t be new to anyone with software experience: when it doubt, break it down.

Sometimes you can say “hey, LLM, do these 5 things” and it will Just Work.

But often it doesn’t.

It’s why I think there’s a lot of value in TDD when vibe coding but the lesson is broader. When in doubt, break a big problem into smaller ones and take them one by one. You’ll get there faster in the end.

Conclusion

Thanks to Netlify, my little app deploys automatically whenever I commit/merge to main. It’s very convenient and really does feel like magic. And it means I have a real thing that my friends and family are beta testing, giving me super useful feedback.

It also means I have something that I can take to market. Not a prototype — a real app, today.

But it’s also little more than a bundle of HTML, CSS, and Javascript. I’m not ready to declare software engineering over. If anything, the concepts I learned in the industry are the reason it’s still going so well. Software engineering is alive and well, just different.

It’s early days for me vibe coding, for vibe coding as a thing. But I do think it’s worth scratching the itch, if only to see how you like it. You might learn something.

— Mort, Product @ Liminal Labs

I keep seeing this thing and I can’t get it out of my head.

It started with a tweet.

And then another:

But it really clicked when I saw this talk:

The key moment:

I had MenuGen basically working on my laptop in a few hours and then it took me a week [to deploy] because I was trying to make it real…

Software development is wildly different from just a couple years ago.

• Cursor for writing code.

• CodeRabbit for pull requests.

• Mintlify for documentation.

I wouldn’t say we’ve all become vibe coders but it’s fair to say that we’ve all become a little bit cyborg.

But DevOps? It still looks kind of … the same. And it’s still kind of a horrorshow?

This is the dirty secret of modern software development. We've gotten incredibly good at building things. Frameworks make UI development almost pleasurable. APIs are cleaner than they've ever been. MCP servers are…less clean but at least they’re an interface that generally work, most of the time.

Yet somehow, the gap between "it works on my machine" and "it works for actual humans on the internet" has become a chasm that swallows developer dreams whole.

The DevOps Gap

That gap between “built a cool app” and “deployed a cool app for the world to use”? It’s just a skill issue.

Don’t get me wrong. It’s a normal skill issue. I don’t know how accurate the old “10,000 hours” saw is, but to develop a skill you still need reps. And the brutal truth is Development and Operations, though we combined them (for very good reasons), they’re still two totally different disciplines.

They require different skills, different reps, different mental models. Even for experienced Devs, context switching between the two takes its toll.

Consider the humble web app.

• You write a Node app that connects to a database. Easy. Now you need to figure out VPC subnets, security groups, and IAM roles just to get those two things talk to each other in the cloud.

• Your app works fine until you get some traffic, then it falls over. Is it container limits? Database connections? Load balancer config? You're debugging infrastructure instead of code.

• You need to add some environment variables. Simple, right? Except now you've got secrets management, key rotation, and sorting out which service has access to what.

None of this is insurmountable. Nor is it rocket science. But it’s a big hunk of stuff that you have to deal with when what you really want to do is focus on your actual app functionality. It’s necessary stuff but it’s not the fun stuff.

If I’m a cyborg when I code, why can’t I be a cyborg when I deploy?

FE-as-as-Service is not Enough

Perhaps you’re thinking: “aren’t there are a bunch of simple deployment tools — Netlify, Vercel, Railway?”

Sure, if you’re building a marketing site, they work great. I use them myself.

But have you ever tried to build a production app with one of those things? Something that requires user state or auth or a real live database? They can’t actually do it. At best, they’re an abstraction layer for deployment. Good for simple use cases but woefully insufficient for a full stack engineer who needs to get the job done.

The Agentic Shortcoming

Andrej, AK, seconds_0: they all see this too. It’s why they’ve taken to Twitter to rant about it.

They have AWS accounts; they have git repos. It all can be made accessible to the context window. Shouldn’t an agent be able to just do the thing? AWS may be a bit inscrutable to navigate but folks muddle through it every day. Why can’t the LLM too?

I haven’t seen a satisfactory answer. My best guess? The LLMs have a skill issue.

Specifically, it’s that errors with agents compound over time. And what is DevOps but a dependent chain of actions? You need to sort out the IAM policies before you set up the VPC. You need the VPC before you provision the container. The container needs to be running to deploy the code. A general purpose agent can’t get through all these steps with any reliability. And no one wants to be on the hook for someone else’s production nightmare.

Yet I caution anyone from thinking that the problem is unsolvable. Sure, it’s a lot of complexity. But break down that complexity into atomic units, solve them one by one, and create agents that can do the orchestration? You just might solve that skill issue after all.

No one’s cracked it quite yet. But it feels inevitable.

And when someone does finally solve this, it will be a big deal. It will make deployments easier, sure, but more importantly, it'll fundamentally change what kinds of projects are worth starting. Today many good ideas die in the 'but then I have to deal with the DevOps' phase. Fix that and it should unleash a ton of pent-up creativity for projects that weren't economically viable, too small or boutique to productionalize.

Personally? I'm looking forward to that most of all.

— Mort, Product @ Liminal Labs

Subscribe now

This is Liminal Labs, the blog.

Expect insights on DevOps, on AI, and on building products with AI, in this feed and in the very near future.

We hope you like it.

Subscribe now