Five Principles for Joyful Observability

Real-World Lessons from Pairing on a Restaurant Rails App

Today, I’ll talk about a hands-on session coaching session with Petro Yakubiv, the founder-developer of a Rails-based restaurant app called Menutaria.

Over our two-hour Zoom deep dive, we tackled the big problem of how to add observability to a Rails system just beginning to get real users—and we didn’t just talk abstract theory, but rolled up our sleeves and doggedly tracked down the most sensible ways to see inside the app as it grows.

What follows are the five underlying principles that guide my approach to observability, drawn straight from this conversation and my wider experience. Whether you’re working on a scrappy greenfield startup, a legacy codebase, or something in-between, my hope is you’ll take away concrete, repeatable strategies to make observability both useful and joyful.

Introduction: Observability Without Overwhelm

Trying to instrument a new Rails app - or tidy up a mess of tangled logging - is enough to leave my brain scattered in a dozen directions. Especially for those of us who are neurodivergent, it’s easy to get caught up in details that feel urgent but don’t actually make your life easier when your app is wobbling at 5pm on a Friday.

In this pairing session, I wanted to give Petro practical, sustainable ways to make his Rails app observable without falling into the trap of “let’s log everything and hope for the best.” The truth is, that’s both a waste of time and a recipe for future pain: cluttered code, slow answers, and stakeholders who are unimpressed because you can’t show concrete results.

Instead, the goal is to build up observability layer bit by bit, with intent - making every event you track pay its way by helping you answer a real question, now or down the line. Here’s how we did that, and the thinking behind each move.

Principle 1: Start With a Clear Question

The first and most foundational principle: Never start observability work by just adding logs or metrics everywhere.

Instead, always begin by asking: Which part of the app, if it broke, would cause real pain for users or the business?

For Petro’s restaurant app, this meant focusing on the absolute must-haves - “Can a customer view a menu after scanning a QR code? Are they able to book a table, call a waiter, or leave a review?” Working from that list, we avoid the endless project of “improve logging,” which usually fizzles into a morass of technical debt and confused developers.

This approach keeps stakeholders happy too. Rather than spending weeks “adding analytics,” you can show: “Here’s how we caught outages in payments, or saw failed menu scans instantly.” In practice, we wrote out the business flows as a numbered list, rock-bottom priorities at the top, and made sure every instrumentation change was about answering one of those core questions.

Principle 2: Value-Driven Instrumentation

Instrumentation is only useful if it drives action. Every metric, event, or trace you add should have a clear value - otherwise, it’s noise. If you’ve ever waded through logs only to find “something went wrong” with zero context, you know how infuriating that is.

In our session, I pushed Petro to think about this upfront. For example: why record a “menu viewed” event? Simple - if those events drop to zero, either something is wrong with the app, or with the restaurant’s business. When we found nil values in the controller (those safe navigators on @menu&.name and @table&.zone), it signaled a perfect opportunity to distinguish between a successful menu view and one with missing data, allowing for alerts to fire with the right context.

This discipline not only makes debugging easier, but reduces cruft for your future self and team. By asking, “If an error fires in here, will I have everything I need to actually fix it?” you end up with clean, actionable events - ones that map to how people actually experience your product.

Principle 3: Think in Events, Not Just CRUD

One of the biggest shifts in mindset I teach around observability: Focus on events (what happened and when), not just current state (what’s in the database right now).

Too many Rails apps obsess over CRUD actions and miss the fact that when something goes wrong - an error when booking a table, or the spinner that spins forever after a failed upload - it’s the event history that explains “why.” Petro’s original design had logs tied closely to analytics tools like Ahoy, but by reframing these as domain events (e.g., restaurant.menu_viewed, waiter.requested, order.placed), we unlocked far richer debugging.

This also naturally blends business analytics with technical observability. Imagine being able to break down events by “physical location” (was the QR scanned indoors, outdoor, from Instagram?), or grouping menu failures by which restaurant, or which table and zone. One structure powers both real-time debugging and boardroom insights. That’s the real power of events.

Principle 4: Make Use of the Fact Bucket

Debugging with a brain that sniffs out oddities but can’t always hang onto them? I’m right there with you. That’s why I teach - and use daily - the Fact Bucket: a scratchpad (could be a markdown file, sticky note, whatever makes you tick) where you jot down anything interesting or weird, as soon as you spot it, without breaking your flow.

In Petro’s app, for example, it was critical to note down that menus or tables could be nil, or that a booking might succeed with a disconnected zone. The act of writing facts down, without immediately diving into rabbit holes, preserves your focus while capturing all those “Hmm, I should check that later…” moments. Over time, reviewing the fact bucket helps you connect dots and tease out hidden patterns.

I keep mine open whenever diving deep into code, code reviews, or architecture chats. It’s saved my bacon more times than I can count, especially when production is wobbling and a seemingly-unrelated detail from last week turns out to be the missing link.

Principle 5: Design for Both Today and Tomorrow

Shipping fast is great - but observability means designing with both today’s problems and future flexibility in mind. This means building mechanisms that let you add new events or attributes easily, without entrenching yourself in technical debt.

During our pairing, we discussed how to model key domain attributes for events, name-spacing them so you can group by restaurant, menu, zone, or access point (indoor/outdoor/Instagram/etc.) as the system grows. Bank on the fact that new business logic will emerge (“oh, now we want to track menu scans by zone, or bookings by QR code origin!”) - so create your logging/events in a consistent, extensible way: nested hashes, semantic event names, and (when you’re ready) even schema validation for observability payloads.

This isn’t about building castles in the sky - it’s about a little up-front structure that saves hours of headache when the business pivots, or users try unexpected flows. The key is not to instrument *everything*, but to make it straightforward to instrument *anything you need*, when you need it.

Real-World Example: Debugging Menutaria’s Menu Scans

Let’s make this concrete. In Menutaria, the most urgent business goal was: “Do menu scans work everywhere - at the table, via a QR code on the door, or from a link on the web?” By listing the business flows and mapping each to a domain event menu.viewed, able.booked, etc), we made sure observability kept pace with what actually matters.

Next, we made our event payloads explicit: If a customer tried to view a menu but saw none, we logged that as a specific failure with context - was the menu missing, was the zone disconnected, what access point did they use? For successful events, we logged the attributes needed to answer “who, where, when, and what.”

We also proactively looked for patterns: What happens if all menus are accidentally disabled? Can the error be explained just by database state, or do we need to capture the chain of events? By dividing errors into “indoor/outdoor/no menu” and always including as much context as possible, we made debugging fast, automatic, and actionable - not just a guess in the dark.

Conclusion: Joyful Confidence in Production

Observability is about cultivating peace of mind, so that you (and your team) can fearlessly ship, debug, and grow your product without dreading the inevitable “Friday Wobble.” By focusing on the five principles above - starting with a question, tying every event to business value, thinking in events, using the fact bucket, and designing for future flexibility - you’ll genuinely see what’s going on in production, and quickly answer the questions that really matter.

I encourage you to grab a sticky note (or spin up a fact_bucket.md in your repo). Start jotting down your next round of critical flows, and ask: “What would I *really* want to know if this broke?” From there, joyfully instrument with intent.

If you want to go even deeper, or get a hand instrumenting your own app, please connect and DM me COACH on LinkedIn.