Over our two-hour Zoom deep dive, we tackled the big problem of how to add observability to a Rails system just beginning to get real users—and we didn’t just talk abstract theory, but rolled up our sleeves and doggedly tracked down the most sensible ways to see inside the app as it grows.

What follows are the five underlying principles that guide my approach to observability, drawn straight from this conversation and my wider experience. Whether you’re working on a scrappy greenfield startup, a legacy codebase, or something in-between, my hope is you’ll take away concrete, repeatable strategies to make observability both useful and joyful.

Introduction: Observability Without Overwhelm

Trying to instrument a new Rails app - or tidy up a mess of tangled logging - is enough to leave my brain scattered in a dozen directions. Especially for those of us who are neurodivergent, it’s easy to get caught up in details that feel urgent but don’t actually make your life easier when your app is wobbling at 5pm on a Friday.

In this pairing session, I wanted to give Petro practical, sustainable ways to make his Rails app observable without falling into the trap of “let’s log everything and hope for the best.” The truth is, that’s both a waste of time and a recipe for future pain: cluttered code, slow answers, and stakeholders who are unimpressed because you can’t show concrete results.

Instead, the goal is to build up observability layer bit by bit, with intent - making every event you track pay its way by helping you answer a real question, now or down the line. Here’s how we did that, and the thinking behind each move.

Principle 1: Start With a Clear Question

The first and most foundational principle: Never start observability work by just adding logs or metrics everywhere.

Instead, always begin by asking: Which part of the app, if it broke, would cause real pain for users or the business?

For Petro’s restaurant app, this meant focusing on the absolute must-haves - “Can a customer view a menu after scanning a QR code? Are they able to book a table, call a waiter, or leave a review?” Working from that list, we avoid the endless project of “improve logging,” which usually fizzles into a morass of technical debt and confused developers.

This approach keeps stakeholders happy too. Rather than spending weeks “adding analytics,” you can show: “Here’s how we caught outages in payments, or saw failed menu scans instantly.” In practice, we wrote out the business flows as a numbered list, rock-bottom priorities at the top, and made sure every instrumentation change was about answering one of those core questions.

Principle 2: Value-Driven Instrumentation

Instrumentation is only useful if it drives action. Every metric, event, or trace you add should have a clear value - otherwise, it’s noise. If you’ve ever waded through logs only to find “something went wrong” with zero context, you know how infuriating that is.

In our session, I pushed Petro to think about this upfront. For example: why record a “menu viewed” event? Simple - if those events drop to zero, either something is wrong with the app, or with the restaurant’s business. When we found nil values in the controller (those safe navigators on @menu&.name and @table&.zone), it signaled a perfect opportunity to distinguish between a successful menu view and one with missing data, allowing for alerts to fire with the right context.

This discipline not only makes debugging easier, but reduces cruft for your future self and team. By asking, “If an error fires in here, will I have everything I need to actually fix it?” you end up with clean, actionable events - ones that map to how people actually experience your product.

Principle 3: Think in Events, Not Just CRUD

One of the biggest shifts in mindset I teach around observability: Focus on events (what happened and when), not just current state (what’s in the database right now).

Too many Rails apps obsess over CRUD actions and miss the fact that when something goes wrong - an error when booking a table, or the spinner that spins forever after a failed upload - it’s the event history that explains “why.” Petro’s original design had logs tied closely to analytics tools like Ahoy, but by reframing these as domain events (e.g., restaurant.menu_viewed, waiter.requested, order.placed), we unlocked far richer debugging.

This also naturally blends business analytics with technical observability. Imagine being able to break down events by “physical location” (was the QR scanned indoors, outdoor, from Instagram?), or grouping menu failures by which restaurant, or which table and zone. One structure powers both real-time debugging and boardroom insights. That’s the real power of events.

Principle 4: Make Use of the Fact Bucket

Debugging with a brain that sniffs out oddities but can’t always hang onto them? I’m right there with you. That’s why I teach - and use daily - the Fact Bucket: a scratchpad (could be a markdown file, sticky note, whatever makes you tick) where you jot down anything interesting or weird, as soon as you spot it, without breaking your flow.

In Petro’s app, for example, it was critical to note down that menus or tables could be nil, or that a booking might succeed with a disconnected zone. The act of writing facts down, without immediately diving into rabbit holes, preserves your focus while capturing all those “Hmm, I should check that later…” moments. Over time, reviewing the fact bucket helps you connect dots and tease out hidden patterns.

I keep mine open whenever diving deep into code, code reviews, or architecture chats. It’s saved my bacon more times than I can count, especially when production is wobbling and a seemingly-unrelated detail from last week turns out to be the missing link.

Principle 5: Design for Both Today and Tomorrow

Shipping fast is great - but observability means designing with both today’s problems and future flexibility in mind. This means building mechanisms that let you add new events or attributes easily, without entrenching yourself in technical debt.

During our pairing, we discussed how to model key domain attributes for events, name-spacing them so you can group by restaurant, menu, zone, or access point (indoor/outdoor/Instagram/etc.) as the system grows. Bank on the fact that new business logic will emerge (“oh, now we want to track menu scans by zone, or bookings by QR code origin!”) - so create your logging/events in a consistent, extensible way: nested hashes, semantic event names, and (when you’re ready) even schema validation for observability payloads.

This isn’t about building castles in the sky - it’s about a little up-front structure that saves hours of headache when the business pivots, or users try unexpected flows. The key is not to instrument *everything*, but to make it straightforward to instrument *anything you need*, when you need it.

Real-World Example: Debugging Menutaria’s Menu Scans

Let’s make this concrete. In Menutaria, the most urgent business goal was: “Do menu scans work everywhere - at the table, via a QR code on the door, or from a link on the web?” By listing the business flows and mapping each to a domain event menu.viewed, able.booked, etc), we made sure observability kept pace with what actually matters.

Next, we made our event payloads explicit: If a customer tried to view a menu but saw none, we logged that as a specific failure with context - was the menu missing, was the zone disconnected, what access point did they use? For successful events, we logged the attributes needed to answer “who, where, when, and what.”

We also proactively looked for patterns: What happens if all menus are accidentally disabled? Can the error be explained just by database state, or do we need to capture the chain of events? By dividing errors into “indoor/outdoor/no menu” and always including as much context as possible, we made debugging fast, automatic, and actionable - not just a guess in the dark.

Conclusion: Joyful Confidence in Production

Observability is about cultivating peace of mind, so that you (and your team) can fearlessly ship, debug, and grow your product without dreading the inevitable “Friday Wobble.” By focusing on the five principles above - starting with a question, tying every event to business value, thinking in events, using the fact bucket, and designing for future flexibility - you’ll genuinely see what’s going on in production, and quickly answer the questions that really matter.

I encourage you to grab a sticky note (or spin up a fact_bucket.md in your repo). Start jotting down your next round of critical flows, and ask: “What would I *really* want to know if this broke?” From there, joyfully instrument with intent.

If you want to go even deeper, or get a hand instrumenting your own app, please connect and DM me COACH on LinkedIn.

Debugging a live production issue, tracing nils through Rails controllers, or untangling a domain model can leave my brain scattered in twenty directions. Especially given I’m neurodivergent. I spot something odd — a controller skipping a check, a field that’s nil when it shouldn’t be — and I think, “I’ll come back to that.”

But I don’t. And what’s more, it distracts me.

And later, when everything collapses, I would think:

Oh yeah, I remember - I saw something weird.

I was diagnosed with ADHD in adulthood. The diagnosis made a lot of things click — especially why I’d dive deep into debugging but forget half the interesting things I noticed along the way.

Debugging with ADHD means balancing hyperfocus with distractibility. If I pause every time I see something "interesting," I lose momentum. But if I don’t note it down, it's gone. That tension led me to a system I now teach every engineer I coach.

The Solution: The "Fact Bucket"

I call it the Fact Bucket. It's a simple idea:

• Capture interesting observations immediately

• Don’t interrupt your current flow

• Review the patterns later

This isn’t a formal tool. It can be a fact_bucket.md in your repo. A Notion doc. A sticky note. Whatever works. The key is — when your Spidey sense tingles, you log it and keep moving.

This fact bucket can be reviewed regularly. As I move through a debugging session with a production defect, I record facts that are pertinent to the issue and facts that are not. Either way, it allows me to build a picture of the app in my head.

Case Study: Spotting Nils & Domain Inconsistencies

Recently, I gave a coaching session with Petro Yakubiv on his restaurant app built in Rails. Customers were scanning QR codes to view menus. Sometimes they’d see... nothing. But no errors were thrown. Petro wanted to know how to get started with observability.

Inside the controller, we spotted safe navigators like @menu&.name and @table&.zone. That meant @menu or @table could be nil.

I noted this in the fact bucket: “table and menu possibly nil – investigate default paths and fallbacks.”

Later, it turned out if a restaurant admin disabled all menus by accident, customers would land on a menu-less screen. No crashes. Just silent failure. That allowed us to come up with ways of mitigating against these kinds of failures by logging relevant events.

Using The Fact Bucket With Steps To Observable Software

Below is the process I use for all my debugging.

I’ve evolved it over years of exploration with real production systems.

A quick recap:

1. Question - Form a hypothesis about what might be going wrong and form a specific question you want answered.

2. Decide - Figure out what data can you collect that answers the question.

3. Build - Create instrumentation in code to collect the data

4. Use - Logs, graphs, traces, use your observability tool to explore the data and ask the question

5. Reflect - An opportunity to reflect on your progress and next steps. What have you learned? Are you still going in the right direction? What’s the next step?

How does the fact bucket work at each step?

1. Question - evolve the next question or hypothesis using the fact bucket

2. Decide - not relevant

3. Build - not relevant

4. Use - collect facts in the bucket

5. Reflect - review the bucket and use it to help you decide what to do next

The fact bucket works beautifully with the Steps To Observable Software - it keeps me on track and focussed on the job at hand.

Broader Applications: Code Reviews, Architecture, Client Calls

I now use the Fact Bucket method everywhere:

• During code reviews, I jot down odd patterns instead of derailing the main feedback.

• In architecture discussions, I log confusing constraints to clarify later.

• On client calls, I capture “that weird thing they said” without breaking the conversation flow.

It's the perfect buffer between noticing and acting.

Tools and Techniques: Keeping Your Fact Bucket Simple

You don’t need a system. Just habits.

• Use bin/fact-bucket to open a Markdown file in your project

• Create a GitHub Gist or Notion page called 🧠 Fact Bucket

• Use VSCode’s TODO extension and prefix with FACT: instead of TODO:

The best tool is the one you’ll use without friction.

Turning Cognitive Challenges into Debugging Superpowers

My forgetfulness isn’t a flaw — it’s a clue that my brain is noticing more than it can handle at once. The Fact Bucket turns that chaos into leverage. I start to spot patterns others miss. I connect threads across time.

Next Steps

1. Sign up for free coaching

I’m currently offering free observability coaching sessions of 90 minutes.

I’ve got 5 slots available a month for a limited time.

DM me on LinkedIn with COACH and I’ll give you more details.

2. Join the waitlist for my course

I’m preparing a real world, hands on practical course on observability and the first module will be FREE.

DM me on LinkedIn with COURSE and I’ll let you know when it’s out.

Structured logging is the application of a consistent format to logs, allowing them to be treating as data sets which can be easily searched and analysed, giving insights into application behaviour.

Structured vs. Unstructured Logs

Unstructured logs have messages in plain text containing all log data in a single string.

Structured logs record log events in a standardised, machine-readable format with key value pairs, usually in JSON format.

Structured Logging Example

Unstructured Logs in Rails

In the code:

job = ... 
Rails.logger.info("Performed #{job.class} in #{job.duration}")

In the logs:

[2022-04-01 14:35.0154] [Sidekiq] Performed Mailer::PasswordReset in 1436.451

Most traditional logs look like this. It stores unstructured data. Any log data in plain text can go in the message. It's not easily parsed by machines - we need regular expressions.

😡 Unstructured logs are difficult to work with.

Structured Logs in Rails

In the code:

job = ...
Rails.logger.info(
  message: "Performed #{job.class} in #{job.duration}",
  event: {
    name: "app.job.performed",
  },
  code: {
    namespace: job.class,
  },
  duration: job.duration,
  messaging: {
    active_job: {
      adapter: {
        name: job.adapter
      }
    }
  }
)

This outputs the following JSON in the logs:

{
  "level": "info",
  "message": "Performed Mailer::PasswordReset in 1436.451",
  "event": {
    "name": "app.job.performed"
  },
  "code": {
    "namespace": "Mailer::PasswordReset",
  },
  "duration": 1436.451,
  "messaging": {
    "active_job": {
      "adapter": {
        "name": "sidekiq"
      }
    }
  }
}

The code for structured logging looks more unfamiliar. Note how the log message is separated. We get contextual information. We can detect patterns in our logs. The log data has a consistent format in key value pairs.

😍 Structured logs are easy to parse, search and aggregate.

What can I do with Structured Logs vs Unstructured Logs?

Structured logs allow engineers to search and aggregate more easily.

What use cases do structured logs support?

Search for Exact Match of a Single Field

Example: All Mailer::DeliverPasswordReset jobs

- Structured Logs Difficulty: Easy

- Unstructured Logs Difficulty: Easy

Search for Substring Match of a Single Field

Example: All `Mailer::*` jobs

- Structured Logs Difficulty: Easy

- Unstructured Logs Difficulty: Medium

Search for Exact Match of Two Fields

Example: All jobs performed for `Mailer::DeliverPasswordReset` jobs

- Structured Logs Difficulty: Easy

- Unstructured Logs Difficulty: Medium

Search for Equality Conditions

Example: Jobs performed with a duration of over 1 second

- Structured Logs Difficulty: Easy

- Unstructured Logs Difficulty: Impossible

Search for Aggregations (Average, Minimum, Maximum, P99, P95, P90)

Example: Jobs performed with a P95 over 3 seconds

- Structured Logs Difficulty: Medium

- Unstructured Logs Difficulty: Impossible

Group by a Field

Example: Graph duration of jobs grouped by job class

- Structured Logs Difficulty: Medium

- Unstructured Logs Difficulty: Impossible

Benefits of Structured Logs

Structured logging has allowed me to:

• Analyse and understand my application behaviour

• Resolve production incidents quickly

• Supply critical business intelligence to other departments

• Aggregate performance data - easily graph percentiles, averages, minimum and maximums

• Create visuals like graphs inside my observability tool

• Diagnose problems 10-100 times faster than with plain text log data

Structured Logging Best Practices

Here's what's worked for me over the years:

• Select a library that supports structured logging and integrates with your web framework

• Establish a consistent format for logs across all services - OpenTelemetry is valuable for this

• Use unique identifiers to trace requests across multiple services

• Convert unstructured logs outside the application into structured logs (apache, nginx, cloud provider)

Structured Logging Libraries

Depending on the tool and framework, here are some logging frameworks that implement structured logging in your software system:

Javascript - winston, pino, bunyan

C# - serilog, nlog

Java - log4j, tinylog

Python - contextual_logger

Ruby - semantic_logger

Implementing Structured Logging

1. Set a specific goal - example - "why are background jobs failing?"

2. Initial goal - focus on migrating existing unstructured logs to a structured format

3. Deal with existing dependencies - do engineers use unstructured logs or are there other systems that depend on this format?

4. Select a library that supports structured logging

5. Patch framework to use the structured logging library chosen above

6. Test logs locally - keep existing format so as not to confuse other engineers

7. Integrate and test with observability tool in staging environment - are structured logs coming through? Are they being parsed correctly?

8. Deploy changes to the log data

9. Use structured logs and list defects - focus on the specific goal - what’s missing or not in the right format?

10. Improve structured logs and redeploy

Use Cases for Structured Logging

Structured logging is great for debugging, troubleshooting, monitoring application performance, auditing security and compliance requirements, and more.

It is particularly useful in complex distributed systems where the log entry needs to be standardized and correlated.

Structured logs can be used to feed log data into a log management tool that allows for querying, correlating, and visualizing log data across all dimensions.

Working with Structured Log Data

Use a log management tool that allows JSON data to be sent.

For unstructured text data from other systems use log parsing tools to transform into structured data with a standard format.

Establish a unified schema for attribute names and values across all log sources to facilitate seamless correlation and cross-analysis of log data.

Overcoming Common Challenges

One of the main challenges of structured logging is ensuring all application logs are standardized and use the same structured format. For this I'd suggest OpenTelemetry's semantic conventions.

Another challenge is the potential performance overhead due to the additional formatting, but modern logging libraries are highly optimized.

Existing unstructured logs from systems with no control over the logging framework can be difficult to work with. We can reformat into a consistent log format using an ingestion pipeline or custom scripts.

Real-World Example of Structured Logging

We received a Slack alert - "log volume is projected to breach quota".

This was a monitor we'd set up in our observability tool to stop cost overages. A lot of structured data was being sent into our logs. That's strange.

Question - when are the number of structured logs spiking?

Looking into log analysis tool, we could see a huge spike in log levels at 3:17pm.

Question - what's generating those logs?

Grouping the structured logs by event name, we see the majority of these events are job.performed.

Question - what jobs are being performed?

Filtering by event name of job.performed, then grouping by job class we see the culprit - GeoDecodingJob

Question - why are there a surge of GeoDecodingJobs?

Graphing the number of retries of GeoDecodingJobs, we can see that these jobs were repeatedly being retried and all the retries were mounting up

Question - why were the jobs retrying?

The API was down for an hour and so all our jobs were retrying.

Action - tweak exponential backoff algorithm of the background job to be more lazy so the same number of jobs wouldn't mount up in the same way again.

Fixed!

With unstructured log data we wouldn't have been able to answer the first question, let alone the others.

Unlike unstructured logs, any data we've recorded in our instrumentation is available to search, sort, filter and group on.

Conclusion

Structured logging is part of essential best practices for any team that wants to improve their log management and analysis capabilities.

By following best practices and using the right tools, companies can overcome common challenges and reap the benefits of structured logging.

With structured logs, I've gained valuable insights into the performance and health of my application, and I've made data-driven decisions to help the software development process.

The following is based on true story. Names have been changed to protect the innocent.

I'm on call. It's a Friday afternoon and I get a Slack notification:

Okay. Looks like password reset emails aren't working:

OK fine, let's go to the logs:

And search them for the job performed event:

Not that helpful. Well, let's check the code:

Code looks fine. Which doesn’t really help.

Meanwhile, our CTO is getting anxious:

People are taking to Twitter and complaining. Nobody can get into their accounts:

I pair with a colleague and we add error logging.

Whilst waiting for the app to deploy, the issue goes away:

We'll do it later. Put it on the backlog:

Might be a while until we can get to the issue. It’s just a one-off incident though. Right? Right.

Two Months Later

Oh no, it's happening again:

Good job we added the extra logging! We'll search the logs:

Nothing. Open the Sidekiq admin dashboard:

We can see jobs processed… which doesn’t really help us.

At BiggerPockets, we found ourselves in this vicious cycle.

Engineers would deploy ad hoc logging, we would deploy the app and wait for the incident to happen again.

The incident happened… but we didn’t have enough data to resolve it.

We were always on the back foot - always being reactive.

The Solution: Steps to Observable Software

I came up with five simple steps to understand our Rails app better in production.

I call it Steps to Observable Software or SOS.

The steps are:

1. Question you want answered

2. Decide the data

3. Build instrumentation

4. Use the graphs

5. Improve

In a later post I’ll detail out these 5 steps.

We've been around the SOS cycle many times at BiggerPockets - and here’s what it gives our team.

Same Outage… With Observability

It's Friday morning this time, not Friday afternoon.

I'm still on call, but I now get a notification from Slack.

This is not a customer, not somebody in support.

This is our monitoring software telling me the queue has breached it’s agreed queue limit.

This is before any issue has happened. No customer has complained yet.

We click the link in the notification and we see this graph:

The blue line is rising fast.

Hovering over it, we can see it’s the within_five_minutes queue:

The process starts by asking a question.

Question: Which jobs are taking most time?

We search for job.performed within the same queue, group by code.namespace (job class), sum up the durations and hey presto:

Answer: Analytics::UpdateUserVisitsJob.

Question: What's enqueuing the Analytics::UpdateUserVisitsJob?

We’ll group by http.resource which is a combination of the controller and action:

Answer: profiles#show

Question: What IP is hitting this controller?

Let’s group by IP and show as a pie chart:

Answer: 15.145.1.3

We’ve spent less than 5 minutes asking questions of our observability tool and we understand what’s going on.

Let's recap.

• A server at 15.145.1.3 is scraping our app

• They’re hitting the show action of the profiles controller

• This floods the queue with analytics jobs

• That delays password reset emails.

The fix

Getting all that data was so quick and easy.

Now we understand what the app is doing, we can block the IP address.

Then go back into the graphs to check on the fix:

OK! The queue is recovering. Done.

No customer impact. Minimal disruption. Easy to debug. Easy to understand.

Feelings, Revisited

How does this feel?

As an engineer, it feels incredible.

I have the tools to ask any question of my app and get the results in seconds.

I can truly understand my app in production.

With this way of working, I’ve seen a common pattern.

I deploy a feature and I think I know how it’s working.

Then I go to the logs, do some queries, and find out how it’s actually working.

I often find myself thinking “ah… I didn’t expect it to do that…”.

This gets addictive. It also means I start to veer off asking other unrelated questions.

“I wonder why the traffic spikes on a Wednesday at midday?”

“I wonder why the vast majority of users are on iOS at 3am?”

“I wonder why the database keeps timing out?”

Results

Our downtime has dropped by 98% in 18 months.

We fixed 83% of our 500 errors.

Finally, we understand our Rails app in production.

Want Help?

Not every team has these kind of superpowers.

I’m holding a set of group coaching workshops.

$150 for a solo engineer and $2000 for your whole team.

Designed for Senior and Lead Rails Engineers.

Sign Up To Workshop

There, I said it.

Normally I'm a lot more nuanced about things.

Normally I don't swear.

But today?

I've got a bit more time for these things.

The idea that you'll be able to understand your app if you implement “the three pillars”?

Ridiculous.

It's also a fiction that plays well into the hands of certain vendors.

Three pillars?

What about three chances to charge your company for the same data?

Ever seen an invoice from an observability vendor and had that sinking feeling in your stomach?

“What? How much?"

You look down the itemised bill…

Logs… OK… expensive but… I guess…

Traces… OK… per host costs… I see… hmm…

Metrics… holy cow… OK…

Then you add it all up.

It comes to an eye popping number.

And you’re spending it month after month.

Not one time. Every. Single. Month.

Five figures. Six figures.

That's not even the worst of it.

There's not even three pillars any more.

What about the whole plethora of other "pillars" that never get mentioned?

RUM? More money.

CI? More money.

Incident management? More money.

Error reporting? More money.

And yes, these things all can link between one another.

But honestly, having used these links? They're not that smart.

They require you to add some obscure tag into every log and ever metric and every trace and every RUM event and...

...JUST STOP.

What the three pillars are...

...is an expensive set of SILOs.

And that makes sense. The clue is in the name. It's a pillar.

The three pillars don't really communicate with each other.

Not pillars. Data types.

These are three data types.

They're the raw ingredients of observability.

And that's not all - I'd argue they can all be derived from a single data type

And that's traces.

What can we do instead?

Well...

For any observability data to be useful, it has to have these following attributes:

1. Event

2. Wide

3. Context aware

Here’s more information:

1. Event

An event is great.

You can aggregate events in any way you would like.

Events can have a name.

Each event can have many attributes.

You can count the uniques of any attribute of an event.

You can group by attributes.

Events are awesome.

2. Wide

The event must have a ton of attributes. The more the better.

Because ultimately at 3am when your site is down and you have to get it back up...

...when it looks like a hacker has broken into your system and you need to assess the damage...

...when you've been on a bug for 2 hours and you still can't figure it out...

...you couldn't have anticipated what you needed 4 months ago.

The most useful attributes? They're "high cardinality".

Huh? High cardinality attributes are IDs.

Git commit SHAs.

Host IDs.

Invoice IDs, Customer IDs, Event IDs. Request IDs. User IDs.

You get the idea.

Anything that will allow you to uniquely search across events for events related to a specific object.

But also we need more than just the IDs.

We need the relevant data at the time the event was issued.

Sending a business event? Let's include all the attributes from that event, right? Right.

Is there an error you want to create an event for that details something's gone wrong with the invoice?

Is the status of the invoice worth including in the event? Yup, you betcha.

Throw it all in there.

3. Context aware

Unless you've spent as long as I have splunking through logs, it's difficult to overestimate the importance of context.

What was going on when this event was triggered?

If you want to understand contexts?

Think about a stack trace.

A stack trace is context.

It’s very basic context - just an array of line numbers

And that’s why a stack trace is almost never enough to go off.

In fact, stack traces nowadays look like positively ancient.

They're an observability tool from a much simpler time when codebases were much smaller and distributed apps weren't a thing.

What we really want is nested contexts.

Nested Contexts

Take an example.

First, an HTTP endpoint was hit.

Then we went out to an API and made a request.

That came back within 0.1 seconds.

Then we enqueued 4 background jobs.

The first one took 13 seconds to complete.

Whilst that one was running, the second was also triggered.

And the error happened within that second background job.

This allows us to join the dots - to see what the system was doing whilst this error occured.

Now let's see which data types give us all those qualities.

Metrics? They give us none of these things.

They just say "hey, this number was X at this time".

Logs? They give us an event. Cool. They give us wide events. Yup.

However… context? They're pretty weak.

Building in context is possible and that's what we do at BiggerPockets - we have attributes that store which request was going on when the event was triggered.

Problem is, the context nesting is very limited.

Since you only have one event, you either have to create multiple events (which duplicates logs) or you can only have one layer of nesting (since the previous layers would override each other).

Traces? Now we're talking. These bad boys have everything above.

And best of all? You can derive metrics from traces. You can derive logs from traces.

The weakness of traces? They're much bigger than logs. They're more costly to store.

It's also harder to get your head around tracing than logging.

It's messier - you need to make contexts which often means passing blocks around.

The code becomes pretty convoluted fast and it obscures the meaning.

But? Traces are the ultimate observability data type.

Common Questions

So are you saying metrics aren't useful?

No! I'm saying they should be derived from more granular sources.

So are you saying that structured logging isn't valuable?

No! I'm saying you need to use it to send wide events.

But it's still got its limitations - no context.

Are you saying that traces don't have value?

No! I'm saying that traces can be appropriated to implement the "single wide event" ethos.

Lets forget the three pillars.

Let's favour traces - the most useful of the three.

Let’s instrument our apps with Open Telemetry.

Let’s vote with our feet.

Down with the three pillars.

Context aware, wide events forever.

Agree with this?

Sign the manifesto at kill3pill.com!

1. Wide

2. Context aware

Let’s go.

1. Wide

A wide event is one that has lots of attributes.

Think 150. 200.

What goes into a wide event?

The common rule of thumb is "whatever you can think of that you have access to that's vaguely relevant to the current operation."

Many systems work off lots of small, narrow events, events with 20 or 30 dimensions.

For example, you have an event for "http request started".

Another event for "http API request started"

Another event for "http API request returned"

Another event for "http request finished".

Where does the duration live? Well... we only know the duration once the event has been delivered back.

So the duration has to live on the "returned" or "finished" events.

Likewise with the status.

But the request attributes are on the "started" events.

This means the information about a single event - the request - is smeared between several events.

This causes big problems when using these events to understand your system.

I've tried to do this - and it's always frustrating when exploring.

You keep having to do skips between duplicate events to get attributes that should have been there to begin with!

Instead, merge attributes that live together into a single wide event.

"HTTP request handled"

"HTTP API request made"

This halves the number of events.

It also makes each event richer - it's more self contained.

Don't stop there. Throw everything you can possibly think of into that event that might be useful.

Which Heroku dyno served the request?

Which host?

What was the incoming IP address?

What were all the headers?

What's the app name?

The server name?

A full list of attributes for inspiration:

https://www.honeycomb.io/blog/event-foo-what-should-i-add-to-an-event

2. Context aware

This is a doozy. And it's something I see teams not even understanding, let alone getting right.

Events are never stand alone!

They almost always happen in some kind of context - in a job, in a server, in an HTTP cycle, within an API request.

So it's critical to capture this context as much as possible.

You can now say "Show me all GET request that trigger a job of class X that makes an API call that's longer than 3 seconds"

Seriously powerful!

And it's all possible because the events are "stacked".

Here's where traces can do things that logs just cannot.

You can still have context in your logs - shared tags spread between multiple logs.

At BiggerPockets we tag all log requests that happen within jobs with the job attributes.

But what happens when two contexts try to write to the same log key? Well... they get overwritten. Not ideal.

Traces are far more powerful.

Traces consist of several spans. Each span can be nested within another.

Ever seen a flame graph?

Yup, that's a trace. And each span can be within another one.

This means you can do the query we have above no problems at all!

The big issue?

There are two - cost and performance.

Traces are slower than logs.

They are also a lot more costly and your expense can easily explode.

All it takes is a few extra spans in some critical high volume requests...

...and your bill skyrockets.

This is all true with observability 1.0 tools.

There's a new breed of tools - LightStep (ServiceNow) and Honeycomb - that support wide events.

They have simple pricing models - based on X million events.

This tends to be more cost effective than with traditional three pillar vendors.

This makes sense - after all, they're not charging for three different kinds of data types, just one.

That's not all - because they only deal with wide, context aware events, there's no silos, no different parts of the app to skip between.

Open Telemetry - the enabler

Open Telemetry is developing fast.

Open Telemetry, for those of you who aren't aware, is finally a standard for observability.

Idea is that you can standardise the tooling across your app.

Then the vendor you choose to ship traces, logs and metrics to will be an implementation detail.

It's a matter of time until we can switch between traditional Observability 1.0 tools and Observability 2.0 tools with three lines of code.

In that new world, many will stick with what they're familiar with - the traditional three pillars.

But it’ll open up options to those who are curious about wide context aware events.

Open Telemetry allows multiple exporters and processors.

So you’ll be able to try out different observability stacks with a handful of lines of code.

Don’t like them? No worries, just stop sending data and revert to using your current system.

Like them? Just switch off the old provider.

Vendor lock in will disappear like a bad dream.

Personally? I cannot wait.

Photo by Kenny Eliason on Unsplash

When I was a kid, I remember asking my Mum if I could have a Mars Bar.

It cost 25p.

"I'm sorry love" she said kindly, "but we just can't afford that right now".

We never went hungry, we always had warm coats and decent shoes. We were always tidily turned out and clean.

But money was always tight.

My parents were incredibly frugal and made ends meet. They never had any debt apart from a mortgage.

This gave me an early respect for money.

Take From The Rich and Give To The Poor

I was fascinated by the story of Robin Hood.

I was a well behaved child and followed all the rules perfectly. I did what I was told and feared authority.

The idea that someone would steal from the rich and give to the poor seemed wrong.

On the other hand there was a big part of me that loved the idea.

I distinctly remember thinking "if only we could figure out a way of doing that lawfully, without the stealing bit, I think the world would be a better place".

Cost of Living Crisis

Fast forward thirty years.

I don't need to tell you that the income gap is wider than ever.

I could quote studies and statistics to persuade you there's a cost of living crisis.

That's not necessary. We all know that this is real. People the world over are hurting.

The poor have been hurting for... well, decades.

Captialism and Technology

Big Technology companies make a lot of money.

If this is in exchange for valuable services, that's fair.

But year after year, there are companies profiting off ignorance and miseducation.

I remember a PPC friend of mine telling me he audited the Google Ads account for a new client.

"How much are you spending?"

"Oh, about 1.2 million dollars a year."

"So, what's your ROI on that?"

They didn't even undersatnd the question.

"Well... everyone should be doing Google Ads, right? So we do it."

At my friend's suggestion, the ads were switched off.

Sales stayed exactly the same.

$100,000 a month... completely wasted.

Observability

This same thing happens in the observabilibity space.

Companies are profiting off ignorance.

A friend of mine recently created a metric in an observability tool with a mere 3 dimensions.

Within a day his manager messaged him - "Would you mind switching that off please? The costs are ramping up fast."

My friend didn't understand the impact of high cardinality dimensions on metrics.

And one dimension he'd picked? IP address. Whoops.

Now, the software knew that IP address was high cardinality.

It knew that this would blow up the bill.

The software didn't try to stop him.

There was no alert when costs spiked.

The software just went ahead and started billing the company.

Thank goodness his manager was diligent. It could have been a lot worse.

Still, the company had just wasted $800 on instrumentation that had zero value.

Why should the observability tool company care? They just added a bit more to their bottom line.

I've talked to one company who are spending seven figure sums on logging ... every MONTH.

How can any company short of Microsoft possibly justify this kind of outlay?

And yet, again, why should the software vendor care? They made a killing.

This is bad business. And it's wrong.

It would be wrong if there was no cost of living crisis.

It would be wrong if there was no income inequality.

It would be wrong if there was no-one starving because they couldn't afford to eat.

But in the light of the state of our world, it seems worse than wrong - it seems outright insane.

What's the Plan?

First, I don't believe that politics can have an impact.

No amount of legislation will stop this kind of bad practice.

Money underpins most politics and so big businesses will always have their thumb on the scale.

I believe the only answer is to play capitalism at it's own game.

My plan is pretty simple.

1. Educate engineers and companies how to observe their apps cost effectively

2. Charge 10-20% of the money I save over a year

3. Put a percentage of profits aside to fund universal basic income for a small number of families every year

The business makes more money.

Engineering teams can move faster.

Engineers save time and a lot of frustration.

Customers get a more reliable web app.

Families that are struggling for money are pulled out of poverty.

Everyone wins.

Oh, apart from large greedy software companies trying to rip off customers and hoping they don't notice.

They lose. As well they should.

I’ve worked with a lot of teams who use Sidekiq in their Rails app.

Not a single one understood what the jobs were doing.

They’d disagree.

They’d say something like:

Oh, I just look at the Sidekiq Web UI. It’s great!

The Sidekiq Web UI

This ^^ is, apparently, a great monitoring tool for production systems in 2024.

Not to piss on anyone’s chips… but this is woefully inadequate.

To understand why we need to back up.

What’s the point of observability?

Here’s my definition:

To quickly answer any question of your app running in production.

The more arbitrary and rare the question, the better your observability.

Over the last 2 years, I’ve asked many questions of Rails apps I help manage.

At the start our team couldn’t answer the most basic of questions.

As I’ve improved the observability, we can ask more complex and nuanced questions.

Important Sidekiq Questions

• Which incoming request enqueued this job?

• Which user enqueued this job?

• How many times did this job get retried?

• How long did each retry take?

• How long did all retries take?

• Is exponential backoff working as I expect?

• What caused the retry? Database request? API request?

• What was the error that caused the retry?

• Which line in the code caused the error?

• Where does each retry fall in the performance percentiles?

• Did this job go to the dead queue?

• How long did it take from enqueue to execute?

• Is the queue for this job performing within it's SLA?

• Did the failure of the job happen at the end? Or the start?

These are incredibly basic questions.

I’ve been in the position where I can only answer a handful of these.

It’s a scary place to be - thousands of jobs being executed an hour without any clue what’s really happening.

The vast majority of these cannot be answered by the Sidekiq Web UI.

So we need something more powerful.

I Wrote The Stupidest TODO App In The World…

I’ve created an app to demonstrate these ideas.

It couldn’t be simpler.

Here's the code on Github

Here is a screenshot:

It downloads TODOs via an API.

It stores them in the database.

It has a super ugly UI.

When you hit “Refresh” it triggers a Sidekiq job.

The job clears out the database and repopulates it.

View the code: https://github.com/JoyfulProgramming/rails_observability_playbook

And yet…

…I Introduced A Bug

I wrote unit tests. I designed my code. I tested it locally.

And the code still had a glaring bug!

Typical.

So how can flame graphs help us debug this?

Introducing Flame Graphs

Here is a small section of a flame graph in Datadog:

Let’s examine what we get:

1. Overview

• Error happened on the within_five_minutes queue

• This attempt took 768ms

• In the 51st percentile of retries

• The attempt took 1.23% of the overall trace

• Full duration of the trace is at the top left - 1m 2s

• Happened in a web request (the little green globe beside rails_observability_playbook)

• Attempt was in a consumer process

All from that teeny tiny rectangle. Phew.

2. Error

That teeny tiny red sliver? That's where the error happened.

It happened after many database requests (the slivers of green blocks).

That means we know it’s a database error.

Turns out, the primary key we’re inserting on has already been used.

After a little sleuthing, I realised that jobs are overlapping and causing conflicts in the database.

3. Latency

This one piece of information has saved me countless hours.

Latency is the time from job being enqueued to starting.

Measured in ms, it’s 1.1 seconds in this example.

Seeing the latency in a single job? Yeah, fine.

But aggregate them over a week?

You'll get a real time view of how long jobs take in your queue.

This answers questions like:

• Which queues are overloaded?

• Which queues are performing well?

• Can you alert me on overloaded queues?

All easy peasy.

Chef's kiss. Mwah. 🤌

4. Retries

I've never seen anyone instrument job retries in production.

This boggles my mind. Again? Basic information.

Aggregating this information gives you answers to questions like:

• What are the average retries for a specific type of job?

• What are the maximum number of retries?

• How often are jobs sent to the dead queue?

• Which jobs are not going to the dead queue?

And many more.

In my example, I could see that some jobs retried a couple of times then succeeded.

The Bottom Line

I’ve had to manage incidents with and without this information.

I’ve had to fix bugs with and without this information.

So I’m speaking from the trenches when I say this…

Instrumenting Sidekiq jobs makes debugging in production a dream.

Want help?

I’m on a mission to teach Rails engineers how to observe their Rails apps in production.

Even if you’re not using Rails, many of the principles can be adapted for your technology.

Check out the resources below.

Hit me up on LinkedIn with any questions you have.

Resources

1. Rails Observability Playbook in Notion - Free

2. Example Rails App Instrumented With OpenTelemetry - Free

3. Joyful Observability for Sidekiq in Rails - $1

Many teams I've worked on work from tickets on a Jira board.

The goal is framed as "get all the tickets for this sprint done".

When that's done, they celebrate with a Slack post and everyone responds with an emoji.

The metrics reflect that they're killing it.

Many sprints later and all the tickets that comprised a project have been deployed.

They're a little behind the deadline but hey - it's software.

They high five each other, have a little zoom meeting with some self congratulatory back slapping.

Why?

Oh because "it's important to celebrate the wins".

Give me a f***king break.

Meanwhile in the real world

Software is fundamentally unpredictable.

So if there's a plan that's more than 1 week long...

...and the plan has been executed...

...and it all looks like it's gone to plan?

The reality is way messier.

The reality is way more toxic.

The reality is this.

In order to meet the arbitrary deadline, the engineers skipped some unit testing.

In the same vein, they have code that's very difficult to understand "because we took on some technical debt and that's ok"

The designers chucked pretty pictures over the wall and the engineers implemented them.

Turns out there was no time in the plan for user testing.

The API they were calling was a bit flaky and they forgot to set timeouts and implement retries.

But that's just a detail.

Testing was done at the end and a ton of issues came out of it.

After every deploy the engineers count that ticket as done.

After all it's deployed! That's the same thing as done.

There's even a column in Jira - they renamed it from "Done" to "Deployed"

So much missing work.

For all the missing work, the same lame excuses are wheeled out time and time again.

"We'll put that on the backlog and do it later"

"We can add that as a fast follow"

"We don't have time for that"

"That's not a priority"

After the project has been released and announced at the company town hall in a 16 slide presentation that sent everyone including the engineers who built the damn thing to sleep...

Turns out they're onto the next project.

The tickets stay on the backlog.

They age.

The "backlog" is over a thousand tickets strong now.

This team has been very busy releasing projects just like this one over the last year.

But so what?

Here's so what.

Meanwhile in the real, non-techie world

A man is visiting his mum in a care home.

It's a stressful period of his life.

He's trying to juggle the relationship with his partner, meet his own needs, make the most of his time away from home, cope with the emotional impact of his mum in slow decline, tidy her house, make sure she's well looked after, do research for his business...

He's travelling from Preston to Leyland by train.

He's in a rush.

He opens the Trainline app.

He adds a favourite journey from Preston to Leyland.

He looks at the next train time.

Oh crap! it's in 2 minutes! platform 1!

He bombs up the stairs over to the platform.

Navigates other passengers moving at a snails pace.

The train is waiting at the platform.

He jumps on.

"Oh man... I'm in such a rush. I'd better slow down and check this is the right train"

He leans out the carriage.

"Blackpool North... 11:49... yup that's right."

That's what the app said.

He sinks into the seat, relieved. He made it!

Now to buy the ticket on the app.

Press buy...

Put his CVC code in...

Bit of a pain - but at least he can remember it.

Press the buy button...

...this is taking ages...

...come on hurry up...

...yes! Finally.

There's an annoying modal that comes up every time.

He hates it. "You're going to <destination>”

There's no way of cancelling it or switching it off and pressing outside the modal does nothing.

And here it comes. With the usual stupid zoomy animation.

"You're going to Kirkham and Wesham!"

That's ... wait what...? He wants to go to Leyland...

He feels a creeping sense of dread.

Beep beep beep!

The doors close.

Too late. He realises what's just happened.

When he added the favourite journey he assumed that the app would then show him that journey.

It didn't.

Adding a new favourite journey, it turns out, sent him back to...

...the first favourite journey.

Which is in the exact opposite direction of his desired travel.

With a deep sigh he tries to find the next train that's actually correct.

It's another £8.20. Total waste of money.

Oh... better not miss the next stop.

It'd be great if the app would give him notifications when he's approaching his stop.

It does give notifications - notifications about train strikes in Birmingham.

And how is that useful to him?

Oh. That's right. It isn't.

His phone regularly pings him about some garbage Tweet from someone he followed 10 years ago...

...but it can't show him what he needs right now.

Thankfully he's not so busy booking the correct train that he misses this stop too.

He jumps off.

Now to pay...

...enter the CVC AGAIN...

...press buy...

...come on...

...and now the app just hangs.

A spinner overlays the purchase button.

He waits. And waits. And waits.

Nothing. No error message. No update.

How long exactly does he have to wait until the app gives him feedback?

It's anyone's guess.

The seconds tick by. He closes the app.

Sigh.

Fine. He'll go to the ticket office.

No-one there.

Fine. The ticket machine then.

He presses "Quick buy".

Nothing happens.

He presses again.

Still nothing.

Finally he figures out you need to press and hold your finger for nearly two solid seconds.

Despite us all having magical devices in our pockets that are super responsive and we can play farmville at 3am and be distracted from our shallow lives by incredible graphics and animations..

...the same technology somehow hasn't translated into touch screens at train stations that actually, like, WORK.

The "common journeys" screen that comes up next has destinations like "Liverpool Lime Street".

What?! How is that a common destination? From freaking Kirkham?

Sigh. He needs Leyland.

Presses L.

Waits for 5 seconds.

E.

5 seconds...

Y.

5 seconds.

It would be easier to forge the train ticket at this point.

Presses buy.

Presses again.

Swipes his card.

After a 20 second delay... it's done!

He's going to...

...erm his destination.

It’s been a stressful experience.

I know it was stressful, because this man was me.

What’s the link with observability?

Almost every dysfunction above is a failure of feedback loops.

Unit testing is a feedback loop to make sure the software is working.

User testing is a feedback loop to make sure the humans understand how to use the software.

Defects in testing are a feedback loop. Feedback that the design is wrong.

Observability is another feedback loop.

The difference?

Observability is a feedback loop in the only environment that matters - production.

If you have poor observability, you’re working in the dark.

How many users are having problems? No idea.

Why, exactly, are they having these issues? We’ve some theories.

How is the app being used? We think we know.

Are we leaking revenue? Probably not.

Wait… does observability replace testing?

Of course not.

Many practices like unit testing and user testing catch a ton of issues and mistakes with both the design and functionality early.

Fixing problems early on is essential to moving fast.

We can catch a ton of obvious mistakes early. And we should.

But no matter how much we try to predict how our systems will behave from our chair…

…it’s a fools errand for anything but the simplest of apps.

We can only have real, actionable insights when the app is being used by real users in real situations.

I’ve deployed pieces of work with and without watching my observability tool.

I’ve seen the difference and it’s stark.

In almost every case with observability, I learned something new, whether intentionally or by accident.

And it’s actually, kinda, (whispers) fun.

So do yourself a favour.

Find a way of adding some basic observability to your app.

Look at it often.

And you’ll start to see patterns you can act on to improve.

Want help?

I’m on a mission to teach Rails engineers how to observe their Rails apps in production.

Even if you’re not using Rails, many of the principles can be adapted for your technology.

Check out the resources below.

Resources

1. Rails Observability Playbook in Notion - Free

2. Joyful Observability for Sidekiq in Rails - $1

Huh? But aren't metrics really useful?

Nope. And here's why.

1. Metrics provide answers to questions you already know.

Question: At 3am when your site is down, it's not obvious what's going wrong and you're half asleep, will metrics help?

Maybe. If you're lucky, you've encountered an issue you already thought might happen.

But maybe, just maybe, this is a non trivial issue that you've never seen before.

I've experienced this many times.

Here's how it goes:

1. Oh crap. What's going on?

2. Check all 10 metrics. All look good.

3. Now what?

If every production issue you have falls into 5-10 different repeatable scenarios, great!

But in 2024 our apps are a *lot* more complex than this.

2. Metrics result in dead ends.

Question: If the metric *is* showing that there's a problem, can it explain *why*?

No. Of course it can't.

It's just a number.

Imagine if you went to the doctor.

"I'm fainting all the time... what's going on?"

"OK, let me take your blood pressure. Wow - that's high. Well, thanks for coming to see me. Laters!"

I don't think you'd be very happy.

Unless you can correlate the metric with other anomalous activity, you're not much further forwards towards a solution.

I've experienced this too.

Here's how it goes:

1. Oh crap. What's going on?

2. Check all 10 metrics. Oh! Background job queue latency is massive.

3. It's something to do with the background jobs.

4. Now what?

OK, sometimes you get lucky and there are other metrics that hint at the reasons.

Maybe you've seen a flood of errors in your error tracking software.

All those other times? You're on your own.

3. Metrics can get expensive.

Question: If you set up a feature in your monitoring tool and 3 weeks later you realise because of one tiny misunderstanding the metric has racked up $7,000...

...is that a good feature?

Erm.... NO.

Months ago I had this exact problem.

I'd set up a metric, added a few dimensions to it. Just a handful.

A week later my manager pinged me...

"Looks like this metric is getting expensive... can we turn it off?"

Turns out that we were being charged extra for each unique value of those "few dimensions".

No worries. There was only... 220 unique values... charged at... oh wait…

...well I'm sure you can fill in the gaps.

We switching off the metric before we'd even had a chance to use it.

Metrics have their place.

But they should be derived from other tools.

What’s the plan Mr Observability Guru?

What should you be doing instead? Try these:

1. Tracing.

Tracing can give you a proper structured map of the call stack.

That's infinitely more useful than a metric.

Make sure you add all the context you care about to each span.

I’ll be discussing this in more detail in a future post.

2. Ditch traditional tools.

If you’re already invested in an observability tool, use it.

However - there are always exceptions.

Sometimes you want to make a break with the past.

Using a next generation tool like Honeycomb can get you an awful lot.

Clear, transparent pricing based on events.

A killer UI with instant drill down into the parts of your stack you care the most about.

Fully Open Telemetry compatible.

And proper, structured, aggregateable events.

3. Logging.

Sigh. Seriously? I looked at the logs the other day and they told me the square root of f**k all!

Engineer Who Hates Logs

Yup, I know, I've experienced useless logs too.

But over the last year I've invested heavily in setup of logs.

And to be crystal clear, when I say “logs”, I mean “structured logs with 50-200 dimensions that prioritise IDs”.

Like… proper, modern logs. Not logs stuck in the 1980s.

Metrics can be derived from logs.

I’ve aggregated logs together into metrics. Totes doable.

This means you can go to the source of the data!

You can look at why that weird slowdown of the site happens on a Monday morning.

Not just that it happens.

Proper logging is paying off dividends for my current client.

Engineers can, in 30 seconds, diagnose a whole bunch of problems from first principles.

Click… type… click… “Ah! So that’s where the errors are coming from…”

However, it takes a lot of setup.

And that’s where I can help.

Free Rails Observability Playbook

Check out my free playbook for observability in Rails.

Sign up for Notion and clone into your account.

Then play around with it and have fun.

It’s still a work in progress, so feedback is very welcome!

1. What is helpful about this playbook?

2. What would you like to see that’s missing?

3. Is the survey a good idea?

Have a great week and I’ll see you* soon!

* Not literally, I’m typing this in Belfast, Northern Ireland. Internet and all that.

Download the playbook

I'm putting together a playbook on how to achieve this in Rails.

Want to take an early sneak peek?

Check out the playbook in Notion

Imagine a company that writes a CRM.

Here’s how developing a new feature might look:

1. Customers ask for a feature “Please can I have email notifications when a contact messages me?”

2. Business gives it a green light. Because they like the idea. “Email notifications will be great! I love it when I get notifications!”. “But will it move the needle?” “Who cares! Email notifications baby!”

3. Product owner makes a list of tasks for how to approach the work over many months. It looks a lot like a gantt chart, but it’s not because it’s in Google Sheets you see. The product owner has no engineering experience, no domain knowledge and has no real understanding of what’s possible. They will plan everything out because building software is just like building a house - you make the plans then build it.

4. User experience designers create some pretty pictures. They look good. There’s not much thought put into making it interactive. That’s a problem for the engineers. No wireframes. No information architecture. There’s little to no user testing. After all, users make things so messy and designers like staying in Figma - it’s such a cool app. When they’re done designing it, they’re really happy. The designs are copy pasted into cards that will be handed off to engineers.

5. Engineers break down the list of tasks and guesstimate points. After the first week of tasks is planned, they’re guessing and they know it. The points are nonsense and we’re all going to realise that soon enough.

6. Engineers take the tickets and work on coding up the solution. The designers love seeing engineers taking their ace Figma design and making it work. The engineers leave 14 comments on their carefully built designs pointing out problems. The designers push back and eventually the engineers just do what they ask, even though it makes the code more difficult to manage.

7. When it comes to the functionality, engineers go fast and break things, because that’s what Facebook do. They hammer in the features as fast as their fingers will let them. They love building! They’ve heard something about “clean code” but they don’t have time for that. Plus they can clean it up later.

8. They work with PRs because the need code reviews and it’s a best practice. Everyone reviews PRs in their own style at their own pace. Some are thorough, others have a shortcut made for “LGTM” because that takes too much time to type. Messages fly around Slack asking for approvals.

9. All the engineers work independently because that’s how you make teams go much faster. Plus, the engineers are all introverts and who wants to sit in another meeting?

10. No-one in the business looks at what the engineers are making. After all, the engineers don’t make the effort to talk to them. Ever. The business leaves them to do their super smart coding thing and they’ll review what they’ve done later. Email notifications will be easy - every app has them.

11. Every engineer has a different coding style. Some people write tests that are barely readable. Others swear by mocking. Others write the odd test that tests nothing. Still others write no tests at all. Missing tests aren’t too much of a problem. We don’t have time to write them, but they’ll do it later.

12. When tests fail, engineers spend hours fixing them using the debugger because it’s difficult to understand what’s going on. Engineers who don’t like writing tests think this is ridiculous. What’s the point in being slowed down? All these tests do is add extra baggage but someone on the Internet told these engineers to write them and so they do. Ah well, group think!

13. The code uses a lot of inheritance because that’s what’s great about object oriented programming. Class names are based on the design pattern the engineer has just learned that weekend so that other engineers can understand the code better.

14. Three weeks in and it’s clear someone didn’t do their planning correctly because cards that seemed easy and were given 1 point seem to be taking forever. Every card seems to breed three more. The product owner is increasingly stressed that the team is falling behind.

15. The existing code tricky is to change and they’re behind, but that’s OK - the engineers have a plan - copy and paste existing code then tweak it. Subclassing is a great idea to reuse existing code too.

16. The engineers have read on the internet that accumulating tech debt is sometimes necessary and the timescale seems impossible, so now is the perfect time to take some short cuts. They’ll can make it correct later. Even the engineers who are keen on writing tests aren’t doing so now.

17. When the feature is ready after 2 months, someone in the business who isn’t as smart as all those engineers takes a look. First impressions? It seems a bit weird. The email notifications seem to have overlapping rectangles in Gmail. The font is strange and the colours are garish. The settings screen looked good on paper… but when you use it, the buttons are glitchy. It wasn’t clear on the design, but the settings hide other parts of the page that looks weird and unintuitive.

18. The business feeds this back. The product owner is nervous. They tells the business that they’re really behind so they won’t be able to fix most of this. They promise to fix the overlapping rectangles at least as that’s the most important issue.

19. When the engineers see bug tickets in the backlog they’re not happy. It feels like they’re going backwards. The first issue to tackle is called - “Fix overlapping rectangles” - it has three paragraphs of waffly text in the description. When the engineer tries to reproduce the issue they can’t and they tell the product owner as much. The card is moved to blocked waiting on investigation. After 3 days of trying to fix it in their local dev environment the engineers give up.

20. The product owner goes back to the business with their tail between their legs. “Sorry, we worked on the overlapping rectangles and couldn’t fix it. We’ll fix it later. Both the PO and The Business knows that this won’t happen. But they both nod their heads and agree to not rock the boat.

21. The feature enters QA. After a few days, the testers have found 78 bugs. The testers are delighted. Everyone else is pissed.

22. The product owner spends 2 days putting all 78 bugs into Jira, complete with their position on an effort / impact matrix and story pointing them all. Jira already has 1425 tickets in the backlog, so those extra 70 floating around in there have lots of friends to keep them company.

23. Time is getting tight and the business are leaning on them heavily now - this feature is turning out to be very expensive - so they fix the most important 8 bugs and leave the rest - they’ll fix those later. After all, every bit of software has bugs.

24. None of the 8 bugs have a regression test written for them because they’re getting short on time. Besides, the engineers are pretty sure those have been fixed now. When the product owner points to a bug they’ve come across, the engineer defends their work - “it works on my machine”. The product owner gives up. Bigger fish to fry.

25. Finally the feature is released. The team merges the last PR, waits 28 minutes for the tests to pass. The moment the commit is released the team high fives. They did it! They celebrate their win by posting onto Slack and everyone chimes in adding emojis. There are 21 emojis! Wow - this feature release is a huge success. Everyone needs a 2 week holiday. But it’s done.

26. The team wants to do the cleanup - there are so many outstanding issues. But sadly, that time that was allocated doesn’t appear. They have 2 days to recover then it’s time for another urgent feature.

27. Whilst they’re developing this next feature, the team start getting support requests mentioning the email notifications. Seems like some customers are not getting notifications. More concerningly, some customers are being repeatedly spammed every hour. The app really shouldn’t be doing that.

28. The engineers look into it. The error shows in Sentry - it’s buried under 534 other errors so they didn’t notice it. One engineer tries to figure out what’s going on. The error contains almost no useful information. Just a cryptic message - “Email wasn’t sent” - with a stack trace. The stack trace points back to a try / catch block that swallows the error and logs a generic message.

29. The engineers can’t replicate this locally. They’re in trouble and they know it. There are now 132 support requests with exactly the same error. There are over 700 errors in Sentry now. The pressure mounts.

30. They redeploy the code, taking the try catch out. The error is hardly more useful. “SSL disconnect - retrying…”. They open New Relic as a last ditch attempt - no-one really looks in there much - and they go to the logs. Nothing of any use - just a bunch of text statements.

31. 48 hours after the first support request, they’re still no nearer to debugging the problem. They’ve added four log statements that point to some kind of strange timeout with the email platform. The CEO is frantic and hounds the product owner for updates. The engineers are stressed out of their minds. The testers feel bad.

32. 4 days after the initial storm, they figured out the issue. They decide to patch it and move on. They’re already late with the next feature after all and stakeholders aren’t happy. They decide to fix the other Sentry errors later.

None of this is joyful.

Nice rant. What’s the point?

In the story above there are a lot of antipatterns. I’ve never seen all these at any one workplace - this is deliberately exaggerated. But hopefully you’ve seen enough of these problems to be able to identify.

I’ve tried to challenge some of these anti-patterns throughout my career and I’ve not been massively effective.

I’ve tried to educate on the merits of software design. It’s my lifelong passion, but it seems to be an uphill battle to persuade folk that this is something worth investing in. I also find software design to be a very difficult discipline to measure the impact of.

Observability is my new focus

However, I’ve seen that improving observability is something I can have a measurable impact with. I’ve learned how to take a Rails app that’s a black box and turn it into an app that you can at least understand some of the basics.

I’ve seen the value of this on other engineers and on the business.

Everything I’ve learned I’ve learned on my own. As far as I’m aware there’s next to zero information on the web for how to do this in Rails properly. And there’s almost zero information for other frameworks too.

The articles I’ve found about structured logging are utterly useless. They just about explain what it is, give one example in Java then treat the problem as “fixed”. Hilarious.

Other articles on observability are also useless - they talk about the three pillars (if I read another generic explanation of logs traces and metrics, I will go actually mad) and “why observing in production is really important” but give zero real life examples. No useful advice. No details.

How do I understand my background jobs?

How do I understand the errors in my Sentry?

How do I track user behaviour?

What conventions should I use in my structured logs?

“Oh, just install New Relic / Datadog / Prometheus.”

OK… and then what?

It’s like asking “How should I invest my money?” and being told “Well, there’s stocks, bonds and Crypto. They’re volatile in different ways. Make sure you save enough for retirement. Best of luck!”

There’s now an excellent book - Observability Engineering: Achieving Production Excellence - by Charity Majors, Liz Fong Jones and George Miranda. This seems like a significant ray of light at the end of this very dark, very depressing tunnel of black box footling around, blindfolded by lack of accurate data… or any data at all.

I want to be crystal clear - I don’t see myself as any kind of expert. I’m an expert in observability in the same way as a 12 year old is an adult. Compared to a toddler, maybe. Compared to a real adult? Not a chance.

The Rails apps I work with are, after 2 years of part time observability improvements, are at 40% of where I want to be.

I’m obsessed with this topic and I’m always pushing the limit of what I know to deliver more value in observing an app in production.

I’m on a journey to learn this too. Come along for the ride!

I’ll share all my learnings here

The learnings I’ll share with you in this newsletter are a tiny fraction of what I think the standard for observability should be for every app we have with any significant workload in production

I’ve talked to some engineers who seem to be quite happy with what they call “the basics” in place, which really means “we installed the standard tools, added lograge and I suppose it’s good enough for now” which really translates to “we don’t have a clue how to debug anything in production but we just create a ticket to add more logging and the bug gets punted off. Plus, we’re onto the next feature now”.

I’m not sure if it’s that engineers don’t care that users are having a terrible time, don’t have bandwidth to take on more work or just don’t even have headspace to think about it.

In any case, the chaos caused by this complete lack of interest in how the app is actually working in production is incalculable. Apps crash, users lose data, the app randomly freezes on a Tuesday, DDOSes, hackers, scams, overage bills, huge P95s - and on and on.

At one point I thought I wanted to build features. Now I know different. I want to help you understand the features you already have.

Please reach out if you have any questions or comments.

Thanks for reading. Onwards and upwards!

PS - A HUGE thank you to Irina Stanescu who writes The Caring Techie for inspiring me to start writing again. You rock.

SOLID was invented by Uncle Bob Martin. It’s a backronym.

The author started with the letters and reverse engineered words to fit them.

There are many critiques out there.

Kevlin Henney’s The SOLID Design Principles Deconstructed is a good place to start.

In this newsletter we’ll examine three alternatives.

1. Four Rules of Simple Design - Kent Beck

2. SHOC - Jason Gorman

3. CUPID - Dan North

1. Four Rules of Simple Design

It's shocking how few engineers have heard of this.

And yet it's one of the best ways I know of evolving a design.

Simple code adheres to all four rules.

Rule 1. The code works.
Rule 2. The code expresses intent
Rule 3. The code has no duplication (including ideas)
Rule 4. The code has no dead parts

We go through the rules sequentially.

We don't go onto the next rule until the previous rule has been satisfied.

Rule 1. The code works

Note it doesn't say "passes all the tests".

Tests can be misleading, depending on how they're written.

Make sure the code works and there are no edge cases that you're aware of.

Rule 2. The code expresses intent.

Put another way - good names.

Read the code literally line by line.

Does every line make sense?

Does it read like English?

Adjust the names until it does.

Rule 3. The code has no duplication.

There are no pieces of code that are duplicated.

This also applies to duplication of ideas.

If there's a similar thing expressed elsewhere but the code is different...

...that's still duplication.

Eliminate it.

Rule 4. The code has no dead parts

Debatably, this is the easiest rule to verify.

There's no unused code.

So in summary:

Rule 1. The code works.
Rule 2. The code expresses intent
Rule 3. The code has no duplication (including ideas)
Rule 4. The code has no dead parts

I've found following these four rules pivotal in building software that's well designed.

2. SHOC

Invented by Jason Gorman

See this SHOC video where he outlines the principles.

* Swappable
* Hides internal workings
* One job
* Client driven interfaces

I find this a much better alternative:

* Easier to understand
* Applies to modules, not just classes
* Plain English

3. CUPID

Invented by Dan North.

He introduces CUPID in this blog post.

• Composable: plays well with others

• Unix philosophy: does one thing well

• Predictable: does what you expect

• Idiomatic: feels natural

• Domain-based: the solution domain models the problem domain in language and structure

CUPID is about finding joy in coding again.

Instead of making the codebase “habitable”, Dan says we can and should shoot much higher.

Let’s make finding defects and adding features joyful!

He also says code that fits in your head is an important metric.

There’s been a lot of pushback from the community.

Personally I think Dan’s really onto something.

I’ve found myself building non idiomatic castles in the sky to try to make code that’s easy to change.

Others find this kind of code difficult to work with and understand.

How can we make software that’s easy to change and joyful to work with?

Only by breaking things apart and creating the right kind of abstractions.

Summary

So there you have it. Three alternatives to SOLID.

Which do I use?

I think all three have their merits.

I’ll give the last word to Denis Cahuk based off his excellent comment:

I don't think these acronyms are a good substitute for education if the acronym (and the underlying meanings) don't simply express principles and patterns.

The S in solid stands for SRP. Who can explain the S in SRP so that we understand a 5th of SOLID.

The L in SOLID stands for LSP. The L stands for Liskov, as in Barbara Liskov. She is still alive. Did anyone watch any of her talks on LSP?

The D in SOLID stands for DIP. That's DI for dependency inversion (principle). That doesn't mean you're using an autowire container, the emphasis is on inverting responsibilities and call-stack hierarchies.

All this information is public and easy to google. Yet after 10-15 years in the industry I still have to look them up because they simply don't make sense and never did.

SOLID was invented to sell a book about Clean Code and Architecture. It was very successful in that. However, it isn't a very successful no-brainer for developers who didn't have the opportunity to work in a great team and have real, tangible problems beyond the ability to adopt SOLID or Clean Architecture.

If you’ve not subscribed to Denis’ excellent Crafting Tech Teams, give it a shot. It’s chock full of interesting information!

What Next?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Buy Software Design Simplified - an introduction for curious, driven engineers

Have a great week!

Bit of a change of pace this week.

I'll be taking a look at a revolutionary new language.

Why? What's this got to do with software design?

I think Unison will change the game of software design.

Trust me - I'm a new language skeptic.

Any time I hear of a new language I roll my eyes skywards. "Not another one..."

However, this new language, coupled with AI, will allow us to build applications we can't dream of today.

Let's get into it.

The Big Idea

The Big Idea

Unison is a typed functional language.

Each Unison definition is identified by a hash of a syntax tree.

Put another way, Unison code is content-addressed.

Example:

increment : Nat -> Nat

increment n = n + 1

The syntax tree that Unison sees is:

increment = (arg1 -> #a8s6df921a8 arg1 1)

• The "+" function has a hash and Unison only sees that

• Arguments are referenced by position

Most importantly...

...names are *just metadata*.

Wrap your head around that for a moment.

• Every function name is metadata

• Every type name is metadata

• Every argument name is metadata

The function name "increment" is just a label that's mapped to a hash.

This one, simple idea unlocks some surprising and powerful benefits.

Here are a few:

• First class refactoring

• No builds

• Easy distributed computing

• Very fast tests

Sounds too good to be true?

Let’s see.

First Class Refactoring

Names are *just metadata*.

Unison knows that #arg1 is "n".

Unison knows that #a8s6df921a8 is actually Nat.+

This has huge implications for refactoring.

Imagine renaming a function...

...and the name is instantly updated everywhere.

Yup. Instantly. No work needed.

You can even create aliases for names in Unison.

Example:

There's an Optional type:

structural type Optional a = Some a | None

It's very similar to Maybe in other languages.

Other languages have Option.

Which is it? You can never remember.

In the Unison Codebase Manager, we do:

.> alias.term Optional Maybe

.> alias.term Optional Option

Now Unison doesn't care which you use.

They're all referring to the exact same type.

No more head-scratching!

This means a good name (one of the hardest things in software)...

...becomes a decision you can keep revisiting.

It gets better...

Let's say you want to do a more complex refactoring.

Maybe you add an argument.

Unison knows the hash of the old function.

It knows the new one doesn't have any callers yet.

So Unison keeps your old code working until your new code compiles!

In fact, the UCM tool steps you through the refactoring.

Yup. It gives you a list of TODOs.

All the time your codebase still works with the old code.

Once you've completed all the TODOs, Unison will use your new function.

Implications:

• Your code is always in a running state

• No more time wasted trawling through long obscure error messages

• Predictable workflow - work through the plan Unison gives you

Imagine the evolutionary design work you can do with this tool.

Refactoring in Unison becomes a breeze and a dream.

It can't teach you refactoring, but it certainly makes it a *lot* easier.

No Builds

Wait... what? No builds?

How is that possible?

First of all, what is a build?

A build wraps up all dependencies and freezes them at a point in time.

It tends to be a slow, expensive, brittle process.

But in Unison, every function is just referenced by a hash.

And every change to a function is actually a new function, referenced by a new hash.

So every function knows it's dependencies.

Let's say you want to deploy your code.

Your machine tells a remote node "Hey, run this function called a8s6df921a8"

The node says "No idea what that hash is. Can you send it me please?"

Your machine says "Sure! Here it is: ..."

Unison will serialize the structure of the function and send it to the remote node.

Then the remote node runs it.

Of course, if it doesn't know other hashes, it'll ask your machine for them too.

So you're transferring functions over the network, not data.

That means it's quick. And it means no builds.

But wait... there's more.

Every time you change a function, Unison actually creates a new function.

So functions are immutable.

And old ones are still referenced by their hashes.

So that means dependency conflicts just go away.

Still don't get it? From the excellent Unison docs:

Dependency conflicts are, fundamentally, due to different definitions "competing" for the same names.

But why do we care if two different definitions use the same name? We shouldn't.

The limitation only arises because definitions are referenced by name.

In Unison, definitions are referenced by hash (and the names are just separately stored metadata), so dependency conflicts and the diamond dependency problem are just not a thing.

https://www.unison-lang.org/learn/the-big-idea/#no-dependency-conflicts

Practical implications

• Deploys are virtually instant (< 0.1 seconds)

• No builds

• No dependency conflicts

• No semantic version pinning (no versions!)

• No CI tools

• No package managers (goodbye Rubygems, NPM, yarn etc)

• No Terraform

• No CloudFormation

• No bash scripts

• No YAML configs

• No Docker (ah, sweet relief)

Just write your code, run it on the Cloud and live your life.

Wow.

Easy Distributed Computing

Here's where things get really mind-blowing.

Remember that every function is identified by a content hash.

Also Unison can serialize any function and send it from one node to another.

Every dependency of one function can be sent to any other node.

If a node doesn't have any of the dependencies it needs it asks the caller...

...hey can you send me that function?

And it will.

Then the function will be cached on that node for next time.

Example:

distributed : Seq k Nat ->{Remote} Nat

distributed dseq =

use Nat + ==

dseq

|> Seq map (x -> x + 1)

|> Seq filter (x -> Nat mod x 7 == 0)

|> Seq reduce 0 (+)

Don't worry about understanding the code.

Just know that this is a distributed map reduce.

It can run on 1 node or 10000.

And it's 7 lines of Unison.

That any programmer can write.

Then it's run like so:

a4.distributed.Remote.pure.run

'(distributed (fromChunkedList 10 (List.range 0 100)))

⧨

Right 735

This runs it locally.

But there will be other runners that can distribute it in parallel across nodes in the cloud.

Want to do big data processing?

• No Hadoop

• No Spark

• No Jar files

• No Docker

• No complex frameworks

Just Unison and a few lines of business logic.

Check out the blog article Spark-like datasets in Unison in Under 100 Lines for more details.

Very Fast Tests

Remember we're in a functional language.

So functions tend to not have any side effects.

You pass a function some arguments.

It returns an output.

No database access, no memory mutation.

Unison handles side effects using something called Abilities, but that's for another day.

Point is, if you have:

• No side effects

• Functions as hashes

• Values in, values out

It has huge implications for test suites.

It means you can cache the results of any test.

Imagine a test suite that contains 100,000 tests.

You write three more unit tests and run the test suite.

The existing tests don't run.

So the test suite takes milliseconds to run.

And when it's run, the three tests you've added are cached.

What does this mean?

Huge test suites can run in fractions of a second.

No more "push tests into CI".

Just run them on your machine. It's easy. It's cheap.

This will make TDD a dream to perform.

I've spent hundreds of hours waiting for tests to pass and code to deploy.

Those days are gone with Unison.

Wrap up

There are so many more cool things about Unison that I've not touched on.

But hopefully I've whetted your appetite to learn more.

A few honourable mentions:

• Abilities - a single way of doing side effects without any Monads

• Codebase storage - all code is stored in a database. No more text files on disk.

• No Github - Unison are creating their own code sharing and PR tools that take advantage of the language

• Unison Cloud - Find out more here: https://buff.ly/3BXioSL

• Typed, durable storage - Imagine if rerunning computations relied on transparent caches of old data!

• Instant deploys - we touched on this, but it bears repeating - sub second deploys directly from your machine.

This language blows my mind.

If I had more time, I'd be investing all of it learning it seriously.

It’s not all sunshine and rainbows though.

Drawbacks

1. Small ecosystem - very few libraries, very little community - although what's there is super friendly and welcoming

2. Unconventional - if you've coded in Elm or Haskell a lot, it looks familiar. Otherwise, it's challenging to learn.

3. Slow - it'll take a long time to develop any kind of working software, but there are tools in the pipeline to help this. Rails it is not!

4. Closed - the language is open source but the cloud platform will not be as that's how they'll make their money.

Phew. So that was a whirlwind tour of what I think is the most promising language I've seen in the last 10 years.

A huge thanks to the whole Unison ecosystem and community.

You're doing some amazing things and I look forward to Unison cloud being in beta for everyone very soon!

What Next?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

This week we’ll be talking about mindsets around being an engineer:

Marketing yourself

You are not a <language> programmer

Focussing on the basics

As always, I welcome your feedback. Just reply to this newsletter. Let’s go.

Marketing Yourself

"Surely I want to appeal to as many companies as I can... right?"

Wrong. Let's see why...

Let's start with the most obvious:

Specialist - deep knowledge in one topic - refactoring, design, architecture, performance

Generalist - shallow knowledge of many topics

Specialist - T-shaped skills

Generalist - _ shaped skills

Specialist - fixes deep complex problems

Generalist - fixes generic surface-level problems

Specialist - well paid

Generalist - poorly paid

Specialist - marketing that appeals to a niche

Generalist - marketing that appeals to no-one

Specialist - difficult to replace

Generalist - easy to replace

Specialist - harder to find (unless marketing is really good!)

Generalist - easy to find

Specialist - unique and interesting

Generalist - generic and boring

Specialist - obsesses over a few topics

Generalist - mildly interested in hundreds of topics

Specialist - less adaptable

Generalist - more adaptable

When selling yourself to employers, specialising is a powerful sales tactic.

If they need what you're selling, you move into a category of one.

You’re Not A Language Programmer

Stop thinking of yourself as a <language> programmer.

Question.

Why are you hired?

Why are you paid money?

There's not a competent business person who wants "a JavaScript developer".

Ok they say they want that... but what they really want is the business value delivered to the customer.

And more than that they want their business to grow.

So no, you're not a "Ruby programmer" or a "python wizard".

You're an engineer.

You solve problems.

And the language is just a tool to get there.

The best engineers use the right tool for the job.

And the best engineers don't see themselves narrowly as tied to a language.

Being an engineer is so much more than that.

And yes, this is the game the job market plays.

But you don’t need to buy into that once you’re in the job.

Focussing On The Basics

What is software engineering all about?

Is it building features?

Is it satisfying requirements?

Is it creating business value?

Jim Rohn said "there are two ways to make more money. One is to find more time. One is to provide more value."

There's an almost infinite amount of value you can give.

This week I've been challenged by various events to provide more value.

Maybe you can relate - as an engineer it's easy to get wrapped up in the tech.

In the possibilities.

In all the options.

And yet... the business is paying us real money to solve their problems.

Not problems we THINK they have...

...but the problems they actually have.

I got lost in my own head.

Overworking. Stressing about lots of small perfectionistic details.

Focus on getting the basics right.

Focus on the problems the business has.

Not the problems you think they have.

Want Some Help?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

Welcome back to Practical Software Design.

Here’s what we’re covering:

Preventing the Big Ball Of Mud

Use Cases

Interfaces > Behaviour

We all want flexible apps. Let’s see a few ways of how we can make this happen.

Preventing The Big Ball Of Mud

Ever worked on an app that's devolved into a big ball of mud?

Yup. Me too.

How can you start improving it?

Bounded contexts can be a great place to start.

Eric Evans has said that if he had to rewrite the DDD book he'd put bounded contexts at the start, not the end.

What are they?

• They split your app up into broad areas

• Within one bounded context the language is consistent

• Across bounded contexts the same word can mean something different

• Example: a customer mean something slightly different between the sales bounded context and operations

• Bounded contexts can talk to each other but it should be fairly minimal chat

What are the advantages?

• Reduces bloat in God classes

• Segregates the app and imposes some order

• Allows different parts of the app to evolve alongside a department

Bounded contexts are difficult to do, especially in many frameworks (I'm looking at you Rails) but they're possible.

And they make reasoning about your code way easier.

Use Cases

Start using use cases.

What's a use case?

Why bother?

Great questions.

A use case is an object that wraps an interaction of some kind.

Example - creating an invoice.

You'd have a class called "InvoiceCustomer"

This class is instantiated from a controller.

It would have a single method to kick off the use case.

That method accepts various parameters the class needs to do its job.

It might be the customer ID. The invoice type maybe. The amount.

Then it talks to many different classes to do its job.

It might talk to the Stripe or FreeAgent API.

It might talk to the database.

Then it delivers back a result - success or failure.

The failure contains the errors to render in the view.

The success returns the invoice.

What's the difference between this and a service object?

Well, use cases generally wrap a whole transaction whereas service objects can do... but don't need to.

Advantages

• Encourages ubiquitous language

• Testable

• Describes the actions the app takes

• Can wrap complex transactions

Disadvantages

• More code

• A later of abstraction

I love use cases - I find them excellent abstractions that sit between controllers and the domain.

Focus on the interface

Want to build flexible object oriented apps?

Ignore implementation.

Focus on the interface.

Huh?

Most engineers obsess about implementation details.

• Should I use a Map? Or a Set?

• Let's use this cool framework shortcut to save 2 lines of code

• Let's add a guard clause

I'm not saying this stuff is irrelevant...

...but it's much less important than the interface to your class.

What's the secret to flexible OO code?

Well... a big one is swappability.

Imagine you want to add a new feature.

What's the usual process?

• Find the code to hook into (2 hours)

• Figure out the tangled web of dependencies (2 hours)

• Think about how you're going to change that code (1 hour)

• Change 4 methods and 2 classes and add 2 arguments (3 hours)

• Fix the 23 tests that break (4 hours)

You've changed so much though...

...what happens if an area with low test coverage breaks in production?

Here's the alternative:

* Find the code to hook into (2 hours)

* Understand 3 interfaces (2 hours)

* Write 2 classes matching the interface with different behaviour using TDD (2 hours)

* Plug into existing code (20 minutes)

Lean back and relax knowing you've hardly changed any code.

Want Some Help?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

Another week, another edition of Practical Software Design.

Here’s what we’re covering:

• The power of bounded contexts

• Making a mess

• Active Record is an Anti-Pattern

It’s going to be a controversial one this week. Let’s get into it!

The Power Of Bounded Contexts

Small Rails apps are easy to manage.

But as the codebase grows, I've run into some big problems:

• God objects - large classes that hide huge numbers of methods

• Duplication of ideas - similar but different concepts across the codebase

• Conflicts - teams stepping on each other's toes

An idea from DDD can help here - bounded contexts.

A bounded context is a self-contained area of the codebase that has it's language.

Example: An app to run a garage might have bounded contexts for sales, marketing, servicing, and accounting.

Each of these contexts will have a concept of a "customer", but it's a different model depending on the context.

Ideally, data should be separate too, but that's tricky in Rails, so in the past, I've shared one database table for all customers.

However, you can have a Sales -> Customer, a Marketing -> Customer etc as Rails models under namespaces.

Once this has been done, you can put sales behaviour in one customer model and marketing behaviour in another.

Bounded contexts are fantastic for breaking down a large monolith.

I'd love to hear other people's experiences. Reply to this email!

Make a Mess

Stop refactoring.

Instead... make a mess.

When I first heard this from my mentor I did a triple take.

"But... surely we want clean code ASAP?"

No. And here's why.

There's a temptation amongst developers like myself who love refactoring to use it whenever they get a chance.

The first chunk of code that's more than three lines... and BLAM!

We extract into a method.

The first bad name we see... and BLAM!

That sucker's been renamed throughout the whole class.

But there's a big problem with this approach.

You can be too DRY.

When you refactor too soon, you've not let enough of a pattern emerge.

As Sandi Metz says "Prefer duplication over the wrong abstraction".

Wait. Wait for that mess to build up.

Wait until it's almost intolerable.

At that point, a pattern will emerge and that pattern will tell you the right abstraction.

Then, BLAM you've refactored, not just for the sake of it, but because the code is telling you to do that very thing.

Controversial? Maybe. Let me know what you think…

Active Record is an Anti-Pattern

Active Record is damaging your codebase.

Repositories are one answer.

Why do I say this?

I've been in a lot of Rails codebases in the past few years.

Many folk think that going against Rails is futile.

"The Rails Way Or Nothing" they cry.

"Let's Try A Different Way" say I.

Rails is brilliant for getting a project off the ground.

Once it grows to a certain size, the Rails Way hinders you.

Rails uses a pattern called Active Record.

It encourages stuffing business logic and persistence into the same class, along with many other unrelated concerns.

One alternative is the repository pattern.

I've introduced repositories in Rails codebases recently.

It's still early days but here are the benefits I'm getting:

* Swappability - we can swap out the database for in memory repos in tests, leading to better design and flexibility (as well as some speed)

* Isolation - we can access APIs via repositories, leading to our code being less coupled to external vendors

* Less mocking - because we're now not coupled to APIs such as Stripe, we don't need complicated mocks in tests

What about you? Have you used repositories before?

Found them helpful? Annoying?

Either way please let me know in the comments or just reply...

Want Some Help?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

Welcome to another issue of Practical Software Design.

This week we'll be talking about:

• Testing Readability

• On Being Human

• Learning things that don't change

Testing Readability

A radical (and maybe silly) idea.

Imagine if you could test the readability of your code.

I declare "that's readable" all the time.

What evidence do I have? 🤷‍♂️

Well... not much.

Apart from that fact that I was in a meeting where me, myself and I all agreed…

…that code I just wrote is awesome.

Yes, smug mode engaged.

My work here is done.

Then 6 months later I return to that "readable" code... and I'm cursing myself.

"What the heck does that do?"

"Yeah, nice job past me."

Turns out the past me was an idiot.

This has happened to me repeatedly...

...and yet I still don't have a good antidote for it apart from "more code reviews".

Which is nice and all... but hardly scientific.

There's a whole discipline devoted to user testing user interfaces.

The UX folk know how to structure the interview, how to ask questions, and how to understand the user's mental model.

Why not an equally disciplined approach to testing the subjective readability of our code?

Imagine if a PR could be run through readability tests, where 10 random developers from all over the world could see your code, and were asked questions like:

• What does this do?

• How does it work?

• What parts do you not understand?

I hear what you're saying "We do have that, John, it's called Open Source!".

Sure, but this would be for private repos.

This app could compile all those responses together and maybe even give you a "readability score" - a bit like Code Climate.

It would give you a sense of which parts were most confusing.

Then you could make changes, resubmit and see if the code was easier to read or harder.

Please let me know your thoughts by replying to this email!

On Being Human

Human > Technology

Let's be human.

Why should you care?

Obsessing about coding, engineering and technical skills is a beautiful thing.

Regularly I find myself up late at night coding.

Especially when I'm obsessed with a topic like I am at the moment. *

But after a while of this, I start to feel weird.

Disconnected.

Empty.

Did you know that 15% of men say they have zero close friends? **

Last night I went out to a meetup, connected with humans in person, and it felt great.

As engineers, we focus too much on the tech and not enough on the human.

Some ways to become more human:

• Write for humans first and computers second

• Take time to get to know your colleagues

• Ditch the jargon and enterprise speak

• Go to meetups (yes, real-life in-person meetups

• Connect with others on LinkedIn and via face-to-face conversations

• Sit in on some user tests of your product. You'll learn a lot. 🤯

As AI becomes more and more A Thing, we'll value human connection more and more.

* The topic is nullable architecture, but that's for another post

** According to American Perspectives Survey, conducted by the Survey Center on American Life

Learning things that don't change

Don't learn another framework.

Don't learn another language.

⭐ Learn things that don't change. ⭐

Jeff Bezos' advice to the founders of BaseCamp 15 years ago?

It was the same.

"Invest in things that don't change".

What doesn't change in software?

Designing software that's easy to change.

"How ironic!" Shut up Alanis.

The majority of talks, courses and books, are on new tools, languages and frameworks.

That's nice... but where are the books on design?

On refactoring?

On making code that's easy to change?

Sure, there are the classics.

But this subject doesn't get anything like the attention it deserves.

The syntax of a language is relatively easy to learn.

The idioms of a language are more difficult and have more value.

They teach you new ways of thinking.

Software design skills last for life.

No matter the framework or language.

What Next?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

This week we’ll be talking about TDD.

TDD is an important practice in making software easy to change.

More TDD topics to come, but this week we’ll be talking about:

• Keeping Builds Green

• The Fourth (missing) Step Of TDD

• TDD Doesn’t Need Unit Tests

Keep Builds Green

"The tests are failing but I'll fix it later..."

TDD is all about fast feedback cycles.

But it's also about reducing waste. Why? 👇

When a test is failing, you're getting feedback that what you've done was a mistake.

When that mistake persists... for seconds... then for minutes... then maybe for hours...

It means you're getting off track.

The longer the failures last, the more work in progress builds.

The more work in progress builds, the more waste you're accruing.

And as waste builds it becomes exponentially harder to clean.

Finally, you throw your hands up "Let's break out the debugger".

That's an admission of failure!

90% of your time should be in green.

And when you're in red, get back to green as soon as you can - by whatever means necessary.

Keep your build green at all times!

The fourth step of TDD

TDD has three steps:

1. Red - write a failing test

2. Green - make the test pass

3. Refactor - restructure the code

But there's a fourth step that you may not be aware of.

And it's critical to long-term agility.

You write a failing test.

Now it's time to make that test pass. Right?

Well... maybe.

Pay attention to how the test fails.

What's the failure message?

You want that failure message to describe what went wrong clearly.

Why?

If a test suite never fails, then it's not protecting you from any failures.

A useful test suite will fail sometimes.

You'll be blocked until you fix them.

With expressive, helpful, failure messages, you can easily reason about what went wrong.

With weird, obtuse, "expected 4 got 2" failure messages you'll need to break out the debugger.

Breezing past an unhelpful error message right now means shooting someone else in the foot later.

And that someone might be the future you.

Do yourself a favour and work on making that error message useful before you move on.

The fourth step of TDD is part of step 1 - make the failure message useful.

TDD doesn't need unit tests. 😮

Say what now?

It all starts with TDD being misnamed.

The acronym stands for "Test Driven Development"

So how can tests not be an integral part of this practice?

Dan North came up with the term "BDD" or Behaviour Driven Development.

Back at the start, BDD meant something very different:

It suddenly occurred to me that people’s misunderstandings about TDD almost always came back to the word 'test'.

I started using the word “behaviour” in place of “test” in my dealings with TDD and found that not only did it seem to fit but also that a whole category of coaching questions magically dissolved.

I found the shift from thinking in tests to thinking in behaviour so profound that I started to refer to TDD as BDD, or behaviour-driven development.

- Dan North

TDD is first and foremost about behaviour.

Any way you have of verifying that behaviour is enough to practice TDD.

An early mentor showed me how to test drive code by only using a browser and refreshing the page.

So the next time you do TDD, whether or not you're doing it with automated tests, focus on behaviour, not the test.

Need More?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!

End to end testing of web software is slow and painful.

The most popular tool is Selenium. But there are much better options.

Selenium was created by Jason Huggins in 2004, so it'll be 20 years old next year.

It's served us well as a developer community, but I think it's time to talk about end to end testing tools for the next 20 years.

There are many candidates out there:

• Cypress

• Nightwatch.js

• WebdriverIO

• Puppeteer

Despite working in Ruby, I was heavily leaning towards Cypress.

That's until my colleague Karl Entwistle pointed me to Playwright.

Why I think it might win:

• More stable than Cypress

• Faster too

• Supported by Microsoft

I'm going to experiment with it over the coming months.

I'll report back when I know more!

DDD In Rails

Small Rails apps are easy to manage.

But as the codebase grows, I've run into some big problems:

• God objects - large classes that hide huge numbers of methods

• Duplication of ideas - similar but different concepts across the codebase

• Conflicts - teams stepping on each other's toes

An idea from DDD can help here - bounded contexts.

A bounded context is a self-contained area of the codebase that has it's language.

Example: An app to run a garage might have bounded contexts for sales, marketing, servicing, and accounting.

Each of these contexts will have a concept of a "customer", but it's a different model depending on the context.

Ideally, data should be separate too, but that's tricky in Rails, so in the past, I've shared one database table for all customers.

However, you can have a Sales -> Customer, a Marketing -> Customer etc as Rails models under namespaces.

Once this has been done, you can put sales behaviour in one customer model and marketing behaviour in another.

And slowly your app will become more segregated. Order will be imposed on the chaos!

The power of mentorship

When I started out in Ruby I sucked.

Now I can evolve a design on my own.

Evolutionary design. As a junior engineer, I could never understand it.

But then I was mentored and my life changed...

As a junior programmer, I really struggled to write code that exhibited any properties of good design.

I knew how far short I was falling.

Time and time again, I wrote code and my mentor showed me how to improve it.

For months I just wasn't getting it.

My code was a spaghetti mess.

My code was fragile, complex, and intertwined.

And most worryingly, I didn't have a clue how to evolve towards sanity.

A good mentor sticks with the junior and encourages them.

• 6 months later, I was starting to understand

• 1 year later, I could evolve a very basic design on my own

• 2 years later I was brimming with confidence

• 10 years later, evolutionary design is still one of my obsessions

I want to pass along my skills and teach the next generation of developers these timeless principles.

And I want to speed up the process for you!

That's why I'm writing this newsletter.

Need More?

I have two options:

1. Book a 1 hour coaching session (normally $200, FREE for a limited time)

2. Join the waitlist for my book

Have a great week!