About

Kubernetes nodes were kernel-panicking under what seemed like a normal workload. After a long investigation, it turned out two VOLUME lines in a Dockerfile — leftovers that hadn’t been relevant in years — were causing containerd to copy gigabytes of data on every container start.

The Setup

We had a service that shipped ML model data baked into its Docker image — about 500MB of serialized model files at /mnt/models. The image also declared a VOLUME at that path, a leftover from the old Docker-era --volumes-from data container pattern:

FROM python:3.11-slim

# ... build steps ...

ADD models/ /mnt/models
VOLUME ["/mnt/models"]

In Kubernetes, the actual volume mounting is handled by the kubelet — pod specs define emptyDir, hostPath, PVC, etc. The VOLUME instruction in the Dockerfile doesn’t do anything useful.. or so we thought.

What VOLUME Actually Does in containerd

When containerd creates a container from an image that has a VOLUME declaration, it copies everything at that path from the image layers into a per-container directory on the node’s disk:

/var/lib/containerd/io.containerd.grpc.v1.cri/containers/<container-id>/volumes/

This happens before the container starts, whether or not Kubernetes mounts anything at that path. Containerd is faithfully implementing the Docker spec — if no external volume is mounted, the container should still see the image’s data at that path. The copy ensures that.

But in Kubernetes, nothing ever reads from this copy. The kubelet manages volumes through pod specs, not Dockerfile declarations. So containerd writes hundreds of megabytes per container to a directory that just sits there on disk doing nothing.

How This Becomes a Node Killer

On NVMe-backed instances with local storage, this is invisible. NVMe throughput is so high that even copying 500MB per container takes fractions of a second.

But when our scheduler started placing these pods on EBS-backed nodes.. things went sideways.

The math:

• Model data per container: ~500MB
• Pods bin-packed on one node: 50
• Total unnecessary writes: 50 x 500MB = ~25GB
• gp3 EBS baseline throughput: 125 MB/s
• Time to write 25GB at 125 MB/s: ~200 seconds

That’s over 3 minutes of sustained disk I/O before a single container actually starts running. During that time, all other container operations on the node are blocked behind this I/O — new containers can’t start, health checks time out, and the kernel’s writeback subsystem gets backed up.

In our case, the I/O pressure triggered kernel panics via hung tasks in the overlay filesystem’s sync path. Nodes rebooted, containerd’s overlay snapshots got corrupted, and pods entered CrashLoopBackOff with exec format error because their binaries were now zero-byte files.

What We Thought the Problem Was

Kubernetes doesn’t have a native “data volume” concept — there’s no way to say “share these files from this image with these containers” without copying. So the pattern we used was a data container: bake the model files into an image, run an init container that copies them from the image to an emptyDir, and mount that emptyDir into the runtime container.

initContainers:
  - name: data-loader
    image: my-models:latest
    command: ["cp", "-r", "/mnt/models/.", "/dest/models/"]
    volumeMounts:
      - name: model-data
        mountPath: /dest/models
containers:
  - name: app
    volumeMounts:
      - name: model-data
        mountPath: /mnt/models

When pods started taking forever on EBS nodes, we assumed the bottleneck was this cp step — 50 init containers all reading from the same image layers on a single EBS volume at once. Shared reads from a throughput-constrained disk.. that seemed right.

But when we dug into the actual disk I/O, the writes were the problem, not the reads. And the writes were happening before any init container ran.

What Was Actually Happening

We checked the node filesystem and found the smoking gun:

$ du -sh /var/lib/containerd/io.containerd.grpc.v1.cri/containers/*/volumes/
533M    /var/lib/containerd/.../containers/abc123/volumes/
533M    /var/lib/containerd/.../containers/def456/volumes/
533M    /var/lib/containerd/.../containers/ghi789/volumes/
# ... repeated for every single container

Every container had a full 500MB copy of the model data sitting in containerd’s volumes directory — completely separate from the emptyDir copy that the init container was making. Containerd was honoring the VOLUME declaration in the Dockerfile, duplicating the data before any of our code even ran.

The init container cp was actually fast — it reads from already-cached image layers in memory. The slow part was containerd writing 500MB x 50 containers = 25GB to EBS in the background.. completely invisible unless you go look at the node’s disk directly.

The Fix

Two lines deleted from the Dockerfile:

FROM python:3.11-slim

# ... build steps ...

ADD models/ /mnt/models
# VOLUME ["/mnt/models"]   <<<<< removed

That’s it. No other changes — the ADD instruction still puts the data in the image layers, the Kubernetes volume mounts still work exactly the same, and rsync-based init containers that copy data at runtime are unaffected.

Should You Ever Use VOLUME in a Dockerfile?

Probably not. The VOLUME instruction made sense in the Docker Compose era for --volumes-from patterns. In Kubernetes, volumes are declared in pod specs, not Dockerfiles. So the Dockerfile VOLUME instruction just.. causes containerd to copy data on every container start for no reason. It can’t be overridden without rebuilding the image, and it doesn’t show up in kubectl describe or any Kubernetes-level tooling — so you won’t even know it’s happening.

If you’re inheriting base images, check if they declare volumes you don’t need:

$ docker inspect <image> --format '{{.Config.Volumes" }}'
map[/mnt/models:{}]

Final Thoughts

The fix was two lines.. finding it was the hard part. The VOLUME instruction is a Docker-era artifact that containerd still faithfully implements, and in Kubernetes it just works against you. If you’re running containers on Kubernetes — especially on EBS or any throughput-constrained storage — audit your Dockerfiles for VOLUME declarations. You might be burning gigabytes of I/O per node and not even know it.

About

I’ve been using Claude Code pretty heavily for both work and personal stuff, and I ran into something that kept bugging me.. there’s no way to keep your sessions separate. Auth tokens, settings, MCP configs, plugins — it all lives in one ~/.claude directory. So I built claude-profile to deal with it.

The Problem

Claude Code stores everything in $HOME/.claude. One directory, shared across every project, every account, every context. If you’re logged into your work account.. your personal projects use those same credentials and configs. There’s no concept of profiles or workspaces.

In practice this means:

• You can only be logged into one account at a time
• MCP servers you set up for work show up in personal projects (and vice versa)
• You can’t run work and personal Claude instances side-by-side

I tried juggling symlinks for a bit and that got old fast. I wanted something that just worked.

How claude-profile Works

It turns out Claude Code supports a CLAUDE_CONFIG_DIR environment variable that redirects where it looks for config. Under the hood it hashes that path with SHA-256 and generates a unique macOS Keychain entry for credential storage.. so each config dir gets fully isolated auth. That’s the key insight — you don’t need to hack anything, you just need to manage that environment variable.

claude-profile is a small Go binary that does exactly that. Each profile gets its own directory at ~/.claude-profiles/<name>/config/, and the wrapper sets CLAUDE_CONFIG_DIR before launching Claude.

# Create a work profile
$ claude-profile create work

# Create a personal profile
$ claude-profile create personal

# Launch Claude with your work profile
$ claude-profile -P work

# Or set it via environment variable
$ CLAUDE_PROFILE=personal claude-profile

Everything passes through transparently — all your normal Claude Code arguments and flags work exactly the same.

Setting Up a Profile

The create command walks you through an interactive wizard:

$ claude-profile create work
Profile name: work
Config directory: /Users/matt/.claude-profiles/work/config
Pick a statusline color (for visual identification): blue
Profile 'work' created successfully!

Each profile gets its own statusline color so you can tell at a glance which account you’re in. I found this surprisingly useful — it’s one of those things where you don’t realize how much you needed it until you have it.

Installation

Prebuilt binaries are available for Linux, macOS, and Windows (both amd64 and arm64):

# Quick install (auto-detects OS/arch, verifies checksums)
curl -fsSL https://raw.githubusercontent.com/diranged/claude-profile/main/install.sh | bash

# Or via Go
go install github.com/diranged/claude-profile@latest

Final Thoughts

Honestly this feels like something that should be a built-in feature. But until it is, claude-profile handles it with zero modifications to Claude Code itself — just a thin wrapper around an environment variable that was already there.

If you’re juggling multiple accounts, check it out at github.com/diranged/claude-profile.

About

We had a problem that I think most teams running Kubernetes on AWS at scale eventually hit: we were paying for Reserved Instances and Savings Plans, but we had no way to tell if we were actually using them. Worse, our node provisioner was actively launching spot instances while pre-paid capacity sat idle. I built two open-source tools to fix this — Lumina for cost visibility and Veneer for cost-aware provisioning.

The Visibility Problem

AWS Savings Plans and Reserved Instances are great for reducing compute costs — you commit to a certain spend level and get a discount in return. The catch is that the discount applies at the organization level, across all accounts, and the allocation logic is non-trivial.

When you want to know “what does this Kubernetes node actually cost me right now?”, the answer depends on:

• Whether the instance type matches a Reserved Instance (exact type, exact AZ, same account)
• Whether an EC2 Instance Savings Plan covers its instance family in that region
• Whether a Compute Savings Plan has remaining capacity to cover it
• How much of that capacity is already consumed by instances in other accounts
• Whether the instance is spot (which gets spot pricing regardless of RI/SP commitments)

None of the existing tools gave me this. Kubecost and OpenCost use published on-demand rates or simple spot pricing — they don’t understand your organization’s Savings Plans allocation. The AWS Cost and Usage Reports (CUR) are comprehensive but delayed by hours, massive in size, and not something you can query in real time. The Cost Explorer API gives you aggregates, not per-instance breakdowns.

We also had multiple capacity managers in play — Karpenter, spot.io, and others — each launching instances across multiple accounts. There was no unified view of what anything actually cost after discounts.

Lumina: Seeing What You’re Actually Paying

Lumina is a Kubernetes controller I built that solves the visibility problem. It queries AWS across your entire organization — every account, every region — and builds a real-time picture of what each EC2 instance costs after all discounts are applied.

The core of it is a Savings Plans allocation algorithm that mirrors how AWS applies discounts, in strict priority order:

1. Spot instances get spot market pricing (no RI/SP applied)
2. Reserved Instances match first — exact instance type, exact AZ, same account
3. EC2 Instance Savings Plans apply next — specific instance family in a specific region
4. Compute Savings Plans apply last — any instance family, any region
5. On-Demand is whatever’s left

Lumina runs this allocation every few minutes and exposes Prometheus metrics for each instance. The two key metrics are ShelfPrice (what the instance would cost at on-demand rates) and EffectiveCost (what it actually costs after RI/SP coverage). The difference between those two numbers is money you’re saving — or if EffectiveCost equals ShelfPrice, money you’re leaving on the table.

The model is rate-based — it gives you $/hour snapshots of the current state rather than trying to replicate AWS’s cumulative billing. It’s an estimate, not a replacement for your AWS bill, but it’s accurate enough to drive real-time decisions. And that’s the point.

The Provisioning Problem

With Lumina running, I could see the problem clearly for the first time. I’d look at the metrics and find situations like this:

• The m8i family had unused Savings Plans capacity at an effective rate of ~$0.13/hour for an m8i.xlarge
• Karpenter was launching m8i.xlarge spot instances at ~$0.09/hour — except during high-demand periods when spot prices spike above $0.15/hour
• During those spikes, we were paying more for spot than the Savings Plan rate we’d already committed to

Karpenter has become the de facto node provisioner for Kubernetes on AWS, but it has a well-known gap: it doesn’t understand Savings Plans or Reserved Instances. It uses published on-demand pricing and spot market rates to make decisions, with no awareness of your organization’s pre-paid commitments. This has been an open request for years, with related issues around RI/SP-aware provisioning and first-class Savings Plans support — but as of today, Karpenter still treats every on-demand instance as full price.

From Karpenter’s perspective, an on-demand m8i.xlarge costs $0.2117/hour and spot is ~$0.09/hour, so spot wins. Every time. It doesn’t know that your Savings Plan brings the effective on-demand rate down to $0.13/hour — which during a spot price spike is actually the cheaper option.

The result is that your Savings Plans sit partially utilized while you pay spot rates on top of them. You’re paying twice — once for the commitment you’re not using, and again for the spot instances you didn’t need.

Veneer: Closing the Loop

Veneer is the second piece — a Kubernetes controller I built that reads Lumina’s cost metrics from Prometheus and translates them into provisioning decisions.

It works through Karpenter’s NodeOverlay mechanism — an alpha feature that lets you adjust Karpenter’s simulated pricing for instance types. When Veneer sees that an instance family has available RI/SP capacity that makes on-demand cheaper than spot, it creates a NodeOverlay that reduces that instance family’s simulated price in Karpenter’s scheduling. Karpenter then naturally prefers it.

The key design constraints were:

Don’t over-subscribe. If there’s Savings Plans capacity for 10 more m8i.xlarge instances and Veneer has already steered Karpenter to launch 10, it removes the overlay. The next instance goes back to spot pricing, which is exactly what you want.

Don’t break Karpenter. Veneer prefers RI/SP-covered instances — it doesn’t require them. If spot capacity is available and RI/SP capacity is exhausted, Karpenter falls back to spot naturally. No NodePool changes, no hard constraints.

React in real time. As nodes launch and terminate, RI/SP utilization changes. Veneer watches these changes through Prometheus and adjusts NodeOverlays accordingly. A Savings Plan that was fully utilized 5 minutes ago might have capacity now because a large instance was terminated.

How They Work Together

The full pipeline looks like this:

AWS Organization (RIs, Savings Plans, Spot Prices, EC2 Instances)
    │
    ▼
Lumina (calculates per-instance effective costs)
    │
    ▼
Prometheus (stores cost metrics, utilization data)
    │
    ▼
Veneer (reads metrics, creates/updates/deletes NodeOverlays)
    │
    ▼
Karpenter (provisions nodes using adjusted pricing)

Lumina is the data layer — it answers “what does each instance actually cost?” Veneer is the action layer — it uses that data to steer provisioning toward the cheapest option, which might be on-demand when you have RI/SP coverage, or spot when you don’t.

You can run Lumina without Veneer if all you need is cost visibility — the Prometheus metrics are useful on their own for dashboards, alerting, and chargeback. Veneer is the optional next step that closes the feedback loop.

Final Thoughts

The underlying issue is a disconnect between how AWS bills you and how Kubernetes provisions compute. AWS applies discounts at the organization level using a complex priority system. Kubernetes provisioners use published pricing and have no idea your organization has pre-paid commitments. The result is predictable: you pay for capacity you don’t use.

Lumina and Veneer bridge that gap. I built Lumina to make the invisible visible — real costs, in real time, as Prometheus metrics. I built Veneer to act on that visibility and make sure pre-paid capacity gets used before reaching for spot.

Both projects are open source and available on GitHub:

• Lumina — real-time Kubernetes cost visibility
• Veneer — cost-aware Karpenter provisioning

About

I ran into a problem that I suspect most teams running Kubernetes at scale have seen but few have a clean answer for: when a new node joins the cluster, workloads start scheduling immediately — before the node is actually ready to serve them. DaemonSets haven’t come up yet, critical services are still initializing, and your application pods land on a half-baked node. This has bitten us enough times that we finally built a solution: Vigil.

The Problem

Kubernetes has a concept of node readiness — the kubelet reports Ready once it can run pods. But “can run pods” and “should run pods” are very different things.

In any production cluster, your nodes run DaemonSets: monitoring agents, CNI plugins, CSI drivers, log shippers, security agents. These are the infrastructure layer that your application pods depend on. The problem is that Kubernetes doesn’t wait for any of them before scheduling workloads.

Here’s what the failure looks like in practice. A new node comes up and your application pod gets scheduled onto it:

$ kubectl get events --field-selector reason=Unhealthy -n app-team
LAST SEEN   TYPE      REASON      OBJECT                MESSAGE
12s         Warning   Unhealthy   pod/api-server-7f8b2   Readiness probe failed: dial tcp 10.0.47.12:8125: connect: connection refused
12s         Warning   Unhealthy   pod/api-server-7f8b2   Readiness probe failed: dial tcp 10.0.47.12:8125: connect: connection refused
18s         Warning   Unhealthy   pod/api-server-7f8b2   Readiness probe failed: dial tcp 10.0.47.12:8125: connect: connection refused

The app starts up and immediately tries to push metrics to the local Datadog agent — which hasn’t started yet. Depending on how the app handles that failure, you get anything from lost metrics to outright crashes.

Or worse — a critical node-level service has a bug in its latest version and never reaches Ready on new nodes. Your existing nodes are fine because they’re running the old version, but every newly launched application pod landing on fresh nodes fails immediately:

$ kubectl get pods -n kube-system -l app=critical-service --field-selector spec.nodeName=ip-10-0-47-12
NAME                        READY   STATUS             RESTARTS   AGE
critical-service-x9k2f      0/1     CrashLoopBackOff   4          3m12s

$ kubectl get pods -n app-team --field-selector spec.nodeName=ip-10-0-47-12
NAME                        READY   STATUS    RESTARTS   AGE
api-server-7f8b2            0/1     Running   0          2m45s
worker-processor-m3k9d      0/1     Running   0          2m38s

Your applications are running on the node, but they’re broken because the infrastructure they depend on never came up. The scheduler has no idea — it sees available CPU and memory and keeps packing pods onto a node that can’t actually serve them.

There’s also the resource accounting problem. The scheduler doesn’t factor in DaemonSet resource consumption when placing workloads. A node with 4 CPU cores looks like it has 4 cores available, even though DaemonSets will eventually claim 1.5 of them. Your application pod and the DaemonSets both get scheduled onto the new node at roughly the same time — but the higher-priority DaemonSets start first and claim their resources. Your application pod gets squeezed out before it ever runs:

$ kubectl get events --field-selector involvedObject.name=api-server-7f8b2 -n app-team
LAST SEEN   TYPE      REASON      OBJECT                 MESSAGE
5s          Warning   OutOfcpu    pod/api-server-7f8b2   Node ip-10-0-47-12 is out of cpu resources

$ kubectl get pods -n app-team --field-selector spec.nodeName=ip-10-0-47-12
NAME                        READY   STATUS      RESTARTS   AGE
api-server-7f8b2            0/1     OutOfcpu    0          2m45s
worker-processor-m3k9d      0/1     OutOfcpu    0          2m38s

The scheduler thought the node had plenty of room, but by the time the application pod tries to start, the DaemonSets have already claimed the resources it was counting on.

What We Tried First

Before building anything, we spent a lot of time trying to make the problem smaller.

Optimizing DaemonSet startup time. If DaemonSets come up fast enough, the race condition window shrinks. We set up per-region ECR image caching so pulls were nearly instant. We tuned kubelet settings aggressively — adjusting API rate limits, registration timing, and pod startup parallelism. We dug into Kubernetes API server throttling configs to make sure the control plane wasn’t the bottleneck.

This helped. It shrank the window from “a couple of minutes” to “tens of seconds.” But tens of seconds is still enough for a fast-starting application pod to land on a node and fail. And it did nothing for the scenario where a DaemonSet has a bug and never comes up.

Searching for existing solutions. We looked hard. We searched for open source projects, Kubernetes-native features, KEPs in progress — anything that addressed “don’t schedule workloads until DaemonSets are ready.” We couldn’t find anything that solved it cleanly.

Kubernetes has PriorityClasses (DaemonSets already get high priority), PodDisruptionBudgets (wrong problem), and various admission webhooks people have cobbled together (fragile and high-maintenance). None of them actually answer the question: is this node ready for workloads?

The Idea

The building block was already there: startup taints.

Kubernetes supports taints and tolerations — you can taint a node so that only pods with a matching toleration will schedule on it. Node provisioners like Karpenter, Cluster Autoscaler, and others support applying taints to nodes at launch time. The idea is straightforward:

1. Apply a startup taint to every new node (e.g., node.example.com/initializing:NoSchedule)
2. Something watches the node, waits for DaemonSets to be ready
3. Remove the taint once the node is actually ready for workloads

Steps 1 and 3 are easy. Step 2 is where it gets interesting.

You could maintain a manual list of “these DaemonSets must be running before the node is ready” — but that’s fragile. Teams add and remove DaemonSets constantly. Someone forgets to update the list and you’re back to square one, or worse, nodes are stuck forever because the list references a DaemonSet that no longer exists.

We wanted something that could figure it out automatically.

Vigil

Vigil is a Kubernetes controller that solves this. It watches for nodes with the startup taint, auto-discovers which DaemonSets should be running on each node, waits for them all to reach Ready, and then removes the taint.

The key insight is the auto-discovery. Vigil uses the same scheduling predicates that the Kubernetes scheduler uses — node selectors, affinities, tolerations — to determine which DaemonSets belong on a given node. No manual allowlist. If you add a new DaemonSet with a node selector that matches, Vigil automatically includes it in its readiness checks.

Here’s what the flow looks like:

New node launches
  → Node has taint: node.example.com/initializing:NoSchedule
  → Vigil detects the tainted node
  → Vigil discovers 8 DaemonSets should run on this node
  → Vigil watches: 5/8 Ready... 7/8 Ready... 8/8 Ready
  → Vigil removes the taint
  → Workload scheduling begins

DaemonSet pods themselves tolerate the startup taint (as they tolerate most taints by default), so they schedule and start normally. Regular workload pods don’t have the toleration, so they wait in Pending until Vigil clears the node.

There’s a safety valve too — a configurable timeout (default 120 seconds). If DaemonSets don’t come up in time, Vigil removes the taint anyway so the node isn’t stuck forever. This is a tradeoff: you’d rather have the node available with a degraded DaemonSet than have it completely unusable. The timeout gives your alerting time to fire while keeping the cluster functional.

One Critical Requirement

Vigil itself has to be running somewhere that doesn’t depend on the taints it manages. Think about it: if Vigil runs on nodes with the startup taint, and Vigil is what removes that taint, you have a deadlock.

There are two clean options:

1. A dedicated node pool without startup taints. A small set of nodes (even just one or two) reserved for cluster infrastructure like Vigil, where the taint isn’t applied.
2. Fargate (or equivalent serverless compute). If you’re on EKS, running Vigil on Fargate profiles means it doesn’t need a node at all — it runs in its own isolated compute environment.

Either way, Vigil must be able to come up independently of the nodes it’s managing.

Final Thoughts

The underlying problem here is that Kubernetes treats node readiness as a binary — the kubelet says “I’m ready” and the scheduler starts packing pods. In reality, readiness is a spectrum. A node isn’t truly ready until its infrastructure layer is running.

Vigil gives you a clean way to express that. It’s a small controller with a single purpose: make sure new nodes are actually ready before they receive workloads. No manual lists to maintain, no fragile webhooks, no crossing your fingers that DaemonSets will win the race.

If this is a problem you’ve hit, check out Vigil on GitHub. It’s open source and ready to use.

About

I ran into a confusing issue recently where I had installed Java via Homebrew, my $PATH was correctly configured, but the java command kept pointing to the wrong version. The solution turned out to be a shell feature I’d never heard of: command path caching. This post explains what happened and how hash -r saved the day.

The Problem

After installing OpenJDK 17 via Homebrew and running brew link openjdk@17, I expected everything to work. My $PATH was configured correctly with /opt/homebrew/bin before /usr/bin:

$ echo $PATH
/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

But when I checked which java command was being used, it was still pointing to the macOS stub:

$ which java
/usr/bin/java

$ java -version
The operation couldn't be completed. Unable to locate a Java Runtime...

This made no sense! The correct java binary existed at /opt/homebrew/bin/java, and that path was first in my $PATH. Why wasn’t the shell finding it?

The Culprit: Shell Command Caching

It turns out that shells like Bash and Zsh don’t search your entire $PATH every single time you run a command. Instead, they maintain an in-memory hash table that caches the full path to each command the first time you run it.

Here’s how it works:

1. First time you run java: The shell searches through each directory in $PATH (from left to right), finds /usr/bin/java, and caches that location.
2. Every subsequent time: The shell uses the cached location - no PATH search needed.

This is a performance optimization… searching the filesystem is expensive, so caching the results makes sense. But it causes problems when:

• You install a new version of a command
• You change your $PATH order
• You run brew link to create new symlinks
• You install global npm/pip packages

In my case, the shell had cached java → /usr/bin/java before I installed Homebrew’s Java. Even though /opt/homebrew/bin/java now existed and should have taken precedence, the shell kept using the stale cached location.

The Solution: hash -r

The hash -r command tells your shell to clear all cached command locations:

$ hash -r

$ which java
/opt/homebrew/bin/java

$ java -version
openjdk version "17.0.16" 2025-07-15
OpenJDK Runtime Environment Homebrew (build 17.0.16+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.16+0, mixed mode, sharing)

Perfect! After clearing the cache, the shell re-searched $PATH and found the correct Java installation.

Inspecting the Hash Table

You can see what’s currently cached in your shell:

$ hash
hits    command
   5    /bin/ls
   2    /usr/bin/git
   1    /opt/homebrew/bin/npm

The “hits” column shows how many times you’ve used each command in this session.

To see a specific command’s cached path:

$ hash -v java
hash: java=/opt/homebrew/bin/java

To clear just one command (instead of the entire cache):

$ hash -d java

Shell Differences: Bash vs Zsh

Bash:

• Uses the hash command
• Usually auto-clears the cache when $PATH changes
• hash -r clears all cached commands

Zsh:

• Uses both hash and rehash (they’re synonyms)
• More aggressive caching - doesn’t always auto-clear on PATH changes
• rehash is the “zsh way” to say hash -r

If you’re using Zsh (which is the default on modern macOS), you might encounter this issue more often than Bash users.

Other Ways to Bypass the Cache

If you don’t want to clear the entire cache, you can force a fresh PATH lookup:

# Use the full path directly
/opt/homebrew/bin/java -version

# Use 'command -v' to bypass the cache
command -v java

You can also check if the cache is causing your issue:

# These should match:
which java
command -v java

# If they're different → your cache is stale!

When to Use hash -r

Run hash -r (or rehash in Zsh) after:

• Installing software via Homebrew: brew install something && hash -r
• Changing your $PATH: export PATH="/new/path:$PATH" && hash -r
• Installing global npm packages: npm install -g some-tool && hash -r
• Updating dotfiles that modify $PATH: source ~/.zshrc && hash -r

Or just open a new terminal - fresh shells start with an empty cache!

Final Thoughts

The shell’s command caching is a clever performance optimization that works great… until it doesn’t. When you install new software or change your environment, stale cache entries can cause confusing behavior where which shows the wrong path despite your $PATH being correct.

Now you know: when commands aren’t being found where you expect them, try hash -r before diving deeper into troubleshooting. It’s saved me hours of debugging time, and hopefully it’ll save you some too!