Every fantasy football ranking answers the same question: who is good?

That is not the question you have on the clock. On the clock the question is who will still be here in twenty-two picks? — and those are different questions with different answers. The tight end you want will last two more rounds. The running back ranked one slot below him will not. A list sorted by projected points cannot tell you which is which, because the list does not know who picks next or what they need.

So I built something that does. It watches a live Sleeper draft, models what each opponent is likely to take before my turn comes around again, and ranks my options by what it costs me to wait.

There is a terminal version too, which is the one I actually trust — one process, no browser, no JavaScript between me and the numbers.

What I did not expect — and the reason this post exists — is how many of the numbers that felt authoritative turned out to be measuring something other than what I thought. The pattern repeated often enough to be the real subject here: each one was caught not by thinking harder about the formula, but by checking it against something outside the formula.

The grounding: someone already did the theory

I did not want to invent a valuation model. Fantasy football analytics has a serious, methodical, open-source treatment, and it is Isaac T. Petersen’s Fantasy Football Analytics: Statistics for Fantasy Football — a free online textbook covering projection, uncertainty, replacement level, tiering and draft strategy with actual statistics rather than podcast intuition. The book is the reason this project has opinions I can defend instead of coefficients I made up.

Here is the honest chapter-by-chapter accounting of what I took:

Two implementation choices worth naming, because they are places where a textbook idea meets a draft clock.

Replacement level is a bench cohort, not a rank. In a 12-team league with two starting running backs, the naive replacement level is RB24. But RB24 is one player having one projected season, and pinning a league-wide baseline to a single row makes every VORP in the position wobble when that row moves. So replacement is the median of the twelve players immediately past the starter cutoff — the same idea, less sensitive to any one projection.

Tiers use Fisher-Jenks natural breaks, not k-means. This is a small decision that matters enormously in practice. k-means has a random initialisation. Run it twice on the same board and you can get two different tierings. That is fine in a notebook and unacceptable when the board refreshes every two seconds while you read it with forty seconds on the clock. Fisher-Jenks is an exact dynamic program over 1-D data: same input, same tiers, every time. At ~215 wide receivers and eight tiers it runs in well under a millisecond.

Read those panels and the whole draft strategy falls out. Quarterback has one huge step at the very top and then a long gentle slope — which is why the one elite quarterback is worth reaching for and the next ten are close enough that reaching among them buys almost nothing. Kicker is a straight line with no cliff anywhere, which is exactly Petersen’s §7.4.1 result and exactly why you take one in the last round.

The core idea: VONA, not VORP

VORP is a static measure. It says this player is worth 59 points more than a typical bench body at his position. True, and not decision-relevant, because if I pass on him I do not get a bench body — I get whoever is best at that position when I pick again.

That is the number I actually want:

VONA (Value Over Next Available) = his points − the expected best player at his position at my next turn

If he scores 195 and the model expects the best available receiver at my next pick to be worth 176, waiting costs me 19 points. That 19 is the entire decision. Positive means take him now; negative means the position will keep.

To compute the second half you need a survival probability for every player — the chance he lasts through every intervening pick.

The normalisation that was missing

My original spec — which I wrote before I knew better — said:

P(team i drafts X at pick j) ∝ Urgency × ADPWeight

and never resolved the ∝. That proportional sign is doing catastrophic work. Unresolved, the “probabilities” are on an arbitrary scale, and every VONA computed from them is a number with no units.

The fix is to notice the constraint that was there all along: exactly one player is taken at pick j. So the scores normalise over the available pool:

score(X, j)     = Urgency(team_at_j, pos(X)) · ADPWeight(X, j)
P(X taken at j) = score(X, j) / Σ_{X' available} score(X', j)
P_survives(X)   = Π_j (1 − P(X taken at j))

A pleasant corollary: the 1/√(2πσ²) in front of the Gaussian ADP kernel is constant across players and cancels in the normalisation, so it can be dropped.

The subtraction that is easy to get wrong

Expected-best-available must exclude the candidate himself. If I draft Chris Olave, Chris Olave is precisely the player who will not be available at my next pick. Leaving him in the expectation makes every top player look like “the same guy you’d get later,” which quietly collapses VONA to zero exactly where it matters most.

Doing that naively is O(n²) — rebuild the expectation once per candidate. It is actually linear, with a forward and a backward pass:

F[i] = s[i]·p[i] + (1−s[i])·F[i+1]     F[n] = replacement level
G[i] = Π_{l<i} (1−s[l])                 every better player is gone
H[i] = Σ_{l<i} s[l]·p[l]·G[l]           value if one of them survives

E[best available, excluding i] = H[i] + G[i]·F[i+1]

That F[n] = replacement tail is the second thing the spec got wrong: it summed over the ranked pool and left probability mass unaccounted for in the case where nobody survives. Somebody is on your roster in that case, and he is worth replacement level, not zero.

Plot survival against VONA and the decision becomes a picture:

Upper-left is “take him now”: expensive to wait, unlikely to last. Lower-right is just as useful — those are the players you can safely pass on this turn, and knowing that is what lets you spend the pick somewhere else. The vertical stripe at 100% is everyone the model is confident nobody wants yet.

Now the part where I was wrong

1. The textbook’s uncertainty metric measured the wrong thing

Petersen defines uncertainty as the spread of a player’s projections across sources (Ch. 6, Eq. 6.1). Sensible: if four projection systems disagree wildly about a player, that is real uncertainty about him.

I had one source. So I substituted week-to-week scoring variance from last season — same coefficient of variation, s/x̄, different input. It felt like a reasonable stand-in. I shipped it as a “risk profile” column: steady / neutral / volatile.

Then I tested it, and it is not a stand-in at all:

The “volatile” third of the league has a lower 90th-percentile week than the “steady” third. Correlation between CV and points per week is −0.58, and between CV and ceiling −0.40. The label was pointing at the opposite of what a risk label is supposed to mean.

The reason is embarrassingly simple once you see it: CV has the mean in the denominator. A player averaging 3 points a week with a 3-point standard deviation has CV 1.0 and looks wildly volatile. He is not volatile; he is bad. CV finds low-volume players whose small scores bounce around zero, not the boom-or-bust starters the draft advice is about.

The substitution was invalid — cross-source spread and within-season variance are not the same quantity, and only one of them is about the projection. So I kept CV as a label for what it actually measures, and added two metrics that answer the questions I had been asking it:

• role — depth-chart position plus projected volume, ranked within position.
• ceiling — the player’s 90th-percentile week last season. The honest late-round upside measure.

On the 312 players with ten or more games last season, correlation with projected points: CV −0.50, role +0.80, ceiling +0.80. The sign on the one I inherited from the formula is the wrong one.

The failure is not Petersen’s. His metric measures what he says it measures. It is a lesson about substituting a proxy into a formula and inheriting its authority without inheriting its meaning.

2. The survival model was confidently wrong in the middle

--replay re-runs a finished draft pick by pick and scores every survival prediction against what actually happened. Pooled across four completed drafts — 3,360 predictions:

Look at the left panel first. The 80–100% bucket is nearly perfect: 99% predicted, 96% actual. That bucket is also 88% of all predictions, which is why the headline Brier score looks respectable and the overall bias is only +6.7 points optimistic.

The middle is a disaster. Players the model called 40–60% survived 14% of the time. Players it called 60–80% survived 32%.

And the direction is exactly what the model’s own assumption predicts. Survival multiplies per-pick probabilities as if the picks were independent. Real drafts have runs — three receivers in four picks, because managers watch each other. Independence is least damaging at the extremes (nobody wants him; everybody does) and most damaging in the contested middle, which is precisely where the left panel falls apart.

The fix is a single exponent — replace p with p^γ. An exponent rather than a linear shrink because p^γ fixes both 0 and 1, and a turn boundary (“the window is empty, he is certainly still there”) has to stay exactly 1.0.

The interesting part is that this was not fittable when I first wrote it up. With three drafts, solving for the γ that matched the overall mean gave γ ≈ 4 and threw the middle buckets onto the other side of the truth — the 60–80% bucket went from 68% predicted to 22% against an actual 30%. I concluded that one knob could not fix a bias whose size depends on where you sit in the range, published the miscalibration instead, and left it.

That conclusion was drawn from too little data, not from too simple a model. A fourth draft — 840 more predictions — changed the answer. At γ = 4.11 the same one-knob correction improves every bucket: observation-weighted mean absolute calibration error 0.066 → 0.021, mean bias +0.066 → +0.005, mean Brier 0.061 → 0.044. That is the right panel.

Two things I would not have predicted. The correction still undershoots the low buckets — it just undershoots by much less than the raw form overshot. And it slightly worsens the one 10-team standard-scoring draft (Brier 0.071 → 0.074) while clearly helping the three 12-team half-PPR ones, which suggests γ wants to vary with league size. Four drafts is nowhere near enough to fit that, so it stays a single number with a caveat attached.

One rule falls out of this that is easy to get backwards: never tune γ against --mock. Mock opponents are sampled from the selection model itself, so a simulated draft contains none of the herding the correction exists for and will score it as a regression. Live drafts are the only valid evidence, which is why --replay appends to a committed calibration log after every one.

The lesson survived the fix, and it is not the one I expected to write. Publishing the miscalibration was still right — a number you know is seven points optimistic is usable, and one you merely believe is not. But “I have not fitted a correction and here is why one cannot work” was an overclaim dressed up as rigour. The honest version was always “not yet, on this much data.”

3. A scoring bug that only a real draft could find

Offline mock drafts sample opponents from the very distribution the opponent model assumes. They validate the arithmetic beautifully and can never disconfirm the model. Almost every real bug surfaced in a live Sleeper mock.

The worst one: my roster had one FLEX slot, shared between running backs, receivers and tight ends. The code asked each position independently whether it had flex capacity, so with one flex slot open, RB and WR and TE each believed the slot was theirs. Tight ends won that argument, because tight end has the steepest cliff, and I finished drafts with 3.5 tight ends for one starting spot.

Then a second bug in the same neighbourhood: an empty dedicated slot was credited at replacement level, but an empty flex slot was credited at zero. Filling a flex hole therefore appeared to be worth a player’s entire projection. Sam LaPorta scored +158.5. After the fix: +9.3. A 158-point tight end had been outranking a 190-point running back.

Neither was a maths error. Both were a modelling error about what a roster slot is, and both are now pinned by tests.

4. The board that was four minutes old

Mid-draft, live, the model recommended a player who had been drafted forty picks earlier. Sleeper’s picks endpoint sits behind a CDN with s-maxage=300, and the response carried Age: 261 — I was reading a board frozen at pick 3 while the draft was at pick 43. I made a recommendation off that board and did not catch it.

The fix is one line — a cache-busting query parameter — and the lesson is the expensive part: a dashboard that has quietly stopped updating is worse than no dashboard, because it looks exactly like a working one. The UI now states when data last changed and goes amber, then red, as that ages.

The peer-advice experiment

My league-mates all say the same thing: spend the first two picks on running backs and receivers, never on a quarterback. Conventional, widely held, and testable — because “where the value is” depends on where you pick.

So I forced the first two picks to a given position pair, let the engine draft normally afterwards, and scored the final starting lineup. Same board, same seeds, same simulated opponents; the constraint is the only thing that varies.

Points of projected starting lineup versus the engine’s own unconstrained choice. 12-team snake, 15 rounds, 16 seeds per arm, 384 simulated drafts.

The advice is correct, and it is correct about a seat I do not sit in.

At slot 1 it is emphatically right. Forcing RB-then-RB costs nothing at all (+1, i.e. it is what the engine wanted), and forcing a quarterback first costs 53 points. Left alone, the engine opens with a running back in 27 of its 32 early picks there.

By slot 12 the ordering has inverted. Forcing RB-then-RB now costs 15 points; forcing the quarterback costs 9. And unconstrained, the engine opens with a quarterback more often than anything else — 14 of its 32 first-two picks at that slot, against 9 running backs.

The mechanism is visible on the value-cliff chart earlier in this post. Six running backs carry an ADP inside the first twelve picks, so by slot 12 what remains at the position is a flat block — the next five backs span nineteen points, and it barely matters which one you get. Meanwhile a 12-team league needs only twelve quarterbacks, so the one genuine outlier at the position is still sitting there, 35 points clear of the next one.

The advice is not wrong. It is a statement about where value sits when you pick early, and it is being repeated in a seat where it no longer applies. That is worth more to me than the six points: it is a reason, and a reason survives a change in the player pool in a way that a rule of thumb does not.

Caveat, stated plainly: this is one projection source and one season’s shape. It should be re-run every year, which is why it ships as a script rather than a conclusion.

It does not draft for you, on purpose

Sleeper’s public API is read-only. There are no documented endpoints for submitting a pick, making a trade, or modifying a roster. You could get around that with session scraping or a headless browser driving the web UI. This project deliberately does not, and the constraint is written into its CLAUDE.md so it stays that way.

Three reasons, in order of how much they matter:

1. Terms of service. Automating writes against a platform that does not offer a write API is asking to have the account removed.
2. Brittleness. A scraped session breaks at the worst possible moment, which is during the ninety seconds you have to make a pick.
3. The model does not know things. Breaking injury news lands on Twitter fifteen minutes before it lands anywhere structured. Human veto is not a limitation of the design; it is the most valuable input in the system.

So it ranks, and I click. Dashboard on one monitor, terminal on the second, Sleeper on the third.

Did it work?

Four live drafts, in order — three public mock rooms, then my actual league:

The improvement between draft 3 and draft 4 was not a smarter algorithm. It was the flex fix, which stopped the engine from spending premium picks on a second elite tight end and let it take a running back at the turn instead.

Grades are computed the same way for every team in the league: fill the best legal starting lineup from each roster, total the projected points, and rank. It is the same lineup solver the recommendation engine uses, which means the grade and the advice cannot disagree with each other.

What I would tell someone building this

Ground it in something. The textbook is why the model has defensible positions instead of tuned constants. Where I departed from it — a bench cohort instead of a single replacement rank, Fisher-Jenks instead of k-means — I can say why, and the reason is about the draft clock rather than about taste.

A proxy does not inherit a formula’s authority. CV in Petersen’s sense and CV over weekly scores are the same three symbols measuring different things. The formula did not warn me. The data did.

Simulate to test arithmetic; go live to test the model. Mock drafts sample opponents from the model’s own assumptions. They cannot disconfirm it. Every genuinely interesting bug came from a real draft with real humans.

Publish the miscalibration. The survival chart is the least flattering thing in this repo and the most useful thing in it. Documenting where a model is wrong is not an admission that it is broken; it is the difference between a number you can reason with and a number you can only believe. And keep publishing it: the log I kept because I could not fix the bias is exactly what made a fourth draft enough to fix it.

Look at the chart. Two visual bugs — a sort order that silently did nothing and a label that ran off the frame — survived every test I had and died the moment I rendered the image and looked at it.

Credits and links

• Isaac T. Petersen, Fantasy Football Analytics: Statistics for Fantasy Football — the methodological backbone: replacement level, dropoff, tiering, uncertainty, and the positional strategy that follows from them. Free, rigorous, and the best thing written on this subject. If you take one link from this post, take that one.
• Sleeper API docs — public, unauthenticated, read-only, and generous. No API key required to run any of this.
• Built in Python with pandas, numpy, rich (terminal), and dash + plotly (browser).
• Source, tests and runbook: github.com/godot107/sleeper-drafter

Everything in this post reproduces from the repo:

python scripts/fetch_projections.py         # real projections + ADP
python scripts/fetch_consistency.py         # weekly variance from last season

python scripts/experiment_cv.py             # the CV failure
python scripts/experiment_first_two.py      # the first-two-picks experiment
python main.py --replay <draft_id>          # survival calibration on a finished draft

python main.py --web --mock --slot 12       # the dashboard, on a simulated draft
pytest -q                                   # 163 tests

The shelf problem

My data science journey started almost six years ago, in a master’s program, with the usual stack of textbooks: The Elements of Statistical Learning, Bishop’s Pattern Recognition and Machine Learning, Han’s Data Mining: Concepts and Techniques, Wasserman’s All of Statistics, Goodfellow’s Deep Learning, and a long tail of others I collected as the field moved.

Six years later, those books have a strange status in my life. They taught me the foundations I use every day. They are, collectively, the most trustworthy reference material I own — written by the people who defined the field, reviewed to within an inch of their lives, far more reliable than a random blog post or a confident-but-wrong answer from a chatbot.

And I almost never open them.

The friction is too high. When I need to remember exactly how the bias–variance decomposition falls out, or the precise steps of the KDD process, or which distance metric Han recommends for mixed attribute types, I don’t go digging through a 700-page PDF. I Google it. Or I ask an LLM and hope it isn’t hallucinating. My best sources sit on disk, indexed by nothing, searchable only by my increasingly unreliable memory.

So I built the thing I actually wanted: a private knowledge base over my own library, with a vector database for retrieval and Claude Code as the layer that does the reasoning on top of it. Ask a question in plain English; get back the exact passages from my books, with citations, and an answer that reasons across them.

This post is about how it works, what surprised me building it, and why I think this pattern is badly underused.

The idea: retrieval and reasoning over sources you trust

Most people now reach for a general-purpose chatbot when they have a question. That’s great for a lot of things and genuinely bad for others: the model answers from a blurry average of the whole internet, it can’t cite where an answer came from, and it will occasionally invent a formula with total confidence.

Retrieval-augmented generation (RAG) flips that. Instead of asking the model to recall facts, you:

1. Retrieve the most relevant passages from a corpus you chose and trust (a vector database does this by meaning, not keywords), and
2. Reason over those passages with an LLM, which now has the right source material in front of it.

The corpus is the part everyone underuses. People build RAG over company wikis and support tickets. Almost nobody points it at the canonical textbooks they already own — the highest-signal technical material most of us will ever have access to. That’s the gap I wanted to close.

The reasoning layer matters just as much. I didn’t want a search box that returns ten snippets and makes me read them. I wired the knowledge base directly into Claude Code as a native tool, so the AI can search my library mid-conversation, pull the passages it needs, and reason across three different books to answer one question — citing each as it goes. Retrieval gives it grounding; the model gives it synthesis. Neither half is enough alone.

How it’s built

The whole thing runs locally on a single machine with a consumer GPU (a 6 GB GTX 1660). No cloud, no API bills, nothing leaves the house. The pipeline:

A few of the choices that mattered:

Embeddings. I use BAAI/bge-large-en-v1.5 to turn each chunk of text into a 1,024-dimensional vector that captures its meaning. BGE has a quirk a lot of people miss: it’s asymmetric — you’re supposed to prepend a short instruction to queries but not to stored passages. Get that wrong and retrieval quietly degrades. That one detail is the difference between “this works” and “this is mediocre and I don’t know why.”

Vector store. ChromaDB with a cosine-similarity HNSW index. At my scale — about 600 books, ~78,000 chunks — the index lives comfortably in RAM at query time, so a question comes back in well under a second.

The retrieval pipeline is more than nearest-neighbor. Raw vector search is stage one. On top of it I added a relevance floor (so the system can honestly say “I found nothing relevant” instead of returning garbage), an optional cross-encoder reranker, and MMR diversification so three near-identical passages don’t crowd out the answer. More on the reranker below, because it taught me a lesson.

The integration is the payoff. A small Model Context Protocol server exposes the knowledge base to Claude Code as tools — search_textbooks, list_sources, get_toc, expand_context. The embedding model loads once and stays resident, so there’s no cold start. From my side it just feels like the AI suddenly knows my library: I ask a question, it searches, it reasons, it cites.

The part nobody warns you about: your data is a mess

Here’s the unglamorous truth of any real RAG project. The retrieval algorithm is maybe 20% of the work. The other 80% is discovering that your data is dirtier than you think.

When I actually audited my corpus, I found:

• Duplicate books. The Elements of Statistical Learning had been ingested twice under two different filenames — over 1,600 redundant chunks competing in every search. My first instinct, “just group by title,” was naive: the same audit flagged a hundred-plus genuinely different files (per-section quiz PDFs that happened to share an embedded title). The fix was to detect duplicates by shared chunk content, not metadata — and to divide overlap by the larger document so a short file merely contained in a big compilation isn’t mistaken for a copy. Correctness on a destructive operation is worth the extra care.

• Garbled extraction. One of my most important books — the pre-publication PDF of Goodfellow’s Deep Learning — extracts as duplicated nonsense (Deep Deep Deep Belief Belief Networks orks orks). Another linear-algebra file was pure mojibake. Bad text in produces bad answers out, silently. I wrote a small heuristic to score each source for “looks broken,” and it correctly surfaced the worst offender as #1.

• Unreliable metadata. Embedded PDF titles are frequently Untitled, duplicated, or shared across dozens of unrelated files. Trusting them for citations gives you nonsense provenance. I reconstruct clean titles from filenames instead.

None of this is exciting. All of it is the actual job. A knowledge base is only as trustworthy as the corpus underneath it, so I built the data-quality checks as first-class features — an audit command, a dedup tool, and a health check that runs automatically and warns me when the corpus drifts. If you take one engineering lesson from this post: budget for data quality, and make it a running process, not a one-time cleanup.

The reranker that didn’t help — and why I kept the result

I added a cross-encoder reranker because the literature says it’s the single biggest precision win in RAG, and I believed it. Then I did something I think more people should do before shipping: I built an evaluation harness and actually measured it.

I wrote a small gold set of questions tied to the books that should answer them, and scored the pipeline with and without each stage (recall@k and mean reciprocal rank). The result was humbling:

The reranker made things worse on my metric.

Now — that metric is imperfect. It checks whether the right book showed up, which is saturated and can’t see whether a passage’s content got more relevant (exactly what a reranker is for). So the honest reading isn’t “rerankers are bad.” It’s “on the evidence I have, this reranker model isn’t earning its place, and I will not turn it on by default and pretend it’s an improvement.” I left it implemented, off by default, one flag away, with a written plan to re-test it with a stronger model and a better gold set.

That discipline — measure before you believe, and report what you find even when it’s inconvenient — is, to me, the whole point of doing this seriously. It’s also the difference between a portfolio project and a demo.

What it actually feels like to use

The abstract pitch is “local RAG over textbooks.” The concrete experience is better than that sounds.

I ask, “Summarize the first two chapters of Han’s Data Mining.” The AI pulls the table of contents, retrieves the relevant passages from chapter 1 (the KDD process, data-mining functionalities) and chapter 2 (attribute types, similarity measures), and writes a grounded summary — citing the actual pages, from the actual book, with zero hallucination, because it’s reading my copy.

I ask about the bias–variance tradeoff and get the passage from Practical Statistics for Data Scientists sitting next to the one from a model-selection chapter, and a synthesis across both. I ask about backpropagation and it spreads its answer across three books that each explain it differently — the intuition from one, the math from another, the code from a third.

It is, functionally, a conversation with my own bookshelf — except the bookshelf has read all of itself and can cross-reference instantly.

Why this is underused — and the benefits

This pattern is mature technology now. Vector databases are a commodity, embedding models are excellent and free, and the tooling to wire it all into an AI assistant exists. And yet hardly anyone points it at their own trusted library. A few reasons it’s worth doing:

• Grounded, not guessed. Every answer traces back to a specific page in a specific book you trust. Hallucination drops to near zero because the model is reading, not recalling.
• Private by default. It runs locally. My notes, my library, my questions never leave my machine. For anyone working with sensitive or proprietary material, that’s not a nice-to-have.
• You curate the truth. A general chatbot averages the internet. This averages nothing — it answers from the canonical sources you deliberately chose. Signal over noise.
• It compounds. Every book I add makes it smarter. Six years of accumulated material stops being dead weight on a drive and becomes a living, queryable asset.
• Retrieval plus reasoning. Search alone gives you ten snippets to read. An LLM alone gives you confident guesses. Together they give you cited, synthesized answers — the actual thing you wanted.

The cost of entry keeps falling and the payoff is immediate. If you’ve got a folder of PDFs you “keep meaning to reference,” you are one weekend away from never having to dig through them by hand again.

Where it’s going next

The natural next step is to lift the vector store off my desk and into the cloud behind a small service endpoint, so the knowledge base is reachable from anywhere and from other tools — not just the machine it lives on. Managed vector databases and a thin API layer make that straightforward, and it turns a personal project into something I could actually share.

But honestly, even as a local tool, it’s already changed how I work. The books that taught me the field six years ago are finally pulling their weight again — not as decoration on a shelf, but as the grounded memory of an AI I can think alongside.

That’s the part I find genuinely exciting. We spend a lot of energy asking what AI knows. It turns out the more useful question is: what do you want it to know? Point it at the right sources, give it a way to retrieve and reason over them, and the answer becomes — whatever you’ve spent your career learning to trust.

Built with Python, ChromaDB, BGE embeddings, and Claude Code. Runs on a single consumer GPU. Code and design notes are on GitHub.

When building my predictive maintenance pipeline, I hit the classic data science milestone: my Convolutional Neural Network (CNN) scored 100% accuracy. After investigating, I realized the dataset itself was exceptionally clean under specific conditions, and the signal processing techniques I used (like the Fast Fourier Transform) had perfectly isolated the faults.

But what was the FFT actually doing mathematically? Why did I need to use imaginary numbers in my Python code? And how do the layers of a CNN actually connect without causing matrix multiplication errors?

To break this down, I fired up the AGY CLI (Anti-Gravity CLI) for an interactive pair-programming session. I wanted to step out of pure software engineering and dive into the domain of applied math. What followed was an incredibly fun journey that transformed those complex “black box” algorithms into pure, visual intuition.

Here is a transcript of my session with Anti-Gravity.

1. The CNN “Matrix Mismatch” Error

Me: I’m new to CNN architecture. How does one design the inputs of one layer and the outputs of the next to be compatible? If I do it by hand, I get a lot of matrix multiplication errors. How do I build intuition for this?

Anti-Gravity: This is the exact hurdle everyone faces when learning deep learning! To build intuition, separate your tensor into two concepts: Channels and Spatial Dimensions (Height x Width).

1. The Plumbing Rule (Channels): When stacking Convolutional layers (Conv2d), you only need to worry about the channels. If Layer 1 outputs 16 pipes, Layer 2 must have exactly 16 intakes. PyTorch handles the spatial dimensions shrinking underneath.
2. The Danger Zone (Flattening): 99% of shape mismatch errors happen when transitioning from Convolutional blocks to Linear (Dense) blocks, which only accept a flat 1D list of numbers.

To avoid the nightmare of manually calculating grid sizes (which breaks if your input image size changes), use the “Cheat Code”: AdaptiveAvgPool2d.

By inserting nn.AdaptiveAvgPool2d((4, 4)) right before flattening, you tell PyTorch: “I don’t care what the height and width of the image are at this point. Mathematically squish whatever you have into a 4x4 grid.” This permanently locks your flattened size to exactly Channels * 4 * 4, completely eliminating shape mismatch errors!

(Note: The math that makes Adaptive Pooling possible was pioneered by Kaiming He et al. in the 2014 paper Spatial Pyramid Pooling, and Global Average Pooling was introduced by Lin et al. in Network In Network. Credit to academia!)

2. What actually is the FFT?

Me: Can you attempt to explain the math behind the Fast Fourier Transform (FFT)? Start simple.

Anti-Gravity: Imagine your messy vibration wave is drawn on a long, straight piece of string. Now, imagine taking that string and wrapping it around a circle.

Most of the time, the string is distributed evenly around the circle, so its “Center of Mass” stays right dead in the middle at (0, 0). But, if you wind the string at a speed that exactly matches a hidden frequency inside the wave (let’s say 107 Hz), every single time the wave hits a peak, it lands on the exact same side of the circle. The string bulges massively to one side, and the Center of Mass shoots outward!

The mathematical equation for this is:

• \(2\pi\): One full rotation around a circle.
• \(\xi\): The frequency (e.g., 107 Hz).
• \(t\): Time.
• \(i\): The imaginary number (which triggers 2D rotation).

The equation simply says: “Spin the wave around a circle at frequency \(\xi\). Does the center of mass bulge outward? If yes, draw a spike on the graph.”

3. Why the “Imaginary” Number?

Me: What is the imaginary number \(i\)? Technically the square root of a negative number is underdetermined and errors out.

Anti-Gravity: You are exactly right! You cannot multiply a number by itself and get a negative number on a standard ruler.

But think about a 1D number line. If you multiply 1 by -1, it flips 180 degrees to -1. But what if you only want to rotate exactly half-way (90 degrees)? You need a number that, if you multiply it twice, gives you -1. That number is \(i\)!

If you multiply 1 by \(i\), it rotates 90 degrees off the number line, pointing straight up into a brand new 2nd Dimension. This creates the Complex Plane, which has an X-axis (Real numbers) and a Y-axis (Imaginary numbers). It shouldn’t be called “imaginary”; it should be called a “Rotational Number.”

Me: Right, because it’s complex math, you’re mapping the imaginary number (rotation) back to the original dimension, the x-axis.

Anti-Gravity: Exactly! The FFT does its rotation math and says, “At 107 Hz, the center of mass shifted to the coordinate \(3 + 4i\).”

But your Machine Learning model can’t read 3 + 4i. So, in Python, we use np.abs(). Under the hood, this calculates the actual physical distance from the center (0,0) to the dot (3, 4) using the Pythagorean theorem: \(\sqrt{3^2 + 4^2} = 5\).

It takes the 2D rotational math, collapses it back down to a single real number (5), and plots that number on your Y-axis. That number 5 is exactly what your CNN or Random Forest looks at to determine if the motor is breaking.

Conclusion

Math gets a bad reputation because it’s usually taught as a bunch of disconnected equations to memorize. But when you realize that things like the imaginary number \(i\) or the AdaptiveAvgPool2d layer are just incredibly clever “hacks” invented to solve real-world geometry and engineering problems, it completely changes the perspective. It stops being “black magic” and starts feeling like a superpower.

Wow, math is fun.

Walk through any refinery, power plant, or pump station and you’re surrounded by machines that spin: motors, compressors, turbines, gearboxes. Almost all of them ride on bearings — and when a bearing starts to fail, it doesn’t send a calendar invite. A cracked race or a spalled ball can take a multimillion-dollar compressor offline in minutes, halt a production line, or turn into a genuine safety incident.

The good news: a failing bearing whispers before it screams. Long before a human can hear or feel anything, the vibration signature changes in tiny, repeating ways. Predictive maintenance (PdM) is the art of listening for those whispers. This is the story of building a system that does exactly that — and of a plot twist that taught me more than the model did.

This article is the narrated version of a project on GitHub: github.com/godot107/predictive-maintenance-cwru. Everything here is reproducible from that repo.

The data: four bearings, four fates

I used the Case Western Reserve University (CWRU) Bearing Dataset, a benchmark in the condition-monitoring world. An accelerometer sampled at 12 kHz records the vibration of a motor’s drive-end bearing under four conditions:

• Normal (healthy)
• Inner Race fault
• Ball fault
• Outer Race fault

The goal: feed the model a slice of raw vibration and have it name the fault.

But you can’t just throw a wiggly line at a neural network and hope. The signal has to be transformed into something a network can see. That’s where two analogies make everything click.

Two analogies that make signal processing click

🥤 The Fourier Transform is a smoothie un-blender

A raw vibration signal is like a smoothie: strawberry, banana, and spinach all blended into one. Looking at the smoothie, you can’t tell what went in.

The Fast Fourier Transform (FFT) is a magic blender run in reverse. Pour in the smoothie and it hands back the ingredients — “30% strawberry, 50% banana, 20% spinach.” For a bearing, those ingredients are frequencies, and a fault adds a specific new “flavour” that shouldn’t be there.

How it works in code (and physics)

Physically, when a bearing is healthy, it hums along at a smooth, predictable baseline. But when a crack forms on the inner race, every time a ball bearing rolls over that crack, it creates a tiny microscopic “click” or impact. Because the motor is spinning at a constant speed, these clicks happen at a very regular interval (a specific frequency).

The FFT takes the messy, noisy vibration wave and mathematically isolates that exact repeating “click”, plotting it as a giant spike on a graph. In Python, isolating these frequencies is incredibly elegant using NumPy’s real-valued FFT (rfft):

import numpy as np

# 'segment' is a slice of our raw vibration array
# 'fs' is our sampling rate (12,000 Hz)
n = len(segment)

# 1. Get the X-axis (the exact frequency values in Hz)
frequencies_hz = np.fft.rfftfreq(n, d=1.0 / fs)

# 2. Get the Y-axis (the magnitude/loudness of each frequency)
magnitude = np.abs(np.fft.rfft(segment)) * (2.0 / n)

Average the FFT across many windows and the fingerprints jump out — the faults light up high-frequency bands the healthy bearing never touches:

🎼 A spectrogram is sheet music

The FFT has a blind spot: it tells you which notes were played, but not when. Imagine being told a song contains a C, an E, and a G — but not the rhythm. You’d never recognize the tune.

A spectrogram puts the notes back on a timeline. It’s sheet music: time runs left to right, frequency runs bottom to top, brightness is loudness. Now a fault isn’t a single frequency — it’s a pattern of impacts repeating over time, exactly the kind of 2-D structure a Convolutional Neural Network (the tech that recognizes cats in photos) is built to spot.

So the high-level pipeline became:

Raw vibration  ─▶  Spectrogram  ─▶  2-D CNN (on GPU)  ─▶  Fault diagnosis

Under the hood: the CNN architecture

To make the “2-D CNN” step concrete, here is the exact dataflow of the model. It’s a lightweight, 3-block architecture (only ~155,000 parameters) that uses an Adaptive Average Pooling layer. This pooling layer is a crucial design choice: it forces the feature map to a fixed 4×4 size before the classifier, meaning the network won’t break if you feed it slightly different sized spectrograms.

Design consideration: avoiding the “matrix mismatch” error

If you are new to building CNNs, the hardest part is usually preventing RuntimeError: shape cannot be multiplied when transitioning from Convolutional layers to Linear layers. This architecture solves that using two rules:

1. The Plumbing Rule: When stacking Conv2d layers, only the channels (depth) need to connect perfectly. Notice how Block 1 outputs 16 channels, and Block 2 takes in 16 channels. It’s like plumbing — the output pipes must match the input valves.
2. The “Cheat Code” (Adaptive Pooling): MaxPool2d halves the height and width of the spectrogram at every block. Normally, you have to manually calculate the exact final grid size to feed into the Linear layer. If your input image changes size, the math breaks and the code crashes. By inserting AdaptiveAvgPool2d((4, 4)) right before flattening, we tell PyTorch: “I don’t care what the height and width are at this point. Mathematically squish whatever you have into a 4×4 grid.” This permanently locks our flattened size to exactly 64 channels × 4 × 4 = 1024, completely eliminating shape mismatch errors.

Academic credit: The math that makes “Adaptive Pooling” possible — allowing CNNs to accept images of any size without breaking the fully connected layers — was pioneered by Kaiming He et al. in the landmark 2014 paper Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. Furthermore, the practice of heavily pooling spatial dimensions to drastically reduce the parameter count of Linear layers was famously introduced by Lin et al. in the 2013 paper Network In Network.

I wired it up in PyTorch, trained on an NVIDIA GPU, and wrapped it in a Streamlit dashboard that shows the raw signal, the FFT, the spectrogram, and the model’s live verdict. It worked. The test accuracy came back at 100%.

🚀 Try it live: the dashboard is deployed on Streamlit Community Cloud — launch the live demo and diagnose a bearing fault yourself.

And that’s where the project got interesting.

The plot twist: when 100% is a red flag

A perfect score should make you suspicious, not proud. Real-world classifiers don’t hit 100% — so either the problem is trivially easy, or something is leaking.

Classic overfitting looks like high training accuracy, low test accuracy — a gap. But here training and test were both ~100%, with no gap. That pointed at the second culprit: data leakage.

Here was the bug in my evaluation. Each fault class is one long continuous recording. I chopped it into overlapping windows and then split those windows randomly into train and test. Because the windows overlapped, near-duplicate slices ended up on both sides of the split. The model wasn’t generalizing — it was recognizing windows it had half-seen in training. The test set was lying to me.

So I rebuilt the evaluation the way it should have been done from the start:

1. Leakage-free splitting — cut each raw recording into train/validation/test spans by time (with a guard gap) before windowing, so no slice is shared.
2. A validation set + early stopping — stop on validation loss, report on a test set the model never touched.
3. 5-fold cross-validation — for a stable estimate instead of one lucky split.

The payoff was immediate and humbling. With honest splitting, the early training epochs now showed training accuracy at 1.00 while validation sat at 0.25 — random chance. That genuine overfitting had been completely hidden by the leaky split. The methodology upgrade was doing real work.

The deeper twist: it was still 100%

Here’s what I didn’t expect. Even after removing every trace of leakage, and across all five cross-validation folds, the score held: 1.000 ± 0.000.

So leakage wasn’t the main story. To understand why, I did what I should have done first: I stopped modeling and explored the data.

I computed a set of classic vibration features (RMS, kurtosis, crest factor — all interpretable measures of how “impulsive” a signal is) and projected them to two dimensions with t-SNE. The picture explains everything:

Four clean, perfectly separated clusters. At a single operating condition — one load, one speed — each fault’s signature is so distinct that the classes barely overlap. To prove the point, I trained a plain Random Forest on those hand-crafted features (no deep learning at all). It also scored 1.000.

The lesson: 100% wasn’t a triumph of my model — it was a sign the benchmark was easy. And knowing the difference is the whole job.

Asking the bearing to confirm its own diagnosis

There was one more thing I wanted: proof the model was learning real physics, not an artifact. Bearing engineering gives us a beautiful test.

The CWRU drive-end bearing is an SKF 6205, and its geometry fixes the exact frequencies at which each defect “rings” — the ball-pass frequencies of the outer race (BPFO), inner race (BPFI), and so on, all set by the shaft speed. Using envelope analysis (the demodulation technique vibration engineers actually use), I extracted the impact-repetition rate from each fault and checked it against theory:

The peaks land exactly where the physics predicts: the Outer Race fault rings at BPFO (107 Hz), the Inner Race fault at BPFI (162 Hz). The signal carries genuine, explainable diagnostic content — and, honestly, the Ball fault is the subtle one, with a weak envelope signature, which is precisely what bearing theory says to expect. Even the exceptions agreed with the textbook.

What this project actually demonstrates

The deliverable people see is a dashboard that classifies bearing faults. The deliverable that matters is the judgment around it:

• Engineering it end-to-end — data ingestion, signal processing, a GPU-trained CNN, and an explainable UI.
• Distrusting a good result until the evaluation earns that trust — and knowing that leakage, not the model, is usually the thing to interrogate first.
• Explaining it in plain language — to a recruiter with a smoothie, to an engineer with an envelope spectrum.

That last part is the heart of an AI Solutioning Consultant’s job: translating between the business problem (“don’t let the compressor fail”) and the technical reality (“here’s what the data can and can’t tell us, and here’s how I know”).

So I ran the benchmark I’d been saving for last — the genuinely hard one: cross-load generalization. CWRU recorded each fault at four motor loads, so I trained on three and tested on a load the model had never seen — a true change of operating condition, not just a clean split. I expected the accuracy to finally crack.

It didn’t. Leave-one-load-out held at 1.000 ± 0.000, and even training on a single load and testing on the most distant one stayed at 0.998. The physics explains it: a defect’s frequencies are fixed by the bearing’s geometry, which barely shifts across this ~4% speed range, while the per-window normalization removes the amplitude changes the load does cause. What survives is the fault-frequency pattern — exactly what the CNN keys on. Load transfer simply isn’t the bottleneck for naming the fault type.

So the genuinely open problems lie elsewhere: severity grading (telling a 0.007″ defect from a 0.021″ one of the same fault type) and noisy, multi-fault field data. Knowing where the difficulty actually lives — and where it doesn’t — turned out to be the most useful thing the whole exercise produced. It’s the next milestone in the repo.

Try it yourself

• Live dashboard: launch the Streamlit app
• Code & full write-up: github.com/godot107/predictive-maintenance-cwru
• Dataset: CWRU Bearing Data Center
• The best 20 minutes on Fourier transforms: 3Blue1Brown

If you take one thing from this: when your model scores 100%, don’t celebrate — investigate. The most valuable result in this whole project was the one that looked too good to be true.

Appendix: Glossary of terms

If you’re new to the signal processing or machine learning terminology used in this article, here is a quick guide to help clarify the concepts:

• Predictive Maintenance (PdM): The practice of continuously monitoring equipment condition (like vibration or temperature) to predict when it will fail. This allows maintenance to be scheduled right before failure, avoiding both unexpected breakdowns and unnecessary routine part replacements.
• Fast Fourier Transform (FFT): A mathematical algorithm that takes a raw signal measured over time (like a complex vibration wave) and breaks it down into the individual frequencies that make it up. Think of it as a recipe that tells you exactly how much of each “pitch” is in a sound.
• Spectrogram: A visual representation of frequencies as they change over time. If an FFT gives you the specific notes played in a single chord, a spectrogram is the sheet music for the entire song, showing when each note is played and how loud it is.
• Data Leakage: A fundamental mistake in machine learning where the model accidentally has access to the test data during training. In this project’s initial approach, overlapping time windows were randomly split, meaning near-identical slices ended up in both the training and test sets. The model essentially “memorized” the answers rather than learning to generalize.
• t-SNE (t-Distributed Stochastic Neighbor Embedding): A technique used to visualize highly complex data by grouping similar data points together on a 2D map. In this article, it visually proves that the different bearing faults are so mathematically distinct that they form their own isolated “islands,” explaining why the model easily scored 100%.
• Envelope Analysis (Demodulation): A signal processing technique that strips away the loud, high-frequency “carrier” noise of a machine to isolate and reveal the quieter, lower-frequency repeating impacts — such as a bearing ball repeatedly hitting a crack on the inner race.
• Cross-load Generalization (Cross-load Benchmark): The process of training a model on data from a machine operating under one set of conditions (e.g., a 1 horsepower load) and testing it on data from a different condition (e.g., 2 or 3 horsepower). Because machine vibrations inherently change under different physical loads, testing “cross-load” proves whether the model actually learned the physics of a failure, or if it just overfit to the exact sound of a 1 HP motor.

This reflection, sparked by David Brooks’ How to Know a Person and a deeper dive into the science of humor, chronicles an internal mid-life exploration-a quest to foster genuine connection and navigate the shifting seasons of life. It reveals a profound realization: the critical, often underestimated, power of humor.

The Intellectual Depth of Comedy

Comedians, as it turns out, are more than just entertainers; they are astute observers of the human condition. They possess a unique intellectual sharpness, capable of dissecting and articulating the inherent absurdities and oddities of everyday life-a talent perfectly embodied by the observational genius of Seinfeld. Their ability to communicate deep, sometimes uncomfortable truths in a relatable manner is a sign of high social and intellectual intelligence.

Levity and Gravity: The Essential Balance

The journey highlighted the crucial contrast between Levity and Gravity.

• Levity (from the Latin levitas, “lightness”): Refers to a lightness in mood, tone, or behavior, expressed through humor, playfulness, and a deliberate lack of excessive seriousness. It is the necessary antidote to the constant pressures of adult life.
• Gravity (from the Latin gravitas, “heaviness”): Encompasses seriousness, solemnity, and physical weight.

In an era defined by demanding work schedules and constant professional pressure, the authors argues that we are often overburdened by Gravity. Understanding and actively seeking Levity is therefore not a frivolous pursuit, but a vital mechanism for psychological health and relational well-being.

Mapping Your Humor Style

The book presented a framework for understanding one’s dominant humor style, categorized by the dimensions of Aggressive vs. Affiliative and Subtle vs. Expressive:

1. The Sniper (Aggressive-Subtle)

Edgy, sarcastic, and nuanced.

Delivers humor with a dry, deadpan style, often under the breath. Unafraid to cross lines for the laugh.

An “acquired taste.” Observes from the sidelines, waiting for the perfect, deadly one-liner.

2. The Stand-up (Aggressive-Expressive)

Bold, irreverent, and natural entertainers.

Comfortable as the center of attention. Uses roasts and pranks, unafraid to ruffle feathers.

Has thick skin; views being the butt of a joke as a sign of affection or engagement.

3. The Magnet (Affiliative-Expressive)

Charismatic, warm, and consistently positive.

Uses animated delivery, including slapstick or impersonations, to radiate good cheer.

A mood-booster. Generous with their own laughter and avoids controversial topics.

4. The Sweetheart (Affiliative-Subtle)

Earnest, understated, and empathetic.

Prefers planned humor (e.g., a thoughtful line in a speech) over off-the-cuff jokes.

Uses humor to uplift others. Avoids teasing to ensure no feelings are hurt.

I personally identifies with The Sweetheart-motivated by a deep-seated fear of causing offense-and looks to The Stand-up with admiration. Trevor Noah’s 2022 White House Correspondents’ Dinner is cited as a masterclass in this style: navigating a politically diverse and tense audience, successfully delivering sharp jabs while simultaneously defending the American press and the 1st Amendment-a rare blend of high-stakes comedy and serious civic commentary.

The Profound Benefits of Laughter

Humor is a tool of immense power, capable of elevating one’s perceived intelligence and earning respect. Its utility extends even into tragedy, acting as a crucial mechanism for collective processing and moving forward.

The physiological effects of laughter are akin to a naturally administered “drug cocktail,” releasing vital neurochemicals:

• Endorphins: Natural pain relievers and mood elevators.
• Dopamine: Associated with pleasure, reward, and motivation.
• Serotonin: Regulator of mood, sleep, and appetite.
• Oxytocin: The “bonding hormone,” which promotes trust and connection.

This chemical release actively counteracts stress by reducing the levels of stress hormones like Cortisol and Epinephrine.

Deconstructing the Mechanics of Humor

Humor is an art form rooted in specific, measurable components:

• Timing: The precise moment of delivery.
• Distance: The necessary space-temporal (from a tragic event), geographical, or emotional-required to find comedy in a situation.
• Surprise (Incongruity): The unexpected twist or deviation from a listener’s expectation.

The Onion ‘s first publication after 9/11 serves as a classic example of using humor-specifically absurdity and incongruity-to process the unimaginable. By creating headlines like “HUGGING UP 76,000 PERCENT” or “NOT KNOWING WHAT ELSE TO DO, WOMAN BAKES AMERICAN-FLAG CAKE”, they provided “comic relief when I desperately needed it to help me process something unimaginable,” turning collective shock into a small, shared step toward healing. As the famous saying goes, there is a thin margin between tragedy and comedy.

The Adult Dip in Joy and the Power of Re-framing

The observation from Inside Out 2 -”I guess that’s what happens when you grow up; you feel less joy.”-highlights a natural dip in the capacity for laughter and joy that occurs after adolescence and persists through the long “middle years” (30s to 60s), only to rise again in later life. Adult responsibilities and “adulting” create a gravitational pull away from Levity.

However, humor offers a vital escape hatch: it allows us to find hidden truths in life, especially when reflecting on past hurts and trauma. By applying a touch of humor and Levity, old stories can be re-framed and viewed in a new, less painful light.

Humor in Power, Leadership, and Communication

Humor is a powerful tool in social and professional dynamics:

• Minimizing Power Distance: A light joke from an executive or CEO in an elevator can instantly reduce the perceived hierarchy, creating a more relaxed and connected environment for co-workers. (Caveat: Excessive self-deprecation, however, can project insecurity.)
• Delivering Hard Conversations: Humor can be used to send a provocative yet non-threatening message. An anecdote details a female boss who, finding her male co-workers unconsciously excluding her by continuing work conversations in the men’s room, joked about joining them. This lighthearted line sent a clear, powerful message of “I see you; please stop,” without escalating into a confrontation.

Cultivating a Culture of Levity

The essay explores inspiring, if extreme, examples of institutionalizing Levity to create psychological safety-a condition where people feel comfortable being themselves and taking creative risks.

• Hiroki Asai (Apple): Believed “fear is the greatest killer of creativity.” He planned elaborate, surprise-filled events designed purely for laughs, using humor as a way to boost energy and create shared joy:
• The Blue Man Group: Employees dressed up to surprise an audience.
• The “Chase” Scenes: Gag videos featuring high-speed pursuits of Asai himself.
• The Gospel Choir: The most successful example, where a dry presentation was transformed into a high-energy moment of joy by revealing a full gospel choir, centered around a “Hidden Gem” (a junior designer who was a trained singer).
• Eric Schmidt (Sun Microsystems): Early in his career, he staged an elaborate prank by disassembling and reassembling a Volkswagen Beetle inside the office. This signaled that the company was a place that empowered people to be themselves and have fun within appropriate boundaries.

A Call to Action for Digital Citizenship

These work environments offer a blueprint for life, especially as modern existence increasingly shifts into the digital world. The final reflection is a personal call to action: to deliberately exercise the muscle of humor, to apply Levity to daily life and digital interactions, and to actively uplift the people around us. The world, burdened by Gravity, desperately needs it.

The journey concludes with the ultimate personal commitment: seeking an improv comedy class-a challenge to overcome the “Sweetheart’s” fear of the spontaneous and potentially offensive, and a commitment to prioritizing shared laughter, even if it means commuting to a club instead of maximizing family time, as the author muses, “what’s more important than taking care of the family, then commute to a comedy club and learn improve with complete strangers?”

Originally published at https://godot107.github.io.

The Question That Wouldn’t Go Away

“How to know a person?”

My initial reaction to that title was a dismissive eye-roll. Seriously? I’ve been navigating this world long enough. I have friends. I can hold a conversation. I can read an article and understand a biography. Why on earth would I need a book to teach me something so fundamental?

But I picked up David Brooks’s How to Know a Person anyway. And I couldn’t put it down.

I found myself struggling with a deeper question-one that felt sudden but was, of course, years in the making: How did we get here?

I’m talking about the atmosphere of political violence and the relentless tribalism that has settled over America. There’s a toxic sense that violence is justified for “my side” because the stakes-the ideology-have become existential. Like the chain of events leading up to the “powder keg” of World War I, our current division feels like the culmination of years of disconnection.

Listening to Brooks discuss this on NPR, I was intrigued by his final note of hope. He suggested the antidote to this existential polarization wasn’t a policy shift, but something far more intimate: a change in how we see each other.

Coincidentally, this quest for connection was also hitting close to home. Between my full-remote work and a changing life stage, I was struggling to genuinely socialize with people. The book hit me right in the soul: everyone on this planet yearns to be seen-not just physically, but inwardly-to be valued, heard, and understood.

The Guilt of the ‘Brand’

As I read, I faced a humbling realization: I am guilty of dehumanization.

I have been reducing people to their “brand”-how they are known, their profession, or the ‘group’ they are most obviously associated with (ethnic, political, etc.). I was “putting people into boxes.” I’d meet someone and immediately start filling in the blanks based on stereotypes or superficial similarities to others I knew.

Brooks argues that every person is a one-of-a-kind at this precise moment in time, shaped by a unique blend of ancestry, circumstance, experiences, decisions, and relationships.

For example, I identify as an Asian-American, and my life is undeniably traced to my ancestry, yet it’s been uniquely shaped by being the son of refugees fleeing a war-torn Cambodia. My “group” gives context, but my personal story defines the individual.

We are not static, either. We are not frozen in our “greatest hits” or our deepest wounds. A person I was ten years ago, or a person who has been changed by trauma-like the survivors of the tsunami mentioned in the book-is different from the person they are today. We are all living, adapting, and changing narratives.

Are We Even Listening? The Epidemic of Blindness

We know that empathy and listening are important. My head knows it. But my heart, my pride, and my ego often fight against it.

I believe our current social and economic structure-from the echo chambers of social media to the pressures of hyper-individualism-incentivizes this “Epidemic of Blindness.” Brooks cites that empathy and community are in decline, evidenced by the struggles of non-profits to fill volunteer roles.

But the core issue is not just a lack of goodwill; it’s a lack of true attention.

Most of the time, we aren’t listening; we’re just waiting for our turn to speak or shift the focus back to ourselves. We treat our attention span like a light dimmer, slowly inching up the focus (if at all) while one eye is on our phone or our mind is elsewhere.

Brooks argues that true listening must be an on/off switch: “May I have your undivided attention.” This is the practice of “framing,” putting the camera entirely onto the other person. He shares his own regretful experience on a panel where he failed to listen, prioritizing his own piece over genuinely engaging with the speaker.

The Humility of ‘Illumination’

The book taught me that empathy is a practiced skill. It is not, as the cliché goes, about “putting yourself in their shoes,” because that would require lived experiences I simply do not possess. I am not a single mother, nor am I a refugee fleeing violence. How I perceive the world is vastly different from their mental model.

Empathy is about recognizing this difference and actively trying to see the object or issue through their perspective.

Brooks calls the moment this clicks “illumination,” and it is incredibly powerful and humbling. He references Jimmy Dorrell, a man who sees the inherent worth in every human being, believing that each person possesses a “transcendent spark” -a soul-which makes us all equal at the deepest level.

This humility means admitting my assumptions are often wrong. It means knowing that I don’t know everything about a person before we even meet.

The Hope That Reaches Across Continents

This book has done the work of softening my heart, especially toward people I might otherwise view as an “enemy” simply because of a political or ideological disagreement. I am learning to silence my ego, which loves a good argument, and simply seek to understand.

What gives me profound hope is the idea that two people, with vastly different experiences, can truly empathize and understand one another. Brooks shares the powerful friendship between a writer and a man who survived the Rwandan genocide, a story that underlines both the danger of dehumanization and the power of connection.

The Rwandan genocide was a horrific reminder of what happens when empathy is completely annihilated-over a million Tutsi people were murdered, often by their own neighbors, because of ethnic hatred woven with a complex history.

Yet, from that tragedy, one man’s story offers redemption. Deo, a survivor, returned to his home country of Burundi to open a hospital that served both Hutu and Tutsi belligerent groups, treating everyone for medical care regardless of which side they were on.

In the end, I’ve found a renewed desire to rediscover what it means to be human-with common struggles, common experiences, and common joys-not just what I do for a living or what content is curated for me.

“I am human, and nothing human is alien to me.” — Terence (p. 121)

Thank you, David Brooks, for intersecting various disciplines and compelling stories into a cohesive journey. This is a book I will be returning to often.

What is a book you’ve read that fundamentally changed how you interact with people?

Consider a read by purchasing or borrowing from your local library.

AI Transparency Note: This article was edited and polished using Generative AI. All opinions expressed are solely my own, and I remain entirely responsible for the final content and accuracy.

Originally published at https://godot107.github.io.

This notebook demonstrates a baseline approach to sentiment analysis using TF-IDF (Term Frequency-Inverse Document Frequency) vectorization and Logistic Regression. While simpler than modern deep learning approaches, this method provides a solid foundation for understanding text classification.

📚 Concepts

TF-IDF Vectorization

TF-IDF is a statistical measure that evaluates how relevant a word is to a document in a collection of documents (corpus). It balances term frequency with word uniqueness, ensuring that frequently occurring but unimportant words do not dominate text representations.

Components of TF-IDF:

1. Term Frequency (TF): Measures how often a term appears in a document.
2. Inverse Document Frequency (IDF): Measures how important or unique a term is across all documents.

Why Use TF-IDF?

• Downweights common words (e.g., “the”, “is”) while highlighting informative terms.
• Emphasizes distinctive words, making text classification more effective.
• Handles varying document lengths by normalizing term frequencies.

Logistic Regression

A linear classification model that predicts probabilities using the sigmoid function.

Key Advantages:

• Handles high-dimensional, sparse data well (like TF-IDF features).
• Interpretable: Provides feature importance via learned weights.
• Efficient and scalable, making it a great baseline model.

🎯 Implementation Details

1. Data Preprocessing

• Handling missing values.
• Converting ratings to sentiment labels (negative, neutral, positive).

2. Feature Engineering

• TF-IDF vectorization to convert text into numerical features.
• Stop word removal to eliminate common words that add little value.
• Feature limitation: Restricting to the top 10,000 most frequent terms to reduce sparsity.

3. Model Training

• Train-test split with stratification to maintain class distribution.

1. Bag-of-Words Limitations

• Loses word order information (e.g., “not happy” and “happy” are treated similarly).
• Cannot capture context beyond individual words.
• Struggles with negations (e.g., “not good” vs. “good”).

2. Vocabulary Issues

• Out-of-vocabulary (OOV) words in test data may not be well represented.
• Sparse feature matrix leads to memory inefficiency.
• Fixed vocabulary: Adding new words requires retraining.

3. Class Imbalance

• Biased predictions toward the majority class.
• Need for proper evaluation metrics beyond accuracy.
• Importance of stratified sampling to maintain label proportions.

🚀 Potential Improvements

1. Text Preprocessing

• More sophisticated text cleaning (handling contractions, special characters, etc.).
• Lemmatization instead of stemming for better root-word representation.
• Explicit negation handling (e.g., replacing “not happy” with “not_happy”).
• Emoji and emoticon processing for sentiment-rich symbols.

2. Feature Engineering

• N-gram features (bigrams, trigrams) to capture phrase-level information.
• Custom stop words tailored to the domain.
• Part-of-Speech (POS) tagging to identify important words.
• Named Entity Recognition (NER) for identifying entities.

3. Model Enhancements

• Ensemble methods (e.g., Random Forest, XGBoost) for better performance.
• Feature selection to remove noisy terms.
• Hyperparameter tuning using GridSearchCV.
• More sophisticated cross-validation strategies (e.g., stratified k-fold).

4. Advanced Techniques

• Word embeddings (Word2Vec, GloVe) for dense representations.
• Deep learning models (BERT, RoBERTa) for context-aware sentiment analysis.
• Transfer learning to leverage pre-trained language models.

📊 Evaluation Metrics

Key Metrics:

• Accuracy: Overall correctness of the model.
• F1-score: Balances precision and recall, especially for imbalanced datasets.
• Precision & Recall: Measures of positive prediction quality and coverage.
• Confusion Matrix: Provides insights into misclassification.
• ROC-AUC Curve: Evaluates classification threshold performance.

🔍 Use Cases

This approach is particularly useful for:

1. Baseline model development before exploring deep learning.
2. Quick prototyping with minimal computational cost.
3. Small to medium-sized datasets where deep learning isn’t necessary.
4. Interpretability-focused applications, such as legal or financial text classification.
5. Limited computational resources, where traditional ML is preferable.

📝 Next Steps

To further improve sentiment analysis:

1. Experiment with advanced feature engineering techniques.
2. Implement more robust text preprocessing strategies.
3. Optimize hyperparameters and cross-validation methods.
4. Compare performance against transformer-based models.
5. Analyze misclassified examples to refine the approach.

📚 References

1. Scikit-learn TF-IDF documentation
2. Introduction to Information Retrieval by Manning, Raghavan, and Schütze
3. Pattern Recognition and Machine Learning by Christopher Bishop
4. Understanding TF-IDF (Term Frequency-Inverse Document Frequency)

Celebrating Four Years with Lady Bird

April 1st marks four years since we brought home our rescue dog, Lady Bird. No joke — April Fools’ Day became the start of a heartfelt journey. Adopting a rescue dog is a leap of faith; you’re taking on an unknown history, and Lady Bird came with her share of challenges, especially separation anxiety.

Understanding Separation Anxiety

Until you’ve experienced a dog with separation anxiety, it’s hard to grasp the weight it adds to daily life. Lady Bird had been through at least five homes before finding her forever home with us. Interestingly, we even ran into one of her previous foster owners in our apartment building — a small world indeed. To help Lady Bird adjust, we sought professional help from a separation anxiety consultant. The process involved structured exercises, gradually increasing the time she spent alone, starting with just a few minutes.

Memorable Incidents and Challenges

Lady Bird’s separation anxiety has led to some memorable incidents:

• She tore up blinds.

• Opened the bottom freezer door by pulling it out. And taken the liberty to eat my chicken tenders!
• Managed to turn on a stove burner.
• Opened kitchen cabinets and pulled out items like an Instant Pot lid.
• Bit into plants.
• Climbed onto furniture to peer out the window.

• Knocked down pillows.
• Broke through her crate.
• Tore into door molding.

She’s also scared of fireworks and thunderstorms, and she loves chasing small animals — very normal dog behaviors! Despite all this, she’s made progress. We’ve learned to control her environment by lifting blinds and removing temptations. To track her progress, we’ve diligently logged every time we left her home alone.

Lessons in Patience and Emotional Growth

Lady Bird has taught us patience and, in many ways, helped prepare us for the arrival of our newborn. She has a soft, gentle energy and would never hurt anyone. I’d like to think she has emotional intelligence — when a guest is sharing something difficult, she’ll lean her body against them as if to offer comfort.

Lady Bird and Kids

Lady Bird is wonderful with kids. They treat her like a petting zoo animal, and she patiently endures until she needs a break. When she’s had enough, she’ll retreat under the table or to one of her three beds around the house.

A Community Effort

Raising a dog like Lady Bird takes a village, and we’re incredibly grateful for our extended family. They’re always willing to dogsit, even for long intervals, and their own rescue Border Collie reflects their big hearts and passion for animals.

Why Adopt a Dog?

Adopting a dog is not for everyone, but if you want to make a difference, celebrate breakthroughs, and enjoy the loyalty of a constant companion, it’s worth it. Lady Bird has been our fitness accountability partner, a source of laughter with her quirks, a zero-waste helper by eating leftovers, and a snuggle buddy on the couch.

Special Thanks

Special thanks to Austin Pets Alive (@AustinPetsAlive) for their mission as a no-kill shelter. Organizations like theirs make stories like Lady Bird’s possible.

As the year ends, I find myself reflecting on the challenges, growth, and victories that have defined 2024. It’s been a year of learning to let go, leaning into gratitude, and striving to become a better version of myself.

Struggles

1. Finding Contentment
Navigating contentment in a world of comparisons is tough. I’ve wrestled with envy—good envy can motivate growth, as explored in Never Enough, but it’s a fine line that often drifts into imposter syndrome. Envy for opportunities sometimes makes it hard to focus on helping others, but I’ve learned from my uncle’s wisdom: generosity has a way of returning, though you shouldn’t expect it. It reminds me of the Swordsmith Village Arc from Demon Slayer—a lesson in humility and perseverance.

Pride has also been a struggle. I’ve felt entitled to more because of the effort I put in, but life doesn’t work on a transaction system. The real reward is in the journey, not the accolades.

2. Information Overload
I’ve read books on fatherhood, faith, and self-improvement, but sometimes it feels like I’m chasing an unattainable ideal. Social media doesn’t help—it’s a highlight reel that amplifies feelings of deficiency. I limit my time on these platforms to avoid falling into the trap of seeking validation. Honestly, I thank God I’m not an influencer; I’d be homeless.

3. Financial Pressures
Inflation has made financial management a constant stressor. I’ve stretched every dollar—shopping at Costco, prepping food, maintaining my car, and even attempting DIY home repairs. Tight finances have taught me to appreciate labor and resourcefulness, values I hope to instill in future generations. They remind me of stories my in-laws shared about the Great Depression: scraping by, learning new skills, and making the most of what you have.

4. Preparing for Fatherhood
Fatherhood has been a mix of anticipation and anxiety. This year, my wife and I immersed ourselves in newborn classes and online content, navigating an industry full of advice and products. Despite the resources, I often wonder about the simplicity of trusting your gut and leaning on intuition.

5. Career Stagnation
While I graduated from grad school this year, the job market has been disheartening. Without new opportunities, it’s hard to feel accomplished, but I remind myself that growth isn’t always linear.

---

Wins

1. Embracing Hospitality
One of my greatest joys this year has been sharing baked goods and hosting guests. Whether serving drinks, baking for my church network, or connecting with neighbors, hospitality opens people’s hearts.

2. Becoming a Father
The biggest win of 2024: becoming a father. Earlier this year, I read a book emphasizing that failing as a father means failing at life. That resonated deeply. It’s about legacy—what you leave behind in your children’s lives. A colleague once told me, “If you don’t spend time with your kids, don’t expect them to spend time with you.” This dual responsibility of loving and leading them humbles me.

Watching my wife transform from a mother-to-be into an incredible mother has been awe-inspiring. Parenthood has taught us to advocate for ourselves and each other, a necessity in today’s complex healthcare system.

---

Lessons

1. Attention Shapes Identity
In my early 30s, I’m starting to recognize the importance of where I direct my attention. John Mark Comer wrote that our habits define who we are, and habits are formed by micro-decisions. The consequences compound as we age. This principle is evident in the elderly around us—some are patient saints, while others are bitter. The difference lies in the habits and attitudes they’ve cultivated over decades.

2. The Power of Mattering
Mattering—feeling needed and valued—is more important than ever. The loneliness epidemic is real, as highlighted by the U.S. Surgeon General’s study. Community breakdowns and isolation are growing issues, especially in car-centric cities like Houston, where intentionality is required to create meaningful connections.

Hosting a small Bible study with my wife has been one way to counter loneliness and build community. It’s a small step, but it reinforces that we matter—not because of our accomplishments but because of our inherent dignity and ability to contribute.

3. Legacy and Gratitude
As the son of refugees, I carry a story of survival and resilience. My family’s journey reminds me of the privilege of being here and the responsibility to make it count. Yet, being Cambodian-American often feels invisible; most people can’t find Cambodia on a map. Despite this, I strive to believe in my worth and see others as people of value.

---

Closing Thoughts

Our pastor often says, “Life moves at the speed of relationships.” This year has reinforced that truth. Relationships—whether through hospitality, community, or fatherhood—are the heart of a meaningful life.

As I look to 2025, my focus is on nurturing these connections, finding contentment, and leaving a legacy of love, kindness, and purpose.

Here’s to growth, gratitude, and grace.

The global refugee crisis, driven by factors such as economic instability, sociopolitical unrest, war, violence, and natural disasters, has led to unprecedented levels of forced displacement. Many host countries, often low-income nations according to the World Development Indicators (WDI), face immense challenges as they strive to accommodate refugees.

As The World Bank notes, “Host communities need support, too. The forcibly displaced often live in poor areas in developing countries that are struggling to meet their own development goals. Accommodating the sudden arrival of masses of newcomers presents a challenge for host governments, putting further pressure on their ability to deliver basic services and infrastructure.”

Understanding the complex dynamics of both displaced populations and the challenges faced by host nations is crucial for addressing this global crisis effectively.

Notebook

Source

Interestingly, when focusing on low-income countries or the ‘Europe & Central Asia’ region, we observe a higher correlation between “GDP (current US$)” and the refugee population by country or territory of asylum. This aligns with the intuition that larger economies, as measured by GDP, have a greater capacity to host refugees. However, even among low-income countries, there is a strong correlation between GDP and refugee populations. This suggests that higher GDP, even within the lower income bracket, may play a role in hosting refugees. In the next iteration of this report, I plan to label-encode income classes and one-hot encode regions before applying Pearson correlation for deeper insights.

To narrow the scope, I will focus on the top 10 countries where refugees seek asylum and the main countries from which they are fleeing. Based on the data, Jordan, Pakistan, and Lebanon rank as the top three countries hosting refugees from other nations.

Alternatively, I aimed to investigate the countries producing refugees to better understand the underlying factors, such as economic hardship or political instability. The findings align with expectations based on international headlines, such as Syria’s ongoing civil war and the political upheaval in Afghanistan following its government transition.

Visualizations:

Time series for Indicators by Income Group

Encoded by year as a position, value of indicator as position, and Income Group by Color.

I wanted to explore Refugee and Asylum trends using Income Group defined by the WDI and as expected the low income countries produce the most refugees.

In the chart, refugee population increases among all income groups, but ‘Low Income’ countries bear the brunt

Time Series Boxplot for Refugee asylum and origin (Top 10, respectively

The distributions of both “Refugee population by country or territory of asylum” and “Refugee population by country or territory of origin” exhibit a strong right skew with notable outliers, indicating that they are not normally distributed. Over time, the spread of “Refugee population by country or territory of asylum” tends to decrease, while the “Refugee population by country or territory of origin” shows an increasing trend. Although we considered using distribution plots and histograms, they were ultimately not chosen due to the temporal dimension of the data.

Refugee activity in top 10 countries of asylum and origin

Interestingly, Germany have been accepting a lot of refugees this past decade. Interactivity is enabled to hover points by year.

Economy analysis of countries with refugee activity

As expected countries where refugees originate have a negative GDP and therefore, economy is shrinking. These countries also have GDP less than the average country GDP of that year.

Summary/Conclusion/Takeaways

1. Economic Disparity Among Host Countries: Apart from Germany, the remaining nine host countries are predominantly low-income nations with GDPs significantly below the average.
2. Increasing Extremes in Refugee Population: From 2010 to 2021, the refugee population has been increasingly skewed, with more extreme variations each year.
3. Economic Strain on Low-Income Countries: Low-income countries are shouldering the majority of the refugee burden, which poses a substantial risk to their own economic stability.

Recommendation: There is a critical need for investment from higher-income countries into lower-income nations to support their capacity to host refugees and mitigate the economic strain.

Bonus Visualization:

An interactive map built using Vega and Altair where the parameters are the year and interested Indicator reported by the World Bank.

References/Resources/Acknowledgements

1. Samples
   - https://datascientyst.com/flatten-multiindex-in-pandas/
   - https://www.statology.org/pandas-pivot-table-to-dataframe/
   - https://jakevdp.github.io/PythonDataScienceHandbook/01.07-timing-and-profiling.html
2. Documentations
   - https://github.com/altair-viz/altair/issues/2044
   - https://www.sfu.ca/~mjbrydon/tutorials/BAinPy/08_correlation.html
   - https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot
   - https://github.com/altair-viz/altair/issues/2044
   - https://thispointer.com/count-number-of-zeros-in-pandas-dataframe-column/
   - https://altair-viz.github.io/gallery/multiline_tooltip.html
   - https://altair-viz.github.io/altair-tutorial/notebooks/06-Selections.html
   - https://stackoverflow.com/questions/14190045/how-do-i-convert-datetime-timedelta-to-minutes-hours-in-python
3. Articles
   - https://www.worldbank.org/en/topic/forced-displacement
   - https://www.unhcr.org/refugee-statistics/
   - https://data.worldbank.org/indicator?tab=featured

4. Special thanks to GIMP and OBS for enabling the creation of web assets for this publication. Both tools offer powerful video capture and photo editing capabilities, and they are free and open-source!