I Thought Loading Information Was the End Line. It Was the Beginning Level.

, I gave myself a 12-month roadmap to go from information analyst to information engineer. I’m solely about two months into it. In that brief stretch I’ve already constructed two ETL pipelines from scratch, the primary one pulling GitHub repo information into SQLite, the second pulling RSS articles into PostgreSQL with Docker and Kestra dealing with the orchestration. I wrote about scheduling that second pipeline to run robotically each hour, and on the time, that felt like an actual milestone. The information was flowing in by itself, no handbook runs, no me remembering to set off something.

However someplace between writing that article and beginning this one, I ran a question by myself information and realized one thing. I couldn’t kind my articles by date correctly. I couldn’t inform which blogs had been publishing probably the most. The information had been sitting in Postgres for weeks, technically “loaded,” and I hadn’t really checked out it carefully till I wanted it for one thing.

Seems I’d constructed two pipelines and skipped the half that makes the information helpful. Extract, load, after which nothing. No transformation, no modeling, no actual construction previous “it’s in a desk now.”

This text is about fixing that. I lastly sat down and realized dbt, and within the course of realized what “evaluation prepared” really means, as a result of it seems loading information and having usable information are two very various things.

The Information Was Loaded. It Simply Wasn’t Usable.

Right here’s what my articles desk really regarded like as soon as I finished and paid consideration to it.

The schema itself was easy, actually about so simple as a desk can get:

CREATE TABLE IF NOT EXISTS articles (
    id TEXT PRIMARY KEY,
    title TEXT NOT NULL,
    hyperlink TEXT NOT NULL,
    abstract TEXT,
    revealed TEXT
);

Discover that final column. revealed is a TEXT area. Not a timestamp, not a date, only a plain string that occurred to seem like a date in the event you squinted at it.

Once I queried the ten most up-to-date articles, that is what got here again:

title                                                    | revealed
----------------------------------------------------------+---------------------------------
Django Weblog: Final Name 2026 Django Developer Survey      | Wed, 08 Jul 2026 19:31:21 +0000
Mike Driscoll: New E book Launch: Python Typing              | Wed, 08 Jul 2026 18:46:18 +0000

That appears wonderful at a look. It’s readable. However attempt to really do something with it. Need the articles from the final 7 days? You’ll be able to’t filter on that with out casting it first, each single time, in each single question. Need to kind chronologically and belief the order? Textual content sorting and date sorting aren’t the identical factor, and relying on the format, they will quietly disagree with one another.

Then there was the second drawback, the one I virtually missed completely as a result of it was hiding in plain sight. Have a look at these titles once more:

Django Weblog: Final Name 2026 Django Developer Survey
Mike Driscoll: New E book Launch: Python Typing

Each single title on this feed follows the identical sample. Writer or weblog identify, a colon, then the precise headline. That’s actual, structured data sitting inside a single textual content area, fully unusable as a filter or a group-by. I couldn’t reply a query so simple as “which blogs put up probably the most on Planet Python” as a result of that data wasn’t a column. It was simply textual content, buried.

In order that was the precise state of issues. Two pipelines constructed, information flowing in on schedule, and I nonetheless couldn’t reply primary questions on my very own information. Loading it was by no means the end line. I simply hadn’t gotten to the start line but.

Why dbt, Particularly

My first intuition was to simply repair this in Python, since that’s the device I already belief. Write a script that reads from articles, parses the dates, splits the titles, writes the outcomes into new columns or a brand new desk. And that might have labored, technically.

However the extra I considered it, the extra that felt like patching the identical gap I’d already dug twice. Each of my pipelines had been extract and cargo, full cease, and if I bolted transformation logic onto a Python script once more, I’d simply be including a 3rd untested, undocumented step to a system that already had two. I wouldn’t be studying something new. I’d simply be writing extra of the identical factor I already knew find out how to write.

dbt does this otherwise, and that distinction is type of the entire level of the device. As an alternative of a script that runs as soon as and produces some output it’s important to belief blindly, dbt fashions are SQL that will get model managed, examined, and documented as a part of the identical workflow. You write a metamorphosis, and in the identical mission you may assert issues about it: this column ought to by no means be null, this ID ought to all the time be distinctive. If these assumptions break, you discover out instantly, not three weeks later when a chart appears improper and you haven’t any concept why.

It additionally matches how the business really works. Each information engineering job put up I’ve checked out over the previous two months mentions dbt, or one thing dbt-shaped. Studying it wasn’t nearly fixing my RSS information, it was about studying the device that’s change into the default method groups deal with the “T” in ETL.

So as an alternative of one other Python script, I made a decision to truly sit down and be taught dbt correctly, on information I already had, with issues I already understood. Right here’s how that went.

Setting Up (and Instantly Hitting a Wall)

Getting dbt put in ought to have been the boring half. It wasn’t.

I attempted pip set up dbt-postgres and received a wall of dependency decision errors, dbt-core had no matching distribution for my surroundings. Seems I used to be working Python 3.14, which is new sufficient that dbt hadn’t caught as much as it but. dbt Core formally helps as much as 3.13 proper now, and there’s normally a lag earlier than it helps no matter Python simply launched.

The repair wasn’t sophisticated as soon as I understood the precise drawback, set up an older, supported Python model alongside my present one, and construct a digital surroundings particularly for dbt utilizing that:

py -3.12 -m venv dbt-env
dbt-envScriptsactivate
pip set up dbt-postgres

That’s a small factor, but it surely’s the type of small factor that eats an hour in the event you don’t know to search for it, and I believe it’s value together with right here as a result of it’s precisely the type of setup friction that doesn’t present up in tutorials. Tutorials assume your surroundings already works. Mine didn’t, and I’d guess I’m not the one one working a Python model that’s forward of what dbt at present helps.

As soon as that was sorted, connecting dbt to my present Postgres database (already working domestically in Docker from my RSS pipeline) was simple. dbt init, decide postgres, plug within the host, port, credentials, and database identify, and dbt debug confirms the connection:

All checks handed!

With that, I had a dbt mission sitting on high of the identical Postgres occasion my RSS pipeline had been writing to for weeks. Time to truly repair the information.

Constructing the Staging Mannequin

The primary actual dbt idea I needed to perceive was the distinction between a supply and a mannequin. My uncooked articles desk isn’t one thing dbt constructed, it’s exterior information that already exists, so dbt calls it a supply. I outlined that in a sources.yml file, which is admittedly simply dbt’s method of formally acknowledging “this desk exists, and I rely upon it”:

sources:
  - identify: rss_pipeline
    schema: public
    tables:
      - identify: articles

From there, I constructed my very first mannequin, stg_articles, a staging mannequin whose complete job is to wash up the uncooked information with out doing something fancy but. That is the place each of my authentic issues received mounted in the identical file.

For the date:

to_timestamp(revealed, 'Dy, DD Mon YYYY HH24:MI:SS OF') as published_at

For the buried writer identify:

split_part(title, ':', 1) as writer,
trim(substring(title from place(':' in title) + 1)) as article_title

I ran dbt run, then went and truly queried the consequence as an alternative of assuming it labored:

published_raw                     published_at
Solar, 05 Jul 2026 16:29:47 +0000   2026-07-05 16:29:47+00

Actual timestamps. And once I checked the writer break up:

writer                      | article_title
Python Software program Basis  | Python Packaging Council Inaugural Election Dates

Clear separation, even on titles with a couple of colon in them, like a PyCoder’s Weekly subject title that had a colon in each the supply identify and the headline itself. The break up logic solely breaks on the primary colon, so it held up wonderful.

Including Assessments

That is the half that made the entire mission really feel much less like “I wrote some SQL” and extra like precise engineering. I added assessments immediately in a schema file subsequent to the mannequin:

columns:
  - identify: article_id
    assessments:
      - distinctive
      - not_null
  - identify: published_at
    assessments:
      - not_null

Working dbt check doesn’t simply test that the SQL runs, it checks that my assumptions in regards to the information really maintain:

PASS=4 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=4

That not_null check on published_at particularly is the one that might have caught it if my date format string had been improper. As an alternative of silently producing nulls I won’t discover for weeks, I’d have seen a failed check the second I ran it.

Constructing a Mart, and Lastly Asking a Actual Query

With clear staging information in place, I constructed another mannequin on high of it, articles_by_author, which aggregates the information into one thing I might really ask a query of: which blogs put up probably the most, and the way just lately.

choose
    writer,
    rely(*) as total_articles,
    max(published_at) as most_recent_article,
    min(published_at) as earliest_article
from {{ ref('stg_articles') }}
group by writer
order by total_articles desc

That ref() perform as an alternative of supply() issues right here, it’s how dbt is aware of this mannequin will depend on stg_articles, not on the uncooked desk immediately. That dependency monitoring is what builds the lineage graph later.

The consequence was the primary genuinely new factor I might see on this information since I began amassing it two months in the past:

writer                       total_articles   most_recent_article
Python Software program Basis   5                2026-07-09 14:11:06+00
Django Weblog                4                2026-07-08 19:31:21+00

A query I couldn’t reply every week earlier, answered in a single question, on information I’d already had sitting round the entire time.

Seeing the Complete Factor

The final step was working dbt docs generate and dbt docs serve, which builds an interactive documentation website with a lineage graph, principally a visible map of how information flows by way of the mission. Mine confirmed precisely three related nodes:

The place This Leaves Me

In order that’s the mission. Two clear fashions, seven passing assessments, and a lineage graph that truly exhibits an actual chain from uncooked information to one thing I can ask questions of. In comparison with the place I began this piece, unable to kind by date or inform which blogs posted probably the most, that’s an actual shift, even when the underlying dataset didn’t change in any respect. Identical information. Very totally different usefulness.

I need to be sincere about what this isn’t, although. That is nonetheless working completely by myself machine. The Postgres database, the dbt mission, all of it lives domestically in Docker, which implies none of this exists wherever the second my laptop computer is off. There’s additionally just one RSS feed feeding into this proper now, so “which blogs put up probably the most” is a reasonably small query with a reasonably small dataset behind it. And I haven’t touched something round alerting or monitoring if a check begins failing quietly within the background.

None of that takes away from what I really realized right here, although. I believe there’s a distinction between a mission being completed and a mission having taught you what it was supposed to show you. This one did the second factor. I perceive the distinction between a supply and a mannequin now. I perceive why assessments aren’t non-compulsory in the event you really need to belief your individual information. And I perceive, in a really concrete method this time, why “the information is loaded” and “the information is usable” are two fully totally different claims.

The following drawback is apparent, actually. Every part I constructed right here nonetheless will depend on my laptop computer being on and Docker working. That’s the following wall I’m going to hit, and doubtless the following factor I write about, taking this complete stack off my machine and placing it someplace it could actually really run with out me.

Two months right into a twelve month roadmap, and I believe that’s about proper. Slower than I’d like some days, however each wall I’ve hit to this point has taught me one thing I couldn’t have realized by studying about it first.

Thanks for studying!

That is a part of my ongoing sequence documenting my transition from techniques analyst to information engineer. Should you’ve been following alongside, thanks.

Join with me on LinkedIn, YouTube, and Twitter.