, I made a decision I needed to transition from information analyst to information engineer.
Like many individuals beginning out, I used to be overwhelmed by the sheer variety of issues I believed I wanted to be taught. Information warehouses, orchestration instruments, distributed processing, streaming methods, cloud platforms, infrastructure. The checklist appeared infinite.
As an alternative of attempting to be taught all the pieces without delay, I took a special strategy.
I created a 12-month self-study roadmap constructed round one easy thought: be taught by constructing.
Quite than leaping from one tutorial to a different, I’d construct a collection of small tasks. Every venture would introduce a number of new ideas whereas reinforcing what I had realized earlier than. The aim wasn’t to construct essentially the most subtle purposes doable. It was to develop the behavior of pondering like an engineer by fixing one drawback at a time.
The primary venture in that journey was a GitHub ETL pipeline. It started as a easy Python script that fetched repository information and exported it to a CSV file. As I realized extra, I steadily improved it. I changed CSV recordsdata with SQLite, made the pipeline idempotent to forestall duplicate information, and finally automated it with GitHub Actions.
By the point I completed, I spotted one thing that hadn’t been apparent after I began.
Constructing the ETL logic was really the straightforward half.
The more durable questions appeared as soon as I ended fascinated with a script that runs as soon as and began fascinated with a system that should run again and again with out me watching it.
- How ought to or not it’s scheduled?
- What occurs if it fails midway by means of?
- The place ought to retry logic dwell?
- How do you bundle the appliance so it runs the identical approach in every single place?
These questions taught me much more about information engineering than parsing JSON or writing SQL ever did.
That first venture left me with one other problem.
GitHub Actions labored nicely for automating a small ETL pipeline, however I needed to grasp what adjustments whenever you use a workflow orchestrator constructed particularly for information engineering. I needed to learn the way engineers separate orchestration from execution, how containerized workloads match into that image, and what a production-minded pipeline really appears like, even on a small scale.
So for my second venture, I made a decision to construct an automatic RSS ingestion pipeline.
On the floor, it’s a reasonably easy utility. It fetches articles from an RSS feed, parses them into structured objects, and shops them in PostgreSQL.
However the aim was by no means to construct an RSS reader.
The aim was to discover the engineering choices that remodel a Python script right into a dependable information pipeline.
On this article, I’ll stroll by means of these choices, the errors I made alongside the way in which, and the teachings I realized constructing my first pipeline with Kestra.
Why Construct One other ETL Pipeline?
After ending my first ETL venture, I thought of transferring on to one thing fully completely different.
Perhaps a knowledge warehouse venture. Perhaps Apache Spark. Perhaps an API with a extra advanced transformation layer.
As an alternative, I constructed… one other ETL pipeline.
At first, that in all probability appears like a step backwards.
In any case, I’d already constructed an extraction pipeline, made it idempotent, and scheduled it with GitHub Actions. Why repeat the identical train?
As a result of I wasn’t attempting to be taught a brand new dataset. I used to be attempting to be taught a brand new mind-set.
One lesson from my first venture caught with me.
Writing the ETL logic wasn’t the tough half. The tough half was all the pieces surrounding it.
- How ought to the pipeline be executed?
- How ought to it recuperate from failures?
- How do you bundle it so it runs persistently on any machine?
- The place does scheduling belong?
And maybe the largest query of all:
The place ought to the obligations of the appliance finish, and the place ought to the obligations of the orchestration layer start?
These questions don’t have a lot to do with RSS feeds or GitHub repositories. They’re engineering questions, and I spotted I may discover them with nearly any information supply.
That’s why I selected an RSS feed.
Not as a result of RSS is especially thrilling, however as a result of it’s deliberately easy.
The extraction logic solely takes a number of strains of Python. That meant I may spend much less time worrying about enterprise logic and extra time fascinated with structure.
For a similar purpose, I made a decision to maneuver away from GitHub Actions for this venture.
GitHub Actions was an excellent introduction to scheduling. It confirmed me automate a workflow and gave me my first style of operating an ETL pipeline with out guide intervention.
However GitHub Actions isn’t designed particularly for orchestrating information workflows.
I needed to grasp what adjustments whenever you use a device that’s constructed with information pipelines in thoughts.
That’s what led me to Kestra.
Quite than asking, “How do I run this Python script each hour?”, I discovered myself asking a special set of questions.
- How ought to retries be configured?
- How are workflow executions tracked?
- How ought to atmosphere variables be handed right into a container?
- What does a failed execution seem like?
- How do you separate the appliance from the infrastructure that runs it?
These questions had been precisely what I needed to discover.
By selecting a easy ETL pipeline, I may deal with the engineering choices as an alternative of getting distracted by sophisticated enterprise logic.
Wanting again, I believe that was the correct determination.
This venture isn’t attention-grabbing as a result of it processes RSS feeds.
It’s attention-grabbing as a result of constructing it pressured me to consider reliability, repeatability, and orchestration in a approach my first venture by no means did.
The First Architectural Choice: Docker Earlier than Kestra
With the venture outlined, my first intuition was to leap straight into Kestra.
In any case, orchestration was one of many fundamental causes I selected this venture. Why not begin there?
As an alternative, I did one thing that turned out to avoid wasting me a number of frustration later.
I ignored Kestra fully.
Which may sound counterintuitive, however I needed to keep away from introducing a number of transferring elements earlier than I knew the core utility really labored.
So I constructed the venture in layers.
First, I wrote the Python ETL.
Its job was deliberately small. Fetch the RSS feed, parse every entry into an Article object, and save the outcomes to PostgreSQL. Nothing extra.
feed = feedparser.parse(RSS_URL)
articles = parse_feed(feed)
save_articles(articles)That’s actually all of the ETL did. I deliberately stored the appliance small as a result of the main focus of this venture wasn’t the transformation logic. It was all the pieces that occurred round it.
As soon as that labored reliably, I turned my consideration to the database.
I needed repeated executions to be secure, so I made the inserts idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING. That approach, operating the pipeline a number of instances wouldn’t create duplicate rows.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;This one line made the pipeline secure to execute repeatedly. Whether or not Kestra ran the workflow as soon as or 100 instances, PostgreSQL grew to become chargeable for stopping duplicate data.
Solely after the ETL and database labored collectively did I introduce Docker.
That call modified how I believed in regards to the venture.
Initially, Docker felt like one other device I wanted to be taught.
By the top of the venture, I spotted it had turn into one thing way more essential.
It grew to become the unit of execution.
FROM python:3.13-slim
WORKDIR /app
COPY necessities.txt .
RUN pip set up --no-cache-dir -r necessities.txt
COPY . .
CMD ["python", "fetch_rss.py"]Packaging the ETL this manner meant the appliance grew to become self-contained. As an alternative of asking Kestra to grasp my Python venture, I may merely ask it to run a container that already knew execute the pipeline.
As an alternative of pondering, “Kestra will run my Python script,” I began pondering, “Kestra will run my Docker picture.”
That distinction might sound delicate, however it fully adjustments the connection between your utility and your orchestration layer.
As soon as the ETL was packaged right into a container, it not mattered whether or not it ran on my laptop computer, inside Kestra, or on one other machine solely.
The runtime atmosphere was at all times the identical.
That consistency gave me one thing I didn’t have earlier than: confidence.
Earlier than introducing Kestra, I may run the container manually and confirm that it fetched the RSS feed, linked to PostgreSQL, and persevered the anticipated data.
If one thing failed, I knew the issue wasn’t hidden behind one other layer of orchestration.
That validation step turned out to be extremely priceless later.
Throughout improvement, I bumped into points with networking, container configuration, and workflow execution. As a result of the Docker picture had already been validated independently, I may instantly rule out the ETL itself and deal with the orchestration layer.
That made debugging dramatically simpler.
Wanting again, I believe this was one of many greatest classes from the venture.
It’s tempting to attach each element collectively as shortly as doable and hope all the pieces works.
A greater strategy is to validate every layer earlier than introducing the following one.
On this venture, the order seemed like this:
- Validate the Python ETL.
- Validate PostgreSQL persistence.
- Validate the Docker picture.
- Lastly, let Kestra orchestrate a container that I already trusted.
Every layer constructed on the earlier one.
By the point Kestra entered the image, I wasn’t attempting to debug Python, PostgreSQL, Docker, and orchestration on the identical time.
I used to be solely fixing one drawback.
That incremental strategy made all the venture really feel way more manageable, and it’s a workflow I’ll in all probability proceed utilizing on future information engineering tasks.
The Assumption That Turned Out to Be Mistaken
With a working Docker picture, I used to be satisfied the onerous half was over.
- I had a Python ETL that labored.
- I had PostgreSQL operating in Docker.
- I had verified that the container may fetch RSS articles and save them to the database.
Now all Kestra needed to do was run it.
Or so I believed.
My authentic assumption was easy.
Kestra would level to my Python recordsdata, execute the script, and all the pieces would work precisely because it had from the command line.
It didn’t.
I shortly found that there was an essential distinction I hadn’t totally appreciated.
Kestra is an orchestrator.
It isn’t chargeable for constructing Python environments or managing utility dependencies. Its duty is deciding when and the way workloads ought to run.
That realization modified the course of the venture.
As an alternative of treating Kestra as one other place to execute Python code, I began treating it because the layer chargeable for orchestrating a workload that already existed.
That workload was my Docker picture.
As soon as I made that psychological shift, the structure grew to become a lot cleaner.
- The ETL grew to become a self-contained utility.
- Docker grew to become the deployment artifact.
- Kestra grew to become the orchestrator.
Every layer had a transparent duty, and none of them wanted to understand how the others labored internally.
Curiously, reaching that time wasn’t fully easy.
My first intuition was to discover a approach for Kestra to execute the Python venture straight. That strategy sounded easier, however the extra I experimented with it, the extra I spotted I used to be asking the orchestrator to take duty for one thing the appliance ought to already present.
As soon as I embraced Docker because the execution artifact, the workflow grew to become a lot easier.
Some approaches seemed promising at first however launched pointless complexity. Others labored, however didn’t align with how Kestra encourages containerized workloads to be executed.
Ultimately, I landed on a workflow that felt surprisingly easy.
As an alternative of attempting to show Kestra run Python, I let Kestra do what it does greatest.
It launches a container.
duties:
- id: run_etl
sort: io.kestra.plugin.scripts.shell.Instructions
containerImage: rss-pipeline-etl:newest
taskRunner:
sort: io.kestra.plugin.scripts.runner.docker.Docker
instructions:
- python /app/fetch_rss.pyWanting on the workflow now, what stands out isn’t how a lot YAML it incorporates. It’s how little Kestra really must learn about my utility. Its solely duty is to launch a container that already is aware of execute the ETL.
Inside that container, my utility already is aware of precisely what to do.
That small architectural change solved extra than simply the rapid execution drawback.
It additionally bolstered an concept that’s changing into a recurring theme in my studying journey.
Good engineering usually isn’t about including one other layer.
It’s about giving every layer a single duty and permitting it to try this job nicely.
- Python shouldn’t fear about orchestration.
- Kestra shouldn’t fear about Python dependencies.
- Docker shouldn’t know something about RSS feeds.
Every element solves a special drawback.
As soon as I ended asking one device to unravel each drawback, all the system grew to become a lot simpler to purpose about.
Wanting again, this was in all probability the largest mindset shift in the entire venture.
I didn’t simply discover ways to use Kestra.
I realized what orchestration really means.
As soon as a Pipeline Runs Robotically, The whole lot Adjustments
Up till this level, I had been operating the pipeline manually.
If one thing failed, I used to be sitting in entrance of my laptop. I may learn the error, make a change, and check out once more.
That security internet disappears the second a pipeline begins operating by itself.
One of many first issues I configured in Kestra was a easy hourly schedule.
triggers:
- id: hourly_schedule
sort: io.kestra.plugin.core.set off.Schedule
cron: "0 * * * *"On paper, it was only a cron expression.
In observe, it represented a a lot greater shift.
The pipeline not trusted me remembering to run it.
Each hour, Kestra would begin a brand new execution, launch the ETL container, and course of the newest articles from the RSS feed.
That instantly raised one other query.
What occurs if a kind of executions fails?
Throughout improvement, I intentionally launched failures to reply that query. I pointed the pipeline at an invalid database host and watched what occurred.
The primary execution failed, precisely as anticipated.
Extra importantly, it didn’t cease there.
As a result of the workflow was configured with retries, Kestra mechanically tried the execution once more after a brief delay.
retry:
sort: fixed
maxAttempts: 3
interval: PT30SThat was certainly one of my favourite moments within the venture.
As soon as I corrected the configuration, the workflow accomplished efficiently with out requiring any adjustments to the appliance itself.
That was certainly one of my favourite moments within the venture.
Not as a result of retries are significantly sophisticated, however as a result of they highlighted one other separation of obligations.
The ETL shouldn’t determine whether or not it deserves one other probability.
That’s an orchestration concern.
By transferring retry logic into Kestra, the Python utility remained targeted on a single duty: course of the feed and persist the outcomes.
The orchestration layer dealt with resilience.
Scheduling launched one other problem that I’d already encountered in my first ETL venture.
Repeated executions imply repeated makes an attempt to jot down information.
If the identical RSS article seems in a number of hourly runs, the pipeline shouldn’t insert it twice.
Thankfully, I had already solved an identical drawback earlier than.
The database layer was designed to be idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;That meant each execution may safely try to insert the identical data with out creating duplicates.
The mix of scheduling, retries, and idempotent writes made the pipeline way more forgiving.
If a run failed, Kestra may retry it.
If a profitable retry encountered information that had already been written, PostgreSQL would merely ignore the duplicates.
Neither layer wanted to know what the opposite was doing.
They every dealt with their very own duty.
The final enchancment was visibility.
Early in improvement, my logs seemed precisely such as you’d anticipate from a venture that was nonetheless being debugged.
I printed whole RSS objects to the console simply to verify the parser was working.
It wasn’t fairly, however it served its goal.
Because the pipeline grew to become extra steady, these debug statements grew to become much less helpful.
I changed them with logs that described the execution as an alternative of dumping uncooked information.
Earlier than
print(feed.entries[0])After
print("=== RSS PIPELINE START ===")
first = feed.entries[0]
print("First entry preview:")
print(f"Title: {first.get('title')}")
print(f"Hyperlink: {first.get('hyperlink')}")
print(f"Feed title: {feed.feed.title}")
print(f"Fetched articles: {len(articles)}")
print(f"Saved {len(articles)} articles to the database.")
print("=== RSS PIPELINE END ===")Every run now tells a easy story.
=== RSS PIPELINE START ===
First entry preview:
Title: Christian Ledermann: Migrate From mypy To ty And pyrefly
Hyperlink: https://dev.to/...
Feed title: Planet Python
Fetched articles: 25
Saved 25 articles to the database.
=== RSS PIPELINE END ===These adjustments didn’t make the appliance smarter. They made it simpler to grasp. And that’s an essential distinction.
Good observability isn’t about producing extra logs. It’s about producing the correct logs.
By the top of the venture, I spotted one thing attention-grabbing. The Python code hadn’t grown dramatically. Many of the work had occurred round it.
Wanting again, a lot of the engineering effort wasn’t spent making the ETL smarter. It was spent making it extra dependable.
- Scheduling ensured it ran with out me.
- Retries helped it recuperate from transient failures.
- Idempotency protected the database from duplicate writes.
- Logging made each execution simpler to grasp.
Individually, none of those adjustments had been significantly advanced. Collectively, they reworked a easy Python script into one thing that behaved way more like a manufacturing system.
The Ultimate Structure
By the top of the venture, the structure had settled into one thing that felt surprisingly easy.
Wanting on the last structure, it’s tempting to assume the venture was at all times heading on this course.
It wasn’t.
Every layer was added solely after the earlier one had been validated.
- The ETL got here first.
- Then PostgreSQL.
- Then Docker.
- Lastly, Kestra.
That order mattered.
As a result of each element had already been examined independently, I by no means discovered myself debugging Python, Docker, PostgreSQL, and Kestra on the identical time. Every determination lowered the variety of unknowns as an alternative of accelerating them.
Extra importantly, each element ended up with a transparent duty.
- Python is aware of fetch, parse, and persist RSS articles.
- PostgreSQL is aware of retailer information safely and stop duplicates.
- Docker supplies a constant execution atmosphere.
- Kestra decides when the workload ought to run and what ought to occur if it fails.
None of these elements try to do one another’s jobs.
Paradoxically, that’s what made the completed system really feel a lot easier than I anticipated.
What This Mission Modified In regards to the Approach I Assume
After I began studying information engineering, I assumed the tough half could be writing ETL code.
That’s what most newbie tutorials deal with.
- You discover ways to name an API.
- You remodel the info.
- You reserve it someplace.
Repeat.
These are priceless abilities, however they’re just one a part of the image.
This venture taught me that the engineering begins after the script works.
As soon as a pipeline is predicted to run each hour, survive transient failures, keep away from duplicate information, and produce logs that specify what occurred, the questions turn into way more attention-grabbing.
You’re not fascinated with particular person features. You’re fascinated with methods.
One of many greatest mindset shifts for me was understanding the distinction between execution and orchestration.
At first, these concepts felt nearly interchangeable. Now they really feel fully separate.
The Python utility ought to deal with enterprise logic. The orchestrator ought to deal with when, the place, and the way that utility runs.
Protecting these obligations separate made all the venture simpler to purpose about.
It additionally modified how I take into consideration Docker.
Earlier than this venture, I noticed Docker primarily as a option to bundle purposes.
Now I see it as a deployment artifact.
As soon as the ETL had been packaged right into a container and validated independently, I may cease worrying about whether or not it could behave otherwise inside Kestra.
That confidence turned out to be one of many greatest benefits of containerizing the appliance.
Maybe crucial lesson, although, had nothing to do with Kestra or Docker. It was the worth of constructing incrementally.
Each main determination adopted the identical sample.
- Construct the smallest factor that works.
- Validate it.
- Solely then introduce the following layer.
That strategy made the venture really feel a lot much less overwhelming than attempting to attach all the pieces collectively from the start.
Wanting again, I believe that’s a lesson I’ll carry into each future venture, whatever the expertise.
Wanting Forward
This RSS pipeline is just the second venture in my information engineering studying journey. In comparison with my first ETL pipeline, the code itself isn’t dramatically extra advanced. What modified was the way in which I approached the issue.
As an alternative of asking, “How do I write this script?”, I discovered myself asking questions like:
- The place ought to this duty dwell?
- What occurs if it fails?
- Can I run it repeatedly with out worrying about duplicate information
- Can I belief it to run after I’m not watching?
These questions pushed me to assume much less like somebody writing Python code and extra like somebody designing a system. And I believe that’s the true worth of constructing tasks.
Each venture teaches a brand new device. However the most effective tasks slowly change the way in which you assume. This one actually did.
I’m certain the following venture will problem a very completely different set of assumptions. Actually, I’m trying ahead to discovering out what they’re.
That is a part of my ongoing collection documenting my transition from methods analyst to information engineer. In the event you’ve been following alongside, thanks.
