. Part of me began this journey as a result of knowledge engineering is among the hottest and highest-paying careers proper now. I’m not going to fake that wasn’t an element.
However there’s extra to it than that.
I’ve been studying knowledge analytics for some time now. SQL, Energy BI, Python (Pandas, NumPy, a bit of Polars), knowledge cleansing, EDA. You identify it, I’ve been within the weeds with it. And I genuinely take pleasure in it. However someplace alongside the best way, I began getting inquisitive about what occurs earlier than the information lands on my desk. How does it transfer? Who builds these pipelines? What does the infrastructure behind all of this truly seem like?
That curiosity planted a seed.
Then AI began making lots of what I do sooner and simpler. Which is nice. However it additionally made me suppose: if AI can deal with the evaluation, what’s my edge? What can I construct and perceive that goes deeper? I work as an IT System Analyst at a startup, and whereas I benefit from the work, I noticed I wasn’t difficult myself the best way I wished to. I used to be prepared for extra.
The ultimate push got here from a video by Information With Baraa, the place he laid out a whole knowledge engineering roadmap. One thing about seeing it structured and damaged down made it really feel actual and doable. So right here I’m.
I’m studying knowledge engineering in public. And this text is the start of that journey.
Additionally, simply leaving a disclaimer that I’m not affiliated with Information with Baraa. I’m simply sharing my private journey. Hope it helps.
Why Information Engineering Particularly
I wish to spend a second right here as a result of I believe this query deserves an actual reply.
Information analytics taught me the way to work with knowledge after it arrives. Clear it, discover it, visualize it, draw insights from it. That skillset is genuinely priceless. However the extra I realized, the extra I saved bumping into the identical wall. The information I used to be working with had already been formed and moved by another person. Somebody had constructed the pipeline that introduced it to me. Somebody had determined the way it was saved, the way it was structured, how typically it refreshed.
I wished to be that particular person.
Information engineering sits upstream from analytics. It’s about constructing the programs that make evaluation attainable within the first place. Information pipelines, storage structure, workflow orchestration, large-scale knowledge processing. These are the foundations every thing else is constructed on. And actually, that form of infrastructure work appeals to me in a approach that pure evaluation now not does.
There’s additionally a sensible argument. Information engineering roles persistently rank among the many highest paying within the knowledge business. As AI instruments get higher at automating the analytical layer, the demand for individuals who can construct and preserve dependable knowledge infrastructure is simply going to develop. I’d slightly be constructing the pipes than simply utilizing them.
And yet another factor. The startup I work at doesn’t use any of the instruments I’m about to be taught. Which implies each hour I put into that is completely self-directed. No group to be taught from, no work initiatives to use it on. Simply me, the web, and no matter I can construct alone. That’s a problem I’m selecting on objective.
Why I’m Doing This in Public
Writing about what I be taught is one thing I already imagine in deeply. It forces you to truly perceive one thing earlier than you clarify it. It retains you accountable. And over time, it builds one thing {that a} resume alone by no means may.
However I’ll be trustworthy about my fears too, as a result of I believe that’s the purpose of doing this publicly.
I’ve shiny object syndrome. There, I stated it. I’ve explored graphic design, animation, writing, advertising, and IT earlier than touchdown in knowledge. There’s all the time one thing new and thrilling pulling my consideration. Information engineering may simply get changed by the following flashy factor in my feed if I’m not intentional about it.
Consistency is one other one. I work a 9-5 the place I barely contact the instruments I’ll be studying. There’s no pure reinforcement at work, no colleague I can bounce Airflow questions off of. I’m constructing this completely alone time, exterior of my job tasks.
And steadiness. Three to 4 hours a day is the purpose. Some days that can really feel simple. Different days it would really feel inconceivable.
Publishing this journey is my accountability system. If I am going quiet, you’ll know I slipped. And I’d slightly not slip.
What I’m Beginning With
I’m not ranging from zero, which helps. I have already got newbie to intermediate SQL data from my knowledge analytics work, primary Python fundamentals, and a few hands-on expertise with Pandas. That offers me a basis to construct on slightly than rebuild from scratch.
Right here’s the complete studying stack, roughly within the order I’ll be tackling it.
1. SQL: Going Deeper Than Analytics
I do know SQL. However analytics SQL and engineering SQL are completely different animals. I’ll be going deeper into question optimization, indexing, working with very massive datasets, and writing SQL that’s constructed for efficiency slightly than simply exploration. For those who’ve solely ever used SQL to tug and filter knowledge, there’s a complete different layer beneath value understanding.
Why it’s first: The whole lot in knowledge engineering ultimately touches SQL. Getting sharp right here earlier than layering in additional complicated instruments makes the remainder of the journey simpler.
2. Python: From Exploratory to Manufacturing-Prepared
I’ve the fundamentals. Pandas, NumPy, some Polars. However the Python I’ve been writing lives largely in notebooks. Exploratory, messy, not constructed to final. The purpose now could be to put in writing cleaner, extra structured, reusable code. Capabilities, modules, error dealing with, scripting. The form of Python you’d truly put in a pipeline.
Why it issues: Python is the glue that holds most fashionable knowledge engineering stacks collectively. Airflow makes use of it. PySpark is constructed on it. Getting comfy right here is non-negotiable.
3. Git and GitHub: Model Management Finished Correctly
I’ll be trustworthy. My Git data is at the moment “copy the command, hope it really works.” That has to vary. Model management is key to working like an engineer slightly than simply an analyst. I’ll be studying branching, pull requests, and the way to handle code correctly throughout initiatives.
Why it issues: Each venture I construct from right here on goes on GitHub. It’s portfolio, it’s self-discipline, and it’s how actual groups work.
4. Apache Spark and PySpark: Huge Information Processing
That is the place issues get genuinely thrilling. Apache Spark is among the most generally used engines for processing large-scale knowledge. PySpark is the Python API for it, which suggests I can use a language I’m already considerably accustomed to to work with distributed knowledge at scale.
The leap from Pandas to Spark is a mindset shift. Pandas works on a single machine. Spark is constructed to run throughout clusters. Studying to suppose in that distributed approach is among the abilities that separates knowledge engineers from analysts.
Why it issues: If you wish to work with huge knowledge in a manufacturing atmosphere, Spark is sort of unavoidable. It exhibits up in job descriptions continuously and is core to the Databricks ecosystem I’ll be constructing towards.
5. Apache Airflow: Orchestrating Information Pipelines
Information pipelines don’t run themselves. You want one thing to schedule them, monitor them, and deal with failures gracefully. That’s the place workflow orchestration instruments are available in, and Airflow is my choose.
I thought-about just a few choices right here. Databricks Workflows is nice if you happen to’re already deep within the Databricks ecosystem. Azure Information Manufacturing unit is sensible for Azure-heavy environments. However Airflow is free, open-source, cloud-agnostic, and extensively used throughout the business. It additionally teaches you the core ideas of orchestration in a approach that transfers to different instruments. Beginning with Airflow felt like the suitable name, particularly since I’m making an attempt to maintain prices low.
Why it issues: Orchestration is what turns a group of scripts into an precise pipeline. Understanding Airflow is knowing how manufacturing knowledge workflows are managed.
6. Databricks: The Information Platform
In some unspecified time in the future it’s worthwhile to choose an information platform and go deep on it. I’m going with Databricks. It’s constructed on prime of Spark, it’s in excessive demand, and it has a free Neighborhood Version that permits you to follow with out paying for cloud credit.
The alternate options are stable too. Snowflake is a clear, quick SQL warehouse that lots of firms love. BigQuery is Google’s absolutely managed, serverless choice and genuinely glorious if you happen to’re leaning towards Google Cloud. However Databricks sits on the intersection of huge knowledge, machine studying, and knowledge engineering in a approach that matches the place I wish to go. It made probably the most sense for my targets.
Why it issues: Employers need you to have platform expertise. Going deep on one is extra priceless than realizing a bit of about all of them.
How I’m Structuring the 12 Months
The trustworthy reply is that this would possibly take longer than 12 months. And I’m okay with that. I’d slightly take 15 months and really perceive what I’m doing than rush by way of in 12 and are available out shaky on the basics.
The overall strategy is to maneuver by way of every talent so as and never advance till I’ve constructed one thing with what I simply realized. Tutorials are high quality for orientation however initiatives are the place actual studying occurs. My plan is to doc every section right here on In direction of Information Science: the ideas, the initiatives, the frustrations, and the wins.
For monitoring progress, I’m utilizing the Notion roadmap from Information With Baraa as my spine. It breaks down every talent into core matters and lets me observe the place I’m with out getting overwhelmed by the complete image suddenly.
As for time dedication, three to 4 hours a day is the goal. A few of that might be structured studying. Some might be constructing. Some might be writing about what I simply realized, which is its personal type of finding out.
What Success Appears Like
Touchdown a high-paying knowledge engineering function is the purpose. That’s actual and I’m not going to decorate it up.
However alongside that, I wish to change into a reputable voice on this house. Somebody who builds issues value speaking about, paperwork the journey with out filtering out the laborious components, and possibly makes the trail a bit of clearer for somebody arising behind me.
The writing and the educational feed one another. The portfolio turns into the proof. The proof builds the model. That’s the imaginative and prescient.
Beginning In the present day
This text is my official begin date. I’m not ready till I really feel prepared or till every thing is completely deliberate. I’m beginning now, writing as I am going, and letting the method be public and a bit of messy.
For those who’re someplace on an analogous path. Whether or not you’re in analytics fascinated by engineering, in IT questioning what’s subsequent, or simply somebody making an attempt to construct abilities that maintain their worth in an AI-accelerated world. Observe alongside.
I believe we’ll have lots to speak about. I’ll even be sharing my learnings on my YouTube channel. So be happy to subscribe under and observe alongside.
That is the primary article in an ongoing collection documenting my knowledge engineering journey. I’ll be publishing often on my progress, the initiatives I’m constructing, and every thing I be taught alongside the best way.
And if you wish to get entry to the Notion template, in case you’re on the identical journey as I’m, you possibly can entry it right here.
Observe alongside on my journey under.
