Back to The Modern Data Engineering Landscape

What Data Engineering Actually Is (2026 View)

Strip away the hype. Data engineering is moving data reliably from where it lives to where it can be used. FIND_VIDEO: search 'what is data engineering modern data stack' — recommended channel: Seattle Data Guy / Kahan Data Solutions. Aim for 10 min or under.

9 minutesVideo LessonPDF notes

Watch What Data Engineering Actually Is (2026 View) on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Ecosystem Capital Explosion — Venture funding in ingestion and storage platforms demonstrates the scale of data tooling fragmentation.
The Three-Pillar Baseline — Early data stack design simplifies architecture into ingestion, storage, and visualization.
Analytics Maturity Curve — Team hiring milestones align with company scale, starting with manual extracts and spreadsheets.
Data Governance Standards — Standardizing column nomenclature and implementing automated tests unblock data engineering velocity.
Raw Access and Reverse ETL — Exploratory raw data access is compared against the cyclical dependency dangers of reverse ETL.
Observability and Roadmap — Late-stage observability needs are introduced alongside the planned deep-dive curriculum.

PDF notes

Bottom line: You will evaluate modern data stack components and structure an ingestion architecture that matches organizational maturity. You will learn when to introduce managed tools versus lightweight manual processes and how to prevent architectural debt like circular pipeline dependencies.

This lesson on What Data Engineering Actually Is (2026 View) is hands-on and example-driven. You will evaluate modern data stack components and structure an ingestion architecture that matches organizational maturity. You will learn when to introduce managed tools versus lightweight manual processes and how to prevent architectural debt like circular pipeline dependencies.

What You'll Be Able To Do

Evaluate organizational maturity stage to determine data engineering hiring and tooling needs
Construct the foundational three-pillar data architecture for early-stage infrastructure
Implement schema naming conventions for timestamps and booleans across data models
Identify data lineage risks to prevent circular dependencies in reverse ETL workflows
Differentiate raw exploratory data access patterns from core production data models

Detailed Concept Walkthrough

1. Analytics Maturity and Architectural Progression

Data infrastructure must scale strictly in proportion to organizational transaction volume, data source count, and staffing levels. Early premature tooling adds maintenance overhead without producing return on investment.

Mechanism: At early maturity (1-4 people), data needs are served by manual developer extracts and spreadsheets. Building an automated warehouse at this stage incurs negative ROI due to maintenance demands.
Execution Flow: Once transaction volume reaches thousands daily across 6-10 distinct sources, hire a dedicated data engineer to deploy the initial automated warehouse stack.
Best Practice: Keep early stacks strictly lightweight; manual spreadsheet tracking of lineage is preferable to paying maintenance and configuration costs on complex governance platforms.

-- Stage 1: Minimal operational extract
-- Direct export from transactional replica to business sheet
SELECT 
  user_id,
  email,
  created_at
FROM production_db.users
WHERE created_at >= CURRENT_DATE - INTERVAL '1 day';

Key Takeaway: Adopt infrastructure tools only when operational complexity and transaction scale justify the maintenance overhead.

2. The Core Three-Pillar Modern Data Stack

The leanest viable automated data infrastructure comprises three primary functions: Ingestion, Storage, and Visualization.

Ingestion Layer: Ingestion tools (e.g., Fivetran, Airbyte) extract and load raw data from source APIs and operational databases directly into destination storage without upfront transformation.
Storage Layer: Central data warehouses or lakes (e.g., Snowflake, BigQuery, Starburst) store raw staging areas and transformed analytical tables.
Consumption Layer: Visualization tools connect directly to transformed warehouse models to expose metrics and business intelligence reports to stakeholders.

# Minimal automated ingestion definition pattern using EL abstraction
class IngestionPipeline:
    def __init__(self, source_connector, destination_warehouse):
        self.source = source_connector
        self.dest = destination_warehouse

    def run_sync(self):
        raw_data = self.source.extract_incremental()
        self.dest.load_raw_stage(raw_data)

Key Takeaway: Build the core ingestion-storage-visualization loop first before introducing secondary tooling.

3. Data Modeling Standards and Schema Governance

Standardized field nomenclature and testing reduce manual pipeline verification and make discovery frictionless across teams.

Syntax Rule: Enforce standard suffixes for time types (such as _ts for timestamps) and standard prefixes for boolean flags (such as is_ or has_).
Data Segregation: Separate core governed production models from experimental sandbox environments and raw datasets to prevent research code from corrupting downstream reporting.
Best Practice: Automate quality assertions using dbt or pipeline tests so engineers spend time developing new pipelines rather than manually checking pipeline accuracy.

-- Staging model applying column naming standards
SELECT
  id AS user_id,
  status = 'ACTIVE' AS is_active,
  created_at AS created_ts,
  updated_at AS updated_ts
FROM raw_data.raw_users;

Key Takeaway: Rigid naming conventions and automated testing eliminate manual data verification bottlenecks.

4. Lineage and the Reverse ETL Spaghetti Risk

Reverse ETL syncs warehouse models back to operational systems, but without robust data lineage, it creates self-referencing circular dependencies.

Mechanism: A reverse ETL tool extracts transformed metrics from the warehouse and writes them into operational CRMs (e.g., Salesforce).
Under the Hood: If an ingestion pipeline then extracts that same CRM table back into the raw warehouse staging layer, a cyclical data dependency loop ('spaghetti data') is established.
Best Practice: Delay reverse ETL implementation until end-to-end data lineage and observability tools are active to map dependencies and break feedback loops.

# Danger: Potential circular pipeline dependency
# 1. Warehouse calculates user metric
# 2. Reverse ETL syncs to CRM
# 3. CRM ingest pipeline pulls CRM data back to Warehouse raw

def check_pipeline_cycle(graph, source_node, target_node):
    # Ensure destination write does not feed source extractor
    return target_node in graph.get_upstream_nodes(source_node)

Key Takeaway: Never deploy reverse ETL pipelines without explicit lineage mapping to avoid cyclical data corruption.

Topics Covered in What Data Engineering Actually Is (2026 View)

Ecosystem Capital Explosion (0:00 - 0:45) — Venture funding in ingestion and storage platforms demonstrates the scale of data tooling fragmentation.
The Three-Pillar Baseline (0:45 - 1:35) — Early data stack design simplifies architecture into ingestion, storage, and visualization.
Analytics Maturity Curve (1:35 - 2:40) — Team hiring milestones align with company scale, starting with manual extracts and spreadsheets.
Data Governance Standards (2:40 - 3:50) — Standardizing column nomenclature and implementing automated tests unblock data engineering velocity.
Raw Access and Reverse ETL (3:50 - 4:40) — Exploratory raw data access is compared against the cyclical dependency dangers of reverse ETL.
Observability and Roadmap (4:40 - 5:38) — Late-stage observability needs are introduced alongside the planned deep-dive curriculum.

Data Engineering Cheat Sheet

Timestamp Naming Convention — Standardizes timestamp column identifiers across all models
```
ALTER TABLE staging_orders RENAME COLUMN created_at TO created_ts;
```
Boolean Naming Convention — Standardizes boolean flag column identifiers across models
```
SELECT id, status = 'ACTIVE' AS is_active FROM raw_users;
```
EL Pattern — Extracts from source and loads directly to destination raw storage
```
airbyte-ci sync --source-id src_pg --destination-id dst_sf
```
Direct Raw Notebook Querying — Allows exploratory analysis without waiting for downstream ETL models
```
df = spark.read.json('s3://lake-bucket/raw/events/*.json')
```
Automated Quality Assertion — Validates table schemas and integrity rules automatically
```
dbt test --select staging_orders
```

Comparison Table

Stage	Team Composition	Stack Components
Startup (Early)	Dev + Business User	Manual Extracts, Excel spreadsheets
Growth (Intermediate)	1 Data Engineer, 1 Analyst	Ingestion, Data Warehouse, BI Tool
Mature (Enterprise)	Full Data Team + Scientists	Warehouse, dbt, Lineage, Observability, Catalog

Common Pitfalls

Mistake: Purchasing complex observability and catalog platforms before building basic pipelines. Avoid: Use spreadsheet tracking until multiple production data sources create manual maintenance bottlenecks.
Mistake: Deploying reverse ETL pipelines without automated lineage tracking. Avoid: Map all upstream and downstream dependencies to eliminate circular data feedback loops into transactional CRMs.
Mistake: Allowing data engineers to manually audit data quality across daily runs. Avoid: Implement automated data assertion tests in transformation pipelines to unblock engineering throughput.

FAQs

Why should a company wait to hire a data engineer? Early companies lack the data volume and multiple source systems needed to justify the ROI of building and maintaining a dedicated warehouse.
What is the primary operational risk of reverse ETL? Without lineage tracking, metrics pushed into operational tools like Salesforce can be re-ingested into the warehouse, creating circular dependency bugs.
Why do data scientists query raw data directly instead of waiting for warehouse models? Data scientists often need to conduct research or validate value in newly introduced datasets before data engineers have time to construct production pipelines.

Common pitfalls

Mistake: Purchasing complex observability and catalog platforms before building basic pipelines. Avoid: Use spreadsheet tracking until multiple production data sources create manual maintenance bottlenecks.
Mistake: Deploying reverse ETL pipelines without automated lineage tracking. Avoid: Map all upstream and downstream dependencies to eliminate circular data feedback loops into transactional CRMs.
Mistake: Allowing data engineers to manually audit data quality across daily runs. Avoid: Implement automated data assertion tests in transformation pipelines to unblock engineering throughput.

if you were sleeping under a rock last year you might have missed the fact that billions and billions of dollars have been thrown into data infrastructure solutions don't believe me look at the fact that dbt has raised 150 million dollars 5 tran is upwards of 565 million dollars airbite has also raised 150 million dollars and starburst 100 million dollars and that's just a few examples if we look at this chart from blockandfiles.com we will notice that again billions have been put into the space of data storage alone now i will point out that for some reason five train is on here which doesn't make sense but if we were to just look at data ingestion tools like five tran we would also see that billions have been thrown into that space as well so these are just two spaces in the whole data infrastructure pipeline ingestion and storage and truthfully speaking there's at least a dozen more categories that we could probably fill out all that have had hundreds of millions if not billions thrown into them i mean just look at matt turk's data and ai landscape that he's put together it makes me feel like madra staring down the shinobi alliance i mean look at all these different sections here there's enough for every one of the different villages for goodness sakes and there are just tons and tons of tools that vps and heads of data can pick from so it really has become a massive challenge to figure out what exactly belongs in your data stack in this video we're going to lay the foundation to help people understand how to develop their data infrastructure or as some people like to call it their modern data stack now i have put together a diagram that kind of outlines your baseline data stack but even this is a little bit intense and generally speaking when i start working with clients what i usually recommend is they focus on three key areas those three key areas are ingestion storage and then data visualization now luckily i posted about this on linkedin and danny from y labs had a great response here just bringing up the fact that these are great from a technical aspect but from a human standpoint there are other things that are nice to have even early on when you're developing your data stack and i will work to comment on that later in this video but the way that i view data stacks i think is very similar to the way that taylor brownlow of count put together this quick and concise chart that helps people understand kind of who they should be hiring and when in terms of a data team as their analytics maturity grows and as you can see in this image when you first start out you pretty much just generally have a dev and an analyst maybe your company is only four people and you know you've got your dev maybe an analyst a few other people that work for you and then you as a ceo so most likely what ends up happening is the dev ends up extracting data manually sends it over to someone from the business side maybe you don't even have an analyst and then that person will end up doing some things in excel here in this data stack you really have nothing right like if you were to look at what your current data stack looks like it's excel you probably haven't put together a warehouse and it would be very expensive to do so and it's probably not worth it from an roi perspective or in terms of your priorities when you have such a small team to kind of try to automate a lot of this and develop things like data warehouse that take at least one person to kind of manage and develop but as your company grows you know you've got 10 employees 12 employees you're starting to handle more and more transactions right like you're starting to see hundreds and thousands of transactions in a day you maybe have six seven ten data sources that now need to kind of all be reported on now is when you can start thinking about hiring someone like a data engineer who can start actually developing a data warehouse you're more likely going to get that roi at this point but again at this point more than likely what your data stack's going to look like is ingestion storage and then some visualization or reporting again this is very lightweight because more than likely if you only have one data engineer and one analyst who are really kind of juggling a lot of these tasks having things like lineage and observability might not be a great investment from a tool perspective and you might want to instead focus on just doing that in like excel just at the very least you kind of understand your general flow but aren't investing a ton of your valuable time and resources in another tool because every tool inevitably creates technical debt maintenance requirements and just something else to upkeep but as your company grows you will inevitably start to get to the baseline kind of modern data stack that i've outlined here in this image because you're going to start needing things like lineage and observability and making it so that data is not just pulled into the data warehouse but it's easy to find i think that's one of the key things in terms of like creating data you need to be able to create a way and a set of processes that don't just pull that data in but one help make it easy for people to understand what data is valid and what data is like test and research data that's just coming from data scientists versus like your more core data sets as well as trying to make it easy for analysts to figure out how all your data connects where it's all coming from and doing things like creating standards in terms of how you're actually marking things like data types you know time stamps should have some sort of consistent nomenclature usually like underscore ts same thing with like booleans oftentimes people pick here something like is underscore active is a pretty common one you know just trying to say if something is active it's usually something like that where you have standards so one you've got all this tooling to help people find the data you also have improved data quality because no longer can a data engineer sit there and manually you know try to figure out if data is accurate or not they're more likely going to need to spend more of their time building new data pipelines as well as data tests anything of that nature so you can automate it that way again that they can move quickly and they're not forced to constantly have to stop do quick manual tests and really slow down their whole process now you're gonna see there's a ton of different components here you've got somewhere where you're likely storing your raw files staging data you're doing something maybe with dbt you've probably got things that involve notebooks and having those on the cloud one thing that's not covered here is more than likely those notebooks might just be attached directly to the raw data section rather than waiting for data engineers to get all of the data sometimes you know data scientists will just do research directly on raw data which can make sense if they believe that there's value in that data but the data engineers have not had time to pull it yet but overall this is a baseline data stack again this is kind of what you will need as you are maturing and growing you'll notice i don't even cover everything here for example i kind of left out reverse etls because more than likely if you don't have data lineage what's going to end up happening if you do implement something like a reverse etl is you're going to have spaghetti data or like just data that basically depends on itself where you end up creating some sort of metric or something and then sending it back to salesforce or somewhere and then pulling that value back into your data warehouse i've seen it happen even at large companies it happens all the time because again data just becomes very hard to track and what feeds what and where's what going gets very lost without having some sort of ability to track data lineage now the reason that i put together this diagram is actually more than just this video what is going to happen here now is i'm going to make a series of videos where we're going to go over each one of these sections individually so for example where we talk about batch data and extraction we're going to probably talk about el i usually compose together because they naturally kind of have an affinity together so we'll talk about what exactly is el kind of what is its purpose what tools can be used here for the el section you know what are kind of the different options that people use from maybe the most custom code option to maybe more of the tooling side where things are just solutions will also go into things like streaming different types of data warehousing dbt and just all of these different sections along the way because here's the thing the modern data stack or whatever you want to call it is very complicated there's a lot of different components and tons of different choices dozens of tools alone are just in data observability alone and that's very meta and very personally down the path of analytical maturity like you've got to be pretty mature as a company to have implemented that kind of a tool and find the value from it you know in a perfect world you would implement it almost immediately as you start developing all of your data pipelines but in reality i often find that this is something that comes down the line once people finally realize that their pipelines are becoming a mess and they need something in terms of a data observability tool to help kind of track it all from a lineage perspective as well as something that helps you search what data actually exists in all of your data warehouses or data lakes or whatever systems that you have put together so this will be our map for this discussion and i'm sure i will add maybe different portions or sections along the way maybe because i missed some things or maybe because as we're going i would like to discuss other components along the way if you feel like something is missing please feel free to write a comment below other than that thank you so much for watching this video i look forward to kind of explaining all of these different sections and components in the next few weeks honestly let's be real probably months because it's going to take me a while to build the series but i am really excited for it and i hope you guys are too have a great day and goodbye

Frequently asked questions

Why should a company wait to hire a data engineer?

Early companies lack the data volume and multiple source systems needed to justify the ROI of building and maintaining a dedicated warehouse.

What is the primary operational risk of reverse ETL?

Without lineage tracking, metrics pushed into operational tools like Salesforce can be re-ingested into the warehouse, creating circular dependency bugs.

Why do data scientists query raw data directly instead of waiting for warehouse models?

Data scientists often need to conduct research or validate value in newly introduced datasets before data engineers have time to construct production pipelines.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.