Knowledgebase

Articles

Aggregate and Summary Tables Precomputing for speed and cost.WHAT THEY ARETables holding pre-aggregated results at a coarser... Aggregation and Grouping Summarising data.WHAT GROUPING DOESCollapses rows sharing values into one row per... Analytical Versus Transactional Databases Two different purposes.WHAT TRANSACTIONAL SYSTEMS OPTIMISE FORMany small reads and writes Strong... Analytics Engineering: Everything That Matters, Briefly The summary.THE WORK IS ENCODING BUSINESS DEFINITIONSGetting them wrong is invisible until... Backfilling and Reprocessing Correcting history.WHEN IT IS NEEDEDA bug is found A source is corrected A definition changes New... Batch and Streaming Approaches When data is processed.WHAT BATCH PROCESSING ISProcessing accumulated data at intervals.WHAT... Building Data Products Treating data as something delivered.WHAT A DATA PRODUCT ISA dataset or service with an owner,... Building Reliable Dashboards Reports people can trust.WHAT MAKES A DASHBOARD FAILNumbers that disagree with other sources... Building Reliable Ingestion Making extraction dependable.WHAT RELIABILITY REQUIRESIdempotence: running twice produces the... Building a Data Platform Incrementally Growing capability sensibly.WHAT TO BUILD FIRSTOne pipeline, from one source, producing one... Building a Semantic Layer Definitions in one place.WHAT IT ISA layer holding metric and dimension definitions, used by... Change Data Capture Reading changes from the source.WHAT IT ISCapturing inserts, updates and deletes by reading the... Choosing a Data Warehouse Selecting the analytical store.WHAT THE MAIN CATEGORIES ARECloud warehouses, billed by usage... Combining Batch and Streaming Architectures using both.WHAT THE TENSION ISStreaming provides freshness; batch provides... Common Table Expressions and Query Structure Writing readable SQL.WHAT A COMMON TABLE EXPRESSION ISA named subquery defined before the main... Cost Management in Data Platforms Controlling spend.WHAT DRIVES COSTCompute for queries and processing Storage Data transfer... Data Contracts and Producer Agreements Agreeing what data will be.WHAT A DATA CONTRACT ISAn explicit agreement between a producer and... Data Engineering Skills and Tools What the work requires.WHAT IS ESSENTIALSQL, thoroughly One programming language, usually Python... Data Engineering for AI Applications Pipelines feeding language models.WHAT RETRIEVAL-BASED APPLICATIONS REQUIREA corpus of documents... Data Governance in Practice Controlling data as an organisational asset.WHAT IT COVERSWhat data exists Who owns it Who may... Data Ingestion Patterns How data arrives.WHAT THE PATTERNS AREFull extraction: reading everything each timeIncremental... Data Lakes and Lakehouse Formats Storage-based platforms.WHAT A DATA LAKE ISObject storage holding files, in any format, at low... Data Modelling Fundamentals Representing a business in tables.WHAT A DATA MODEL ISA representation of entities, their... Data Modelling: Everything That Matters, Briefly The summary.DECLARING THE GRAIN IS THE MOST IMPORTANT MODELLING DECISIONWhat one row represents.... Data Quality Dimensions What quality actually means.WHAT THE DIMENSIONS AREAccuracy: values reflect realityCompleteness:... Data for Machine Learning: Everything That Matters, Briefly The summary.LEAKAGE IS THE MOST DAMAGING ERROR AVAILABLEThe model tests excellently and fails... Databases and Storage: Everything That Matters, Briefly The summary.ROW STORAGE SUITS TRANSACTIONS; COLUMN STORAGE SUITS ANALYSISWhich is why a read... Defining Metrics Consistently One definition, everywhere.WHAT THE PROBLEM ISThe same metric calculated differently in different... Delivery Guarantees What a system promises about each record.WHAT AT-MOST-ONCE MEANSEach record processed zero or one... Designing Event Schemas Structuring what flows.WHAT EVERY EVENT SHOULD CARRYA unique identifier The event type When it... Designing Pipelines for Failure Assuming things break.WHAT WILL FAILSources unavailable Credentials expiring Schemas changing... Designing a Data Warehouse Structuring the whole.WHAT LAYERS TO BUILDRaw: data as received, unmodifiedStaging or cleaned:... Designing for Business Intelligence Tools Modelling for how tools query.WHAT REPORTING TOOLS DOGenerate queries from user interactions with... Dimensional Modelling Facts and dimensions.WHAT A FACT TABLE HOLDSMeasurements of events: sales, payments, page views,... Distributed Processing Frameworks Processing across machines.WHAT THEY PROVIDESplitting work across machines Handling of machine... Enabling Self-Service Analytics Letting others answer their own questions.WHAT IT REQUIRESData modelled so it is understandable... Extracting From Application Interfaces Pulling data from services.WHAT TO ESTABLISH FIRSTRate limits Pagination approach Whether... Feature Engineering Pipelines Building model inputs.WHAT A FEATURE ISA value derived from data, used as model input.WHAT COMMON... Feature Stores Managing features centrally.WHAT ONE ISA system storing feature definitions and values, serving... File Formats for Data How data is physically stored.WHAT TEXT FORMATS PROVIDEHuman readability Universal supportWHAT... Foundations: Everything That Matters, Briefly The summary.A PIPELINE THAT RUNS SUCCESSFULLY AND PRODUCES WRONG NUMBERS IS WORSE THAN ONE THAT... Geospatial Data Location in data pipelines.WHAT THE TYPES AREPoints: a single locationLines: routes and... Graph and Relationship Data Modelling connections.WHAT IT REPRESENTSEntities and the relationships between them.WHERE IT... Handling Change in Dimensions When descriptive data changes.WHAT THE PROBLEM ISA customer moves city. Do past sales belong to... Handling Currency and Exchange Rates Money across currencies.WHY IT MATTERS HERERevenue and costs frequently span local and foreign... Handling Files and Delimited Data The most common and most troublesome source.WHAT GOES WRONG WITH DELIMITED FILESSeparators... Handling Personal Data in Pipelines Privacy in data engineering.WHAT TO ESTABLISH FIRSTWhat personal data you hold, and why.WHAT... Handling Schema Changes When sources change shape.WHAT CHANGES OCCURColumns added Columns removed Columns renamed Types... Implementing Data Quality Testing Catching problems automatically.WHAT TO TEST AT INGESTIONSchema matches expectation Row counts... Incremental Processing Processing only what changed.WHY IT MATTERSReprocessing everything becomes infeasible as volume... Ingestion and Integration: Everything That Matters, Briefly The summary.DELETES BREAK INCREMENTAL EXTRACTIONA deleted row simply stops appearing. Use change... Integrating Data From Several Sources Combining systems that disagree.WHAT THE PROBLEMS AREThe same entity identified differently in... Joins in Practice Combining tables correctly.WHAT THE TYPES DOInner: rows matching in bothLeft: every row from the... Managing Pipeline Code Engineering practice for data.WHAT TO VERSION CONTROLTransformation code Pipeline definitions... Modelling Customer Data A single view of a customer.WHAT IT MEANSOne record per customer, combining information from... Modelling Product and Web Events Behavioural data.WHAT AN EVENT REPRESENTSAn action taken, at a time, by someone.WHAT EVERY EVENT... Modelling Slowly Changing Facts and Events Handling corrections and late data.WHAT LATE-ARRIVING DATA ISRecords for a period that arrive... Modelling Subscription and Billing Data Recurring revenue.WHAT THE ENTITIES ARECustomers Subscriptions Plans and prices Invoices Payments... Monitoring Data Pipelines Knowing they worked.WHAT TO MONITORWhether each run completed Duration, against normal Row counts... Naming and Documentation Standards Making a warehouse usable.WHY IT MATTERSA warehouse nobody can navigate is used by nobody,... NoSQL Stores in Data Engineering Non-relational systems.WHAT THE CATEGORIES AREKey-value: fast lookup by identifierDocument:... Operating Streaming Systems Running them reliably.WHAT TO MONITORConsumer lag, per partition Throughput, in and out Error and... Orchestration and Operations: Everything That Matters, Briefly The summary.NEVER WRITE A TASK THAT IMPLICITLY USES THE CURRENT DATEIt cannot be backfilled, and... Partitioning and Data Layout Organising for query efficiency.WHAT PARTITIONING DOESDivides data so queries read only relevant... Preparing Training Data Data for model building.WHAT DISTINGUISHES IT FROM ANALYTICAL DATAIt must reflect what will be... Processing Data With Python Practical transformation code.WHAT THE COMMON LIBRARIES AREA long-established tabular library... Quality, Governance and Practice: Everything That Matters, Briefly The summary.MOST DATA INCIDENTS ORIGINATE UPSTREAM, IN CHANGES NOBODY COMMUNICATEDData contracts... Query Performance and Optimisation Making queries fast.WHAT TO DO FIRSTRead the query plan.WHAT TO LOOK FORFull scans where an index... Reproducibility in Data Science Work Making results repeatable.WHAT MUST BE REPRODUCIBLEThe dataset The transformations The code The... SQL Across Different Engines Dialect differences.WHAT IS STANDARDCore query structure, joins, aggregation, and much else.WHAT... SQL Beyond the Basics What separates competent from capable.WHAT MOST PEOPLE KNOWSelecting, filtering, sorting, and... SQL: Everything That Matters, Briefly The summary.UNDERSTAND THE EXECUTION ORDERJoin, filter, group, filter groups, compute, sort,... Scheduling and Dependencies When things run.WHAT TIME-BASED SCHEDULING ASSUMESThat inputs are ready by the scheduled time.WHY... Serving Data to Consumers Making it usable.WHO THE CONSUMERS AREAnalysts, querying directly Reporting tools Applications... Serving Data to Models Delivering inputs at prediction time.WHAT THE PATTERNS AREBatch prediction: scoring everything on... Storage Cost and Lifecycle Managing what accumulates.WHAT DRIVES COSTVolume stored Access frequency and pattern Data... Stream Processing Concepts Computing over flows.WHAT STATELESS PROCESSING ISEach record processed independently: filtering,... Streaming Platforms and Tools The available systems.WHAT LOG-BASED PLATFORMS PROVIDEDurable, ordered, replayable streams with... Streaming: Everything That Matters, Briefly The summary.ORDERING IS GUARANTEED WITHIN A PARTITION, NOT A TOPICEvents for one entity must... Structuring Transformation Layers Organising models.WHAT LAYERS TO USESources: declared references to raw tablesStaging: one model... Testing Data Transformations Verifying correctness.WHY IT DIFFERS FROM SOFTWARE TESTINGThe code may be correct while the data... The Data Engineering Lifecycle How work proceeds.WHAT THE STAGES AREUnderstanding the requirement Identifying and assessing... The Data Platform Landscape Components and how they fit.WHAT THE LAYERS ARESources: applications, databases, files, external... Transformation Frameworks Managing transformation as code.WHAT THEY PROVIDETransformations defined as select statements,... Transformation: Everything That Matters, Briefly The summary.LOAD RAW, THEN TRANSFORM IN PLACETransformation is far easier to test, version and... Understanding Data Sources Where data comes from.WHAT THE COMMON SOURCES AREApplication databases Event streams from... Understanding Data Vault and Alternative Approaches Other modelling methods.WHAT DATA VAULT ISA modelling approach separating business keys,... Understanding Data at Different Scales Matching approach to volume.WHAT SMALL DATA MEANSFits comfortably in memory on one machine.WHAT... Understanding ETL and ELT Where transformation happens.WHAT ETL ISExtract, transform, then load the transformed result.WHAT... Understanding Event Streaming Data as a continuous flow.WHAT AN EVENT ISA record that something happened, at a point in... Understanding Pipeline Orchestration Coordinating work.WHAT AN ORCHESTRATOR DOESDefines tasks and their dependencies Schedules them... Vector Storage and Retrieval Storing embeddings.WHAT A VECTOR STORE DOESHolds embeddings and finds the most similar to a query... What Analytics Engineering Is The role between engineering and analysis.WHAT IT ISTransforming raw data into modelled, tested,... What Data Engineering Is The discipline and its purpose.WHAT IT ISBuilding and operating the systems that collect, move,... Window Functions Calculations across related rows.WHAT THEY DOCompute a value for each row, using a set of related... Working With Business Teams Getting definitions right.WHAT THE COMMON FAILURE ISBuilding what was asked for rather than what... Working With Dates and Times in SQL The commonest source of subtle errors.WHAT TO ESTABLISH FIRSTWhat time zone data is stored in,... Working With Unstructured Data Text, documents, images and audio.WHAT MAKES IT DIFFERENTNo schema Large individual objects... Working as a Data Engineer The role in practice.WHAT THE WORK ACTUALLY ISBuilding and maintaining pipelines Investigating... Writing Maintainable SQL Queries others can read.WHAT MAKES SQL UNMAINTAINABLEVery long single queries Deeply nested...
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot