Knowledgebase

Feature Engineering Pipelines Print

  • dataengineering, data, guide, howto, solution, zillionkinghost, hosting, support
  • 0

Building model inputs.

WHAT A FEATURE IS

A value derived from data, used as model input.

WHAT COMMON TRANSFORMATIONS ARE

Aggregations over windows Ratios and differences Encoding of categories Handling of missing values Scaling

WHAT THE HARD REQUIREMENT IS

The same computation at training and at prediction.

WHAT HAPPENS OTHERWISE

Training and serving skew: the model receives different values than it learned from.

WHY THAT IS COMMON

Training features are computed in SQL over history; serving features are computed in application code.

WHAT PREVENTS IT

One definition, used by both.

WHAT TO AVOID

Reimplementing feature logic for serving.

WHAT TO BE CAREFUL WITH

Missing value handling differing between paths Category values unseen at training Scaling parameters computed from the whole dataset

WHAT TO DO ABOUT UNSEEN CATEGORIES

Define explicit handling, rather than failing.

WHAT TO VERSION

Feature definitions, since changing one changes the model's behaviour.

WHAT TO MONITOR IN PRODUCTION

Feature distributions, against training.

WHAT DIVERGENCE INDICATES

Drift, or a broken pipeline.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot