Batch vs streaming at a glance
| Aspect | Batch | Streaming |
|---|---|---|
| Typical sources | Cloud storage, databases, CRM extracts | Web, mobile, server events, HTTP API |
| Latency | Scheduled | Near real time |
| Best for | History, profile attributes, large volumes | Behavioural events, triggers for journeys |
Step-by-step ingestion framework
- Confirm the target schema and dataset.
- Choose the source connector or API and configure authentication securely.
- Map source fields to XDM with Data Prep; add calculated fields where needed.
- Run a sample load and review errors and partial-ingestion thresholds.
- Schedule the dataflow and set up monitoring and alerts.
- Reconcile record counts with the source system before go-live.
Best practices
- Send only the fields your use cases need.
- Keep identity values clean and consistently formatted (for example lowercase, trimmed, hashed where required).
- Use streaming only for data that needs real-time action.
- Document every dataflow owner, schedule and dependency.
Who this guide is for
This guide is written for beginners, students and career switchers who want a clear mental model, and for developers, data engineers, implementation specialists, consultants, architects, marketers, agencies and enterprise teams who need practical, step-by-step detail they can apply on real projects.
Frequently asked questions
What is Data Prep in AEP?
Data Prep maps, transforms and validates source fields into XDM fields during ingestion, using mapping rules and calculated fields.
Is Web SDK a form of streaming ingestion?
Yes. Web SDK sends events to the Adobe Edge Network, which forwards them through a datastream into AEP datasets in near real time.
Last updated: