Overview
Data filtering is a process in RWS Integration that allows you to exclude specific datapoints from your integration based on conditional criteria. You can configure filters in the Extract phase to remove unwanted records before they proceed to transformation and loading. Filtering occurs after data enrichment but before transformation. This ensures that only relevant data proceeds through the rest of the integration workflow, reducing processing time and ensuring data quality.Common filtering scenarios
Remove records based on field values
Filter out records where specific fields don’t meet your criteria. Example:- Remove products with price 0
Remove records based on age or date criteria
Exclude records based on date calculations or age requirements. Example:- Remove minors (users under 18 years old)
Remove records based on enrichment data
Use data from enrichment sources to determine which records to exclude. Example:- Remove products that are out of stock (using enrichment data)
Remove records from specific date periods
Exclude records that fall within certain time ranges. Example:- Remove orders from a specific date range