Document Level Filtering

Open Q & A
1

Is there a way to use a field mapping to drop a row of data on ingest if a field does not equal some value? or even better some combination of fields/values?

1 Like
2

Hi @jordanfm,

We can filter. In the example below, we don’t have to insert a document if the field is not = to the Filter:

field_mappings:
  - name: filter
    input_fields:
      - field_name: 'field'
        if_missing: 'SKIP'
        is_drop: false
        param: 'country_code'
    output_field:
      field_name: 'field'
      value:
        sql: "if(:field = 'FILTER', True, cast('raise exception' as int))"
      on_error: 'FAIL'

Please let me know if the helps,
n