How often Rockset performs LIST and GET operations on S3

Question: How often does Rockset perform LIST or GET operations on S3 sources? For example, are the LIST or GET calls triggered only when there is an update in the bucket or subb-dir ?

Answer: There is a periodic scan that monitors the timestamp on the bucket objects and from that determines whether the object needs to be re-ingested.

We are sensitive to the fact that these ‘add up’ in your costs. There is a back-off strategy, so we check less frequently when we don’t find new documents.

1 Like