QUESTION
The goal is to create a snapshot of the file from our incremental data. The filename of the parquet will be different each time we generate the snapshot. Is there any way to only read the latest parquet in a collection?
ANSWER
One way to achieve this is to drop all the existing records and only ingest the new parquet file. You can achieve this by creating a new collection that is pointed to the new parquet file and then dropping the old Rockset collection. Please reference: https://rockset.com/docs/amazon-s3/#create-a-collection
You can also do this in a way that your queries and applications do not break by using aliases. You can switch the alias to the newly created collection on the fly. Please reference: https://rockset.com/docs/views/#aliases
Have more questions on this topic? Please respond to the thread!