When using the new kafka integration how does Rockset handle upserts and deletes coming in for a particular topic?
1 Like
Thanks for the question. By default Rockset doesn’t do any upsert for your Kafka collection. If you want to enable upserts, consider defining the _id Special Fields | Rockset Documentation in the input data such that Rockset will use this special field to trigger override.
Also, Rockset doesn’t handle document deletion the same way as Kafka by specifying null record value. So if your record contains a null value, it will not trigger a document deletion as there is no root level path to be upserted.
1 Like
What is the best way to remove a record from a Rockset collection?
1 Like
Hi @jonatmpx -
Are you using a client to remove documents? Here is something in the Python client.
Would something like this be helpful? Let me know if I should point you to a different client or if there’s something different you are wanting to use.
–nadine
Ok, great, that is exactly what I was looking for. I am still evaluating Rockset as a real-time analytics solution and needed to know how to respond to kafka topics that store messages for deleted records. I will create a consumer that will read the specific topic and then call the api directly or use the client to remove the document from the appropriate Rockset collection.
1 Like
@jonatmpx awesome. Please, don’t hesitate if you have any questions along the way. Also, you can set a retention period for your documents as well, if that’s helpful when you create a collection.
In the mean time, I hope you voted for your favorite meme of the week 
Happy Friday
Best,
n
1 Like