Thanks, @nadine that helped. I’ve got a new issue here, though. It look like the _event_time field isn’t being updated as the sentiment scores come in, which makes me unable to plot a time-series graph on Graphana.
Any idea if Rockset can generate a timestamp for when a Kafka record is received and store that as a field in the collection? As in, could I add a Time column beside the Sentiment column here using the SQL rollup query?
YES ! You can do SQL transformations and remap _event_time Rockset has to the event_time in Kafka. Can you send you share your Kafka fields and data type?
Well, almost. I have what I think is the last problem to solve, but that’s on Grafana’s end. I’ll put it here for completeness’ sake:
The time field is being sent to Grafana’s end and being received just fine, but the graph is not rendering. I’ll put it up as and when I figure that out.
I’m actually using the JSON API plugin mentioned in the previous YouTube video you linked. My problem is that each time I hit refresh, the graph updates, but only plots a point, while I’d like continuous, connected lines from point to point with the REST JSON API being called every 5 seconds to get a new sentiment value.
This is the graph before and after hitting the Refresh Dashboard button on the top right.
I actually came up with a pretty bootleg solution just a couple of minutes ago, and it’s working. The author of the JSON API plugin recommended that if I want to visualize how the data changes with time, I’d be better off using a database. But since that isn’t an option due to the Rockset Backend Source plugin not working (I ended up using Grafana cloud instead), what I did is create another collection to store the averaged sentiment values, created a Lambda that adds the current record from the original sentiment score collection to this new collection, and called it using a local Python script.
I also tweaked the Lambda that Grafana uses to return the 20 most recent rows from the new collection I made. And it works now, kind of.
But this, of course, isn’t ideal. But since my course project review is tomorrow, this will have to do for now I’d like to find a more elegant solution eventually, though.
And to answer your question, I couldn’t really figure out S3 on my local machine, but I did end up using a Kafka topic as the source for Rockset.
Kafka works great with Rockset- but I’d love to figure out the s3 part with you, when you have time. By returning more rows on the query, you’re now seeing it be plotted on the sentiment. You can also modify the timestamp to return the rows within the last day or so- I can help you fine tune your query to do that too.