Connecting Rockset to Grafana

I’ve been trying to connect to Grafana from Rockset, and even after modifying the grafana.ini file to

[plugins]
allow_loading_unsigned_plugins = "rockset-backend-datasource"

The Grafana server doesn’t acknowledge it. Instead I see this error:

This is the doc from where I’m following the instructions.

Could I get some help on this?

1 Like

Hi! yes-

It sounds like you were able to send the data from Spark to S3?

@bryfi walks through the set up here-- scroll towards the end of the video at 55 min in-

Also, take a look at this:
https://community.rockset.com/t/rockset-and-grafana/543

1 Like

Thanks, @nadine that helped. I’ve got a new issue here, though. It look like the _event_time field isn’t being updated as the sentiment scores come in, which makes me unable to plot a time-series graph on Graphana.

Any idea if Rockset can generate a timestamp for when a Kafka record is received and store that as a field in the collection? As in, could I add a Time column beside the Sentiment column here using the SQL rollup query?

Hi @kaiser -

YES :slight_smile: ! You can do SQL transformations and remap _event_time Rockset has to the event_time in Kafka. Can you send you share your Kafka fields and data type?

Check out these resources:

Does this help?

So I solved this in a different way: I set the query lambda to also sent the current time as a JSON field to Grafana, and voila!

1 Like

yay! so it sounds like your project is done @kaiser :slight_smile: ! Can you share what you built via here:
Showcase - Rockset Community

1 Like

Well, almost. I have what I think is the last problem to solve, but that’s on Grafana’s end. I’ll put it here for completeness’ sake:
The time field is being sent to Grafana’s end and being received just fine, but the graph is not rendering. I’ll put it up as and when I figure that out. :smiley:

1 Like

I may be able to help with Grafana- are you writing time-series data? Can you share a query you wrote that you want to visualize?

Check out what I did here at the AWS workshop with grafana:

I’m actually using the JSON API plugin mentioned in the previous YouTube video you linked. My problem is that each time I hit refresh, the graph updates, but only plots a point, while I’d like continuous, connected lines from point to point with the REST JSON API being called every 5 seconds to get a new sentiment value.

This is the graph before and after hitting the Refresh Dashboard button on the top right.

Before

After

The query lambda is:
SELECT SENTIMENT, UNIX_SECONDS(CURRENT_TIMESTAMP()) as time FROM “sent-analysis”.tweet_sentiments t

1 Like

Let me work on a loom video for you on Grafana. Give me some time to put it together- I’ll share the video soon.

1 Like

@kaiser can you share the results of the query on the rockset console on what you see for the timestamp?

1 Like

Sure, here it is:

Here’s how the JSON looks from Rockset’s side:
It looks like this in JSON:

[
  {
    "SENTIMENT": 0.3358208955223881,
    "time": 1655446419
  }
]

And this is how the received JSON looks on Grafana’s end:

ahh @kaiser → you’re only returning 1 row. That’s why it’s only plotting one.

Do you have a query where multiple rows are returning with different timestamps?

Is there a way to have Grafana call the API at set intervals to join the dots?

Let me ask some more clarifying q’s: is Rockset getting updated with new data from S3? or is the data in s3 fixed- as in no new data is getting added?

separately, on grafana, to refresh, you can click this drop down to i.e. every 5 seconds or so:

I actually came up with a pretty bootleg solution just a couple of minutes ago, and it’s working. The author of the JSON API plugin recommended that if I want to visualize how the data changes with time, I’d be better off using a database. But since that isn’t an option due to the Rockset Backend Source plugin not working (I ended up using Grafana cloud instead), what I did is create another collection to store the averaged sentiment values, created a Lambda that adds the current record from the original sentiment score collection to this new collection, and called it using a local Python script.

I also tweaked the Lambda that Grafana uses to return the 20 most recent rows from the new collection I made. And it works now, kind of.


But this, of course, isn’t ideal. But since my course project review is tomorrow, this will have to do for now :stuck_out_tongue: I’d like to find a more elegant solution eventually, though.

And to answer your question, I couldn’t really figure out S3 on my local machine, but I did end up using a Kafka topic as the source for Rockset.

1 Like

Kafka works great with Rockset- but I’d love to figure out the s3 part with you, when you have time. By returning more rows on the query, you’re now seeing it be plotted on the sentiment. You can also modify the timestamp to return the rows within the last day or so- I can help you fine tune your query to do that too.

1 Like