However, the problem of manually having to refresh the dashboard with the Refresh Dashboard button still persists, even after trying the solution you just gave. Let’s hope the profs don’t knock too many points off my grade xD
You should be able to automatically refresh, without pressing the button on Grafana.
1 Like
The nature of Grafana is it has to refresh, it doesn’t automatically update. But, you can set the refresh rate.
1 Like
That sounds like a plan. Do you think S3 would have any advantages over Kafka in terms of plotting time-series in Grafana, though?
1 Like
Kafka is a streaming source and S3 is a data lake. Since you’re working with Spark, typically, the data is sent to S3, from there, you connect to Rockset.
I would boil it down to what your architecture is and what you want to accomplish. If you can share your architecture [you can write it up quickly here: https://excalidraw.com/ ] I can answer this question in terms of what you are trying to do.
1 Like
Yup, done. Here it is:
Working on a more detailed response- will share by Monday!
Sure thing, thank you so much for your time!
hey @kaiser -
A little late-
For Rockset, it is best to send as many documents in a single update as much as possible. You can do 1 record at a time as you would with Kafka. It’s more efficient if documents are combined into a group like an S3 file. Since you are creating your results with Spark, they are already naturally in a group. So, S3 is the most efficient option.
Rockset automatically assigns an _event_time attribute at ingestion time. You can use SQL transformation to override _event_time to use a timestamp from your record. From there, you can use SQL to select a timestamp range. Grafana will plot this over time.
1 Like