Kafka events with avro timestamp logical type

Open Q & A Data connectors
1

I have a topic with a timestamp avro logical type.

{
      "name": "updated_date",
      "type": {
        "logicalType": "timestamp-millis",
        "type": "long"
      }
    }

I sent some events with today’s date but the values which i got in my collection are in 1970-01-20

event in the topic

{"id":"17613938-f4b3-42df-b2c1-c5dc8d58d6b0","updated_date":1655133282170,"name":"123","favorite_number":{"int":6},"favorite_color":{"string":"111"}}

Date in rocketset :

_id,_event_time,_meta,favorite_color,favorite_number,id,name,updated_date,
"17613938-f4b3-42df-b2c1-c5dc8d58d6b0","2022-06-13T15:19:34.855000Z","{""kafka"":{""offset"":632,""topic"":""test"",""partition"":0}}","111",7,"17613938-f4b3-42df-b2c1-c5dc8d58d6b0","123","1970-01-20T03:45:33.573745Z",

are the avro logical types supported in rockset? Am I doing something wrong?

Thank you for you help

1 Like
2

Hi @mourad.d !
Can you try doing a SQL transform to coerce the LONG to a timestamp with TIMESTAMP_MILLIS? We are going to investigate to see if it’s supported on our side by default.

3

Hi @mourad.d , I guess you are using some avro converter when setting up your kafka connect as specified here: Kafka ?

I guess the transformation of that timestamp from long to string might be done in the avro converter instead by of rockset. But I might be wrong.

If you could provide more information on how you setup the kafka connect, that would be helpful for us to debug.

4

my connector config

name = RocksetSinkConnectorConnector_0
connector.class = rockset.RocksetSinkConnector
tasks.max = 10
key.converter = org.apache.kafka.connect.storage.StringConverter
value.converter = io.confluent.connect.avro.AvroConverter
topics = test
rockset.apiserver.url = https://api.euc1a1.rockset.com
rockset.integration.key = kafka://mykey@api.euc1a1.rockset.com
rockset.task.threads = 5
format = AVRO
value.converter.schema.registry.url = http://schema-registry:8081
1 Like
5 
SELECT
    TYPEOF(users.updated_date)
FROM
    commons.users
LIMIT
    1

shows that updated_date type is timestamp

6

oh nvm. I misunderstood the question. Please follow @nadine 's suggestion.

7

Hi @nadine,

When I add the transformation, no event is received

SELECT id as _id, TIMESTAMP_MILLIS(updated_date) as updated_date, * FROM _input
8

Investigating- one moment.

9

@mourad.d - can you confirm the topic is getting data?

10

@nadine
yes - created a second collection without the transformation and I can see data coming

1 Like
11

thanks- we’re looking into it

12

@mourad.d - do we have your permission to create a support key for your account? We’re going to investigate. Your explicit permission allows us to investigate your account.

13

Hi @nadine
yes, you have the permission

1 Like
14

Thanks @mourad.d - we are looking into it.

15

@mourad.d I’m going to follow up Monday with our engineering team. I’ll work on response for you early next week.

16

HI @mourad.d !

We have investigated, and it’s an issue on our side. We are working on a fix.

The workaround:
You can use the following sql transformation. This will break when we roll out the actual fix, but if you want to use it now, I can update this thread when the fix happens.

TIMESTAMP_MICROS(UNIX_MICROS(updated_date) * 1000)

Thanks for posting! Let me know if this resolves your issue!

nadine