Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom SchemaReader/SchemaWriter are not working for Avro schemas #8430

Closed
geomagilles opened this issue Nov 2, 2020 · 7 comments · Fixed by #8464
Closed

Custom SchemaReader/SchemaWriter are not working for Avro schemas #8430

geomagilles opened this issue Nov 2, 2020 · 7 comments · Fixed by #8464
Labels
help wanted type/bug The PR fixed a bug or issue reported a bug

Comments

@geomagilles
Copy link
Contributor

geomagilles commented Nov 2, 2020 •
edited

Describe the bug
When using Schema.AVRO(mySchemaDefinition) custom SchemaWriter/SchemaReader provided in mySchemaDefinition are not used even if present in mySchemaDefinition.

The use case is to be able to serialize/deserialize Kotlin data classes using avro4k package by providing custom Avro reader/writer.

Expected behavior
This could work

    val schemaDefinition = SchemaDefinition.builder<TaskEngineEnvelope>()
        .withJsonDef(Avro.default.schema(MyMessage.serializer()).toString())
        .withSchemaReader(MyMessageReader())
        .withSchemaWriter(MyMessageWriter())
        .withSupportSchemaVersioning(true)
        .build()

    val client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build()

    val consumer = client.newConsumer(Schema.AVRO(schemaDefinition))...

Additional context
Apparently a similar issue was solved for Json Schema (#6528 (comment))
@geomagilles geomagilles added the type/bug The PR fixed a bug or issue reported a bug label Nov 2, 2020
@sijie sijie mentioned this issue Nov 2, 2020
Closed
@codelipenghui
Copy link
Contributor

@congbobo184 Could you please take a look at this issue?

@congbobo184 congbobo184 mentioned this issue Nov 6, 2020
Merged
@congbobo184
Copy link
Contributor

@geomagilles #8464 this PR add the support of avro schema user define reader and writer and add a interface to handle the multi version schema. you can define the reader and writer by yourself.

@geomagilles
Copy link
Contributor Author

geomagilles commented Nov 8, 2020 •
edited

@congbobo184 Thx! I'm not able to run the PR myself as I do not manage yet to run Pulsar from sources - but it seems good to me.

Note: I'm not sure what is the purpose of the getSchemaReaderBySchemaInfo interface - can you clarify that for me, please?

@congbobo184
Copy link
Contributor

@geomagilles hello, if use AvroSchema will have multi version schema, so we should generate different version reader to decode the data. this PR will change, add the MultiVersionReader to solve the question.

@geomagilles
Copy link
Contributor Author

@congbobo184 thx - do you think it could be included in the next release?

@congbobo184
Copy link
Contributor

@geomagilles 2.6.2 have been voted, so it will move to 2.7.0, 2.7.0 will release immediately after 2.6.2 finish.

@geomagilles
Copy link
Contributor Author

@geomagilles 2.6.2 have been voted, so it will move to 2.7.0, 2.7.0 will release immediately after 2.6.2 finish.

ok, thx

@geomagilles geomagilles mentioned this issue Dec 4, 2020
Closed
@sijie sijie mentioned this issue Dec 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Schema] Multi version schema reader implementation.
3 participants
@geomagilles @codelipenghui @congbobo184