public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Modules: Add patched goavro dependency for extensions (#21027)

Browse Source
This commit is contained in:
Emil Tullstedt
2019-12-11 14:44:29 +01:00
committed by GitHub
parent 32c9d558f9
commit fe8b533497
50 changed files with 8198 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vendor/github.com/linkedin/goavro/v2/.gitignore
Generated
+1
View File
@@ -0,0 +1 @@
*.test
vendor/github.com/linkedin/goavro/v2/AUTHORS
Generated Vendored
+38
View File
@@ -0,0 +1,38 @@
Goavro was originally created during the Fall of 2014 at LinkedIn,
Corp., in New York City, New York, USA.
The following persons, listed in alphabetical order, have participated
with goavro development by contributing code and test cases.
Alan Gardner <alanctgardner@gmail.com>
Billy Hand <bhand@mediamath.com>
Christian Blades <christian.blades@careerbuilder.com>
Corey Scott <corey.scott@gmail.com>
Darshan Shaligram <scintilla@gmail.com>
Dylan Wen <hhkbp2@gmail.com>
Enrico Candino <enrico.candino@gmail.com>
Fellyn Silliman <fsilliman@linkedin.com>
James Crasta <jcrasta@underarmour.com>
Jeff Haynie <jhaynie@gmail.com>
Joe Roth <joseph_roth@cable.comcast.com>
Karrick S. McDermott <kmcdermott@linkedin.com>
Kasey Klipsch <kklipsch@mediamath.com>
Michael Johnson <mijohnson@linkedin.com>
Murray Resinski <murray.resinski@octanner.com>
Nicolas Kaiser <nikai@nikai.net>
Sebastien Launay <sebastien@opendns.com>
Thomas Desrosiers <thomasdesr@gmail.com>
kklipsch <junk@klipsch.net>
seborama <sebastien.chatal@sainsburys.co.uk>
A big thank you to these persons who provided testing and amazing
feedback to goavro during its initial implementation:
Dennis Ordanov <dordanov@linkedin.com>
Thomas Desrosiers <thomasdesr@gmail.com>
Also a big thank you is extended to our supervisors who supported our
efforts to bring goavro to the open source community:
Greg Leffler <gleffler@linkedin.com>
Nick Berry <niberry@linkedin.com>
vendor/github.com/linkedin/goavro/v2/LICENSE
Generated Vendored
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
vendor/github.com/linkedin/goavro/v2/README.md
Generated Vendored
+410
View File
@@ -0,0 +1,410 @@
# goavro
Goavro is a library that encodes and decodes Avro data.
## Description
* Encodes to and decodes from both binary and textual JSON Avro data.
* `Codec` is stateless and is safe to use by multiple goroutines.
With the exception of features not yet supported, goavro attempts to
be fully compliant with the most recent version of the
[Avro specification](http://avro.apache.org/docs/1.8.2/spec.html).
## Dependency Notice
All usage of `gopkg.in` has been removed in favor of Go modules.
Please update your import paths to `github.com/linkedin/goavro/v2`. v1
users can still use old versions of goavro by adding a constraint to
your `go.mod` or `Gopkg.toml` file.
```
require (
github.com/linkedin/goavro v1.0.5
)
```
```toml
[[constraint]]
name = "github.com/linkedin/goavro"
version = "=1.0.5"
```
## Major Improvements in v2 over v1
### Avro namespaces
The original version of this library was written prior to my really
understanding how Avro namespaces ought to work. After using Avro for
a long time now, and after a lot of research, I think I grok Avro
namespaces properly, and the library now correctly handles every test
case the Apache Avro distribution has for namespaces, including being
able to refer to a previously defined data type later on in the same
schema.
### Getting Data into and out of Records
The original version of this library required creating `goavro.Record`
instances, and use of getters and setters to access a record's
fields. When schemas were complex, this required a lot of work to
debug and get right. The original version also required users to break
schemas in chunks, and have a different schema for each record
type. This was cumbersome, annoying, and error prone.
The new version of this library eliminates the `goavro.Record` type,
and accepts a native Go map for all records to be encoded. Keys are
the field names, and values are the field values. Nothing could be
more easy. Conversely, decoding Avro data yields a native Go map for
the upstream client to pull data back out of.
Furthermore, there is never a reason to ever have to break your schema
down into record schemas. Merely feed the entire schema into the
`NewCodec` function once when you create the `Codec`, then use
it. This library knows how to parse the data provided to it and ensure
data values for records and their fields are properly encoded and
decoded.
### 3x--4x Performance Improvement
The original version of this library was truly written with Go's idea
of `io.Reader` and `io.Writer` composition in mind. Although
composition is a powerful tool, the original library had to pull bytes
off the `io.Reader`--often one byte at a time--check for read errors,
decode the bytes, and repeat. This version, by using a native Go byte
slice, both decoding and encoding complex Avro data here at LinkedIn
is between three and four times faster than before.
### Avro JSON Support
The original version of this library did not support JSON encoding or
decoding, because it wasn't deemed useful for our internal use at the
time. When writing the new version of the library I decided to tackle
this issue once and for all, because so many engineers needed this
functionality for their work.
### Better Handling of Record Field Default Values
The original version of this library did not well handle default
values for record fields. This version of the library uses a default
value of a record field when encoding from native Go data to Avro data
and the record field is not specified. Additionally, when decoding
from Avro JSON data to native Go data, and a field is not specified,
the default value will be used to populate the field.
## Contrast With Code Generation Tools
If you have the ability to rebuild and redeploy your software whenever
data schemas change, code generation tools might be the best solution
for your application.
There are numerous excellent tools for generating source code to
translate data between native and Avro binary or textual data. One
such tool is linked below. If a particular application is designed to
work with a rarely changing schema, programs that use code generated
functions can potentially be more performant than a program that uses
goavro to create a `Codec` dynamically at run time.
* [gogen-avro](https://github.com/alanctgardner/gogen-avro)
I recommend benchmarking the resultant programs using typical data
using both the code generated functions and using goavro to see which
performs better. Not all code generated functions will out perform
goavro for all data corpuses.
If you don't have the ability to rebuild and redeploy software updates
whenever a data schema change occurs, goavro could be a great fit for
your needs. With goavro, your program can be given a new schema while
running, compile it into a `Codec` on the fly, and immediately start
encoding or decoding data using that `Codec`. Because Avro encoding
specifies that encoded data always be accompanied by a schema this is
not usually a problem. If the schema change is backwards compatible,
and the portion of your program that handles the decoded data is still
able to reference the decoded fields, there is nothing that needs to
be done when the schema change is detected by your program when using
goavro `Codec` instances to encode or decode data.
## Resources
* [Avro CLI Examples](https://github.com/miguno/avro-cli-examples)
* [Avro](https://avro.apache.org/)
* [Google Snappy](https://google.github.io/snappy/)
* [JavaScript Object Notation, JSON](https://www.json.org/)
* [Kafka](https://kafka.apache.org)
## Usage
Documentation is available via
[![GoDoc](https://godoc.org/github.com/linkedin/goavro?status.svg)](https://godoc.org/github.com/linkedin/goavro).
```Go
package main
import (
"fmt"
"github.com/linkedin/goavro/v2"
)
func main() {
codec, err := goavro.NewCodec(`
{
"type": "record",
"name": "LongList",
"fields" : [
{"name": "next", "type": ["null", "LongList"], "default": null}
]
}`)
if err != nil {
fmt.Println(err)
}
// NOTE: May omit fields when using default value
textual := []byte(`{"next":{"LongList":{}}}`)
// Convert textual Avro data (in Avro JSON format) to native Go form
native, _, err := codec.NativeFromTextual(textual)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to binary Avro data
binary, err := codec.BinaryFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// Convert binary Avro data back to native Go form
native, _, err = codec.NativeFromBinary(binary)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to textual Avro data
textual, err = codec.TextualFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// NOTE: Textual encoding will show all fields, even those with values that
// match their default values
fmt.Println(string(textual))
// Output: {"next":{"LongList":{"next":null}}}
}
```
Also please see the example programs in the `examples` directory for
reference.
### ab2t
The `ab2t` program is similar to the reference standard
`avrocat` program and converts Avro OCF files to Avro JSON
encoding.
### arw
The Avro-ReWrite program, `arw`, can be used to rewrite an
Avro OCF file while optionally changing the block counts, the
compression algorithm. `arw` can also upgrade the schema provided the
existing datum values can be encoded with the newly provided schema.
### avroheader
The Avro Header program, `avroheader`, can be used to print various
header information from an OCF file.
### splice
The `splice` program can be used to splice together an OCF file from
an Avro schema file and a raw Avro binary data file.
### Translating Data
A `Codec` provides four methods for translating between a byte slice
of either binary or textual Avro data and native Go data.
The following methods convert data between native Go data and byte
slices of the binary Avro representation:
BinaryFromNative
NativeFromBinary
The following methods convert data between native Go data and byte
slices of the textual Avro representation:
NativeFromTextual
TextualFromNative
Each `Codec` also exposes the `Schema` method to return a simplified
version of the JSON schema string used to create the `Codec`.
#### Translating From Avro to Go Data
Goavro does not use Go's structure tags to translate data between
native Go types and Avro encoded data.
When translating from either binary or textual Avro to native Go data,
goavro returns primitive Go data values for corresponding Avro data
values. The table below shows how goavro translates Avro types to Go
types.
| Avro | Go     |
| ------------------ | ------------------------ |
| `null` | `nil` |
| `boolean` | `bool` |
| `bytes` | `[]byte` |
| `float` | `float32` |
| `double` | `float64` |
| `long` | `int64` |
| `int` | `int32`   |
| `string` | `string` |
| `array` | `[]interface{}` |
| `enum` | `string` |
| `fixed` | `[]byte`       |
| `map` and `record` | `map[string]interface{}` |
| `union` | *see below*    |
Because of encoding rules for Avro unions, when an union's value is
`null`, a simple Go `nil` is returned. However when an union's value
is non-`nil`, a Go `map[string]interface{}` with a single key is
returned for the union. The map's single key is the Avro type name and
its value is the datum's value.
#### Translating From Go to Avro Data
Goavro does not use Go's structure tags to translate data between
native Go types and Avro encoded data.
When translating from native Go to either binary or textual Avro data,
goavro generally requires the same native Go data types as the decoder
would provide, with some exceptions for programmer convenience. Goavro
will accept any numerical data type provided there is no precision
lost when encoding the value. For instance, providing `float64(3.0)`
to an encoder expecting an Avro `int` would succeed, while sending
`float64(3.5)` to the same encoder would return an error.
When providing a slice of items for an encoder, the encoder will
accept either `[]interface{}`, or any slice of the required type. For
instance, when the Avro schema specifies:
`{"type":"array","items":"string"}`, the encoder will accept either
`[]interface{}`, or `[]string`. If given `[]int`, the encoder will
return an error when it attempts to encode the first non-string array
value using the string encoder.
When providing a value for an Avro union, the encoder will accept
`nil` for a `null` value. If the value is non-`nil`, it must be a
`map[string]interface{}` with a single key-value pair, where the key
is the Avro type name and the value is the datum's value. As a
convenience, the `Union` function wraps any datum value in a map as
specified above.
```Go
func ExampleUnion() {
codec, err := goavro.NewCodec(`["null","string","int"]`)
if err != nil {
fmt.Println(err)
}
buf, err := codec.TextualFromNative(nil, goavro.Union("string", "some string"))
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf))
// Output: {"string":"some string"}
}
```
## Limitations
Goavro is a fully featured encoder and decoder of binary and textual
JSON Avro data. It fully supports recursive data structures, unions,
and namespacing. It does have a few limitations that have yet to be
implemented.
### Aliases
The Avro specification allows an implementation to optionally map a
writer's schema to a reader's schema using aliases. Although goavro
can compile schemas with aliases, it does not yet implement this
feature.
### Kafka Streams
[Kafka](http://kafka.apache.org) is the reason goavro was
written. Similar to Avro Object Container Files being a layer of
abstraction above Avro Data Serialization format, Kafka's use of Avro
is a layer of abstraction that also sits above Avro Data Serialization
format, but has its own schema. Like Avro Object Container Files, this
has been implemented but removed until the API can be improved.
### Default Maximum Block Counts, and Block Sizes
When decoding arrays, maps, and OCF files, the Avro specification
states that the binary includes block counts and block sizes that
specify how many items are in the next block, and how many bytes are
in the next block. To prevent possible denial-of-service attacks on
clients that use this library caused by attempting to decode
maliciously crafted data, decoded block counts and sizes are compared
against public library variables MaxBlockCount and MaxBlockSize. When
the decoded values exceed these values, the decoder returns an error.
Because not every upstream client is the same, we've chosen some sane
defaults for these values, but left them as mutable variables, so that
clients are able to override if deemed necessary for their
purposes. Their initial default values are (`math.MaxInt32` or
~2.2GB).
### Schema Evolution
Please see [my reasons why schema evolution is broken for Avro
1.x](https://github.com/linkedin/goavro/blob/master/SCHEMA-EVOLUTION.md).
## License
### Goavro license
Copyright 2017 LinkedIn Corp. Licensed under the Apache License,
Version 2.0 (the "License"); you may not use this file except in
compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
### Google Snappy license
Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
## Third Party Dependencies
### Google Snappy
Goavro links with [Google Snappy](http://google.github.io/snappy/)
to provide Snappy compression and decompression support.
vendor/github.com/linkedin/goavro/v2/SCHEMA-EVOLUTION.md
Generated Vendored
+92
View File
@@ -0,0 +1,92 @@
From the Avro specification:
default: A default value for this field, used when reading instances
that lack this field (optional). Permitted values depend on the
field's schema type, according to the table below. Default values for
union fields correspond to the first schema in the union. Default
values for bytes and fixed fields are JSON strings, where Unicode code
points 0-255 are mapped to unsigned 8-bit byte values 0-255. I read
the above to mean that the purpose of default values are to allow
reading Avro data that was written without the fields, and not
necessarily augmentation of data being serialized. So in general I
agree with you in terms of purpose.
One very important aspect of Avro is that the schema used to serialize
the data should always remain with the data, so that a reader would
always be able to read the schema and then be able to consume the
data. I think most people still agree so far.
However, this is where things get messy. Schema evolution is
frequently cited when folks want to use a new version of the schema to
read data that was once written using an older version of that schema.
I do not believe the Avro specification properly handles schema
evolution. Here's a simple example:
```
Record v0:
name: string
nickname: string, default: ""
```
```
Record v1:
name: string
nickname: string, default: ""
title: string, default: ""
```
Okay, now a binary stream of records is just a bunch of strings. Let's
do that now.
```
0x0A, A, l, i, c, e, 0x06, B, o, b, 0x0A, B, r, u, c, e, 0x0A, S, a, l, l, y, 0x06, A, n, n
```
How many records is that? It could be as many as 5 records, each of a
single name and no nicknames. It could be as few as 2 records, one of
them with a nickname and a title, and one with only a nickname, or a
title.
Now to drive home the nail that Avro schema evolution is broken, even
if each record had a header that indicated how many bytes it would
consume, we could know where one record began and ended, and how many
records there are. But if we were to read a record with two strings
in it, is the second string the nickname or the title?
The Avro specification has no answer to that question, so neither do I.
Effectively, Avro could be a great tool for serializing complex data,
but it's broken in its current form, and to fix it would require it to
break compatibility with itself, effectively rendering any binary data
serialized in a previous version of Avro unreadable by new versions,
unless it had some sort of version marker on the data so a library
could branch.
One great solution would be augmenting the binary encoding with a
simple field number identifier. Let's imagine an Avro 2.x that had
this feature, and would support schema evolution. Here's an example
stream of bytes that could be unambiguously decoded using the new
schema:
```
0x02, 0x0A, A, l, i, c, e, 0x02, 0x06, B, o, B, 0x04, 0x0A, B, r, u, c, e, 0x02, 0x0C, C, h, a, r, l, i, e, 0x06, 0x04, M, r
```
In the above example of my fake Avro 2.0, this can be
deterministically decoded because 0x02 indicates the following is
field number 1 (name), followed by string length 5, followed by
Alice.
Then the decoder would see 0x02, marking field number 1 again,
which means, "next record", followed by string length 3, followed by
Bob, followed by 0x04, which means field number 2 (nickname), followed
by string length 5, followed by Bruce.
Followed by field number 1 (next record), followed by string length 6,
followed by Charlie, followed by field number 3 (title), followed by
string length 2, followed by Mr.
In my hypothetical version of Avro 2, Avro can cope with schema
evolution using record defaults and such. Sadly, Avro 1.x cannot and
thus we should avoid using it if your use-case requires schema
evolution.
vendor/github.com/linkedin/goavro/v2/array.go
Generated Vendored
+226
View File
@@ -0,0 +1,226 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"io"
"math"
"reflect"
)
func makeArrayCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
// array type must have items
itemSchema, ok := schemaMap["items"]
if !ok {
return nil, fmt.Errorf("Array ought to have items key")
}
itemCodec, err := buildCodec(st, enclosingNamespace, itemSchema)
if err != nil {
return nil, fmt.Errorf("Array items ought to be valid Avro type: %s", err)
}
return &Codec{
typeName: &name{"array", nullNamespace},
nativeFromBinary: func(buf []byte) (interface{}, []byte, error) {
var value interface{}
var err error
// block count and block size
if value, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary array block count: %s", err)
}
blockCount := value.(int64)
if blockCount < 0 {
// NOTE: A negative block count implies there is a long encoded
// block size following the negative block count. We have no use
// for the block size in this decoder, so we read and discard
// the value.
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can never be made positive
return nil, nil, fmt.Errorf("cannot decode binary array with block count: %d", blockCount)
}
blockCount = -blockCount // convert to its positive equivalent
if _, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary array block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, nil, fmt.Errorf("cannot decode binary array when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
// NOTE: While the attempt of a RAM optimization shown below is not
// necessary, many encoders will encode all items in a single block.
// We can optimize amount of RAM allocated by runtime for the array
// by initializing the array for that number of items.
arrayValues := make([]interface{}, 0, blockCount)
for blockCount != 0 {
// Decode `blockCount` datum values from buffer
for i := int64(0); i < blockCount; i++ {
if value, buf, err = itemCodec.nativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary array item %d: %s", i+1, err)
}
arrayValues = append(arrayValues, value)
}
// Decode next blockCount from buffer, because there may be more blocks
if value, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary array block count: %s", err)
}
blockCount = value.(int64)
if blockCount < 0 {
// NOTE: A negative block count implies there is a long
// encoded block size following the negative block count. We
// have no use for the block size in this decoder, so we
// read and discard the value.
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can
// never be made positive
return nil, nil, fmt.Errorf("cannot decode binary array with block count: %d", blockCount)
}
blockCount = -blockCount // convert to its positive equivalent
if _, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary array block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, nil, fmt.Errorf("cannot decode binary array when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
}
return arrayValues, buf, nil
},
binaryFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
arrayValues, err := convertArray(datum)
if err != nil {
return nil, fmt.Errorf("cannot encode binary array: %s", err)
}
arrayLength := int64(len(arrayValues))
var alreadyEncoded, remainingInBlock int64
for i, item := range arrayValues {
if remainingInBlock == 0 { // start a new block
remainingInBlock = arrayLength - alreadyEncoded
if remainingInBlock > MaxBlockCount {
// limit block count to MacBlockCount
remainingInBlock = MaxBlockCount
}
buf, _ = longBinaryFromNative(buf, remainingInBlock)
}
if buf, err = itemCodec.binaryFromNative(buf, item); err != nil {
return nil, fmt.Errorf("cannot encode binary array item %d: %v: %s", i+1, item, err)
}
remainingInBlock--
alreadyEncoded++
}
return longBinaryFromNative(buf, 0) // append trailing 0 block count to signal end of Array
},
nativeFromTextual: func(buf []byte) (interface{}, []byte, error) {
var arrayValues []interface{}
var value interface{}
var err error
var b byte
if buf, err = advanceAndConsume(buf, '['); err != nil {
return nil, nil, fmt.Errorf("cannot decode textual array: %s", err)
}
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, fmt.Errorf("cannot decode textual array: %s", io.ErrShortBuffer)
}
// NOTE: Special case for empty array
if buf[0] == ']' {
return arrayValues, buf[1:], nil
}
// NOTE: Also terminates when read ']' byte.
for len(buf) > 0 {
// decode value
value, buf, err = itemCodec.nativeFromTextual(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual array: %s", err)
}
arrayValues = append(arrayValues, value)
// either comma or closing curly brace
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, fmt.Errorf("cannot decode textual array: %s", io.ErrShortBuffer)
}
switch b = buf[0]; b {
case ']':
return arrayValues, buf[1:], nil
case ',':
// no-op
default:
return nil, nil, fmt.Errorf("cannot decode textual array: expected ',' or ']'; received: %q", b)
}
// NOTE: consume comma from above
if buf, _ = advanceToNonWhitespace(buf[1:]); len(buf) == 0 {
return nil, nil, fmt.Errorf("cannot decode textual array: %s", io.ErrShortBuffer)
}
}
return nil, buf, io.ErrShortBuffer
},
textualFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
arrayValues, err := convertArray(datum)
if err != nil {
return nil, fmt.Errorf("cannot encode textual array: %s", err)
}
var atLeastOne bool
buf = append(buf, '[')
for i, item := range arrayValues {
atLeastOne = true
// Encode value
buf, err = itemCodec.textualFromNative(buf, item)
if err != nil {
// field was specified in datum; therefore its value was invalid
return nil, fmt.Errorf("cannot encode textual array item %d; %v: %s", i+1, item, err)
}
buf = append(buf, ',')
}
if atLeastOne {
return append(buf[:len(buf)-1], ']'), nil
}
return append(buf, ']'), nil
},
}, nil
}
// convertArray converts interface{} to []interface{} if possible.
func convertArray(datum interface{}) ([]interface{}, error) {
arrayValues, ok := datum.([]interface{})
if ok {
return arrayValues, nil
}
// NOTE: When given a slice of any other type, zip values to
// items as a convenience to client.
v := reflect.ValueOf(datum)
if v.Kind() != reflect.Slice {
return nil, fmt.Errorf("cannot create []interface{}: expected slice; received: %T", datum)
}
// NOTE: Two better alternatives to the current algorithm are:
// (1) mutate the reflection tuple underneath to convert the
// []int, for example, to []interface{}, with O(1) complexity
// (2) use copy builtin to zip the data items over with O(n) complexity,
// but more efficient than what's below.
// Suggestions?
arrayValues = make([]interface{}, v.Len())
for idx := 0; idx < v.Len(); idx++ {
arrayValues[idx] = v.Index(idx).Interface()
}
return arrayValues, nil
}
vendor/github.com/linkedin/goavro/v2/binaryReader.go
Generated Vendored
+160
View File
@@ -0,0 +1,160 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"io"
"math"
)
// bytesBinaryReader reads bytes from io.Reader and returns byte slice of
// specified size or the error encountered while trying to read those bytes.
func bytesBinaryReader(ior io.Reader) ([]byte, error) {
size, err := longBinaryReader(ior)
if err != nil {
return nil, fmt.Errorf("cannot read bytes: cannot read size: %s", err)
}
if size < 0 {
return nil, fmt.Errorf("cannot read bytes: size is negative: %d", size)
}
if size > MaxBlockSize {
return nil, fmt.Errorf("cannot read bytes: size exceeds MaxBlockSize: %d > %d", size, MaxBlockSize)
}
buf := make([]byte, size)
_, err = io.ReadAtLeast(ior, buf, int(size))
if err != nil {
return nil, fmt.Errorf("cannot read bytes: %s", err)
}
return buf, nil
}
// longBinaryReader reads bytes from io.Reader until has complete long value, or
// read error.
func longBinaryReader(ior io.Reader) (int64, error) {
var value uint64
var shift uint
var err error
var b byte
// NOTE: While benchmarks show it's more performant to invoke ReadByte when
// available, testing whether a variable's data type implements a particular
// method is quite slow too. So perform the test once, and branch to the
// appropriate loop based on the results.
if byteReader, ok := ior.(io.ByteReader); ok {
for {
if b, err = byteReader.ReadByte(); err != nil {
return 0, err // NOTE: must send back unaltered error to detect io.EOF
}
value |= uint64(b&intMask) << shift
if b&intFlag == 0 {
return (int64(value>>1) ^ -int64(value&1)), nil
}
shift += 7
}
}
// NOTE: ior does not also implement io.ByteReader, so we must allocate a
// byte slice with a single byte, and read each byte into the slice.
buf := make([]byte, 1)
for {
if _, err = ior.Read(buf); err != nil {
return 0, err // NOTE: must send back unaltered error to detect io.EOF
}
b = buf[0]
value |= uint64(b&intMask) << shift
if b&intFlag == 0 {
return (int64(value>>1) ^ -int64(value&1)), nil
}
shift += 7
}
}
// metadataBinaryReader reads bytes from io.Reader until has entire map value,
// or read error.
func metadataBinaryReader(ior io.Reader) (map[string][]byte, error) {
var err error
var value interface{}
// block count and block size
if value, err = longBinaryReader(ior); err != nil {
return nil, fmt.Errorf("cannot read map block count: %s", err)
}
blockCount := value.(int64)
if blockCount < 0 {
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can never be
// made positive
return nil, fmt.Errorf("cannot read map with block count: %d", blockCount)
}
// NOTE: A negative block count implies there is a long encoded block
// size following the negative block count. We have no use for the block
// size in this decoder, so we read and discard the value.
blockCount = -blockCount // convert to its positive equivalent
if _, err = longBinaryReader(ior); err != nil {
return nil, fmt.Errorf("cannot read map block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, fmt.Errorf("cannot read map when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
// NOTE: While the attempt of a RAM optimization shown below is not
// necessary, many encoders will encode all items in a single block. We can
// optimize amount of RAM allocated by runtime for the array by initializing
// the array for that number of items.
mapValues := make(map[string][]byte, blockCount)
for blockCount != 0 {
// Decode `blockCount` datum values from buffer
for i := int64(0); i < blockCount; i++ {
// first decode the key string
keyBytes, err := bytesBinaryReader(ior)
if err != nil {
return nil, fmt.Errorf("cannot read map key: %s", err)
}
key := string(keyBytes)
if _, ok := mapValues[key]; ok {
return nil, fmt.Errorf("cannot read map: duplicate key: %q", key)
}
// metadata values are always bytes
buf, err := bytesBinaryReader(ior)
if err != nil {
return nil, fmt.Errorf("cannot read map value for key %q: %s", key, err)
}
mapValues[key] = buf
}
// Decode next blockCount from buffer, because there may be more blocks
if value, err = longBinaryReader(ior); err != nil {
return nil, fmt.Errorf("cannot read map block count: %s", err)
}
blockCount = value.(int64)
if blockCount < 0 {
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can never be
// made positive
return nil, fmt.Errorf("cannot read map with block count: %d", blockCount)
}
// NOTE: A negative block count implies there is a long encoded
// block size following the negative block count. We have no use for
// the block size in this decoder, so we read and discard the value.
blockCount = -blockCount // convert to its positive equivalent
if _, err = longBinaryReader(ior); err != nil {
return nil, fmt.Errorf("cannot read map block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, fmt.Errorf("cannot read map when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
}
return mapValues, nil
}
vendor/github.com/linkedin/goavro/v2/boolean.go
Generated Vendored
+72
View File
@@ -0,0 +1,72 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"errors"
"fmt"
"io"
)
func booleanNativeFromBinary(buf []byte) (interface{}, []byte, error) {
if len(buf) < 1 {
return nil, nil, io.ErrShortBuffer
}
var b byte
b, buf = buf[0], buf[1:]
switch b {
case byte(0):
return false, buf, nil
case byte(1):
return true, buf, nil
default:
return nil, nil, fmt.Errorf("cannot decode binary boolean: expected: Go byte(0) or byte(1); received: byte(%d)", b)
}
}
func booleanBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
value, ok := datum.(bool)
if !ok {
return nil, fmt.Errorf("cannot encode binary boolean: expected: Go bool; received: %T", datum)
}
var b byte
if value {
b = 1
}
return append(buf, b), nil
}
func booleanNativeFromTextual(buf []byte) (interface{}, []byte, error) {
if len(buf) < 4 {
return nil, nil, fmt.Errorf("cannot decode textual boolean: %s", io.ErrShortBuffer)
}
if bytes.Equal(buf[:4], []byte("true")) {
return true, buf[4:], nil
}
if len(buf) < 5 {
return nil, nil, fmt.Errorf("cannot decode textual boolean: %s", io.ErrShortBuffer)
}
if bytes.Equal(buf[:5], []byte("false")) {
return false, buf[5:], nil
}
return nil, nil, errors.New("expected false or true")
}
func booleanTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
value, ok := datum.(bool)
if !ok {
return nil, fmt.Errorf("boolean: expected: Go bool; received: %T", datum)
}
if value {
return append(buf, "true"...), nil
}
return append(buf, "false"...), nil
}
vendor/github.com/linkedin/goavro/v2/bytes.go
Generated Vendored
+537
View File
@@ -0,0 +1,537 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"encoding/hex"
"errors"
"fmt"
"io"
"os"
"unicode"
"unicode/utf16"
"unicode/utf8"
)
////////////////////////////////////////
// Binary Decode
////////////////////////////////////////
func bytesNativeFromBinary(buf []byte) (interface{}, []byte, error) {
if len(buf) < 1 {
return nil, nil, fmt.Errorf("cannot decode binary bytes: %s", io.ErrShortBuffer)
}
var decoded interface{}
var err error
if decoded, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary bytes: %s", err)
}
size := decoded.(int64) // always returns int64
if size < 0 {
return nil, nil, fmt.Errorf("cannot decode binary bytes: negative size: %d", size)
}
if size > int64(len(buf)) {
return nil, nil, fmt.Errorf("cannot decode binary bytes: %s", io.ErrShortBuffer)
}
return buf[:size], buf[size:], nil
}
func stringNativeFromBinary(buf []byte) (interface{}, []byte, error) {
d, b, err := bytesNativeFromBinary(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode binary string: %s", err)
}
return string(d.([]byte)), b, nil
}
////////////////////////////////////////
// Binary Encode
////////////////////////////////////////
func bytesBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var someBytes []byte
switch d := datum.(type) {
case []byte:
someBytes = d
case string:
someBytes = []byte(d)
default:
return nil, fmt.Errorf("cannot encode binary bytes: expected: []byte or string; received: %T", datum)
}
buf, _ = longBinaryFromNative(buf, len(someBytes)) // only fails when given non integer
return append(buf, someBytes...), nil // append datum bytes
}
func stringBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var someBytes []byte
switch d := datum.(type) {
case []byte:
someBytes = d
case string:
someBytes = []byte(d)
default:
return nil, fmt.Errorf("cannot encode binary bytes: expected: string; received: %T", datum)
}
buf, _ = longBinaryFromNative(buf, len(someBytes)) // only fails when given non integer
return append(buf, someBytes...), nil // append datum bytes
}
////////////////////////////////////////
// Text Decode
////////////////////////////////////////
func bytesNativeFromTextual(buf []byte) (interface{}, []byte, error) {
buflen := len(buf)
if buflen < 2 {
return nil, nil, fmt.Errorf("cannot decode textual bytes: %s", io.ErrShortBuffer)
}
if buf[0] != '"' {
return nil, nil, fmt.Errorf("cannot decode textual bytes: expected initial \"; found: %#U", buf[0])
}
var newBytes []byte
var escaped bool
// Loop through bytes following initial double quote, but note we will
// return immediately when find unescaped double quote.
for i := 1; i < buflen; i++ {
b := buf[i]
if escaped {
escaped = false
if b2, ok := unescapeSpecialJSON(b); ok {
newBytes = append(newBytes, b2)
continue
}
if b == 'u' {
// NOTE: Need at least 4 more bytes to read uint16, but subtract
// 1 because do not want to count the trailing quote and
// subtract another 1 because already consumed u but have yet to
// increment i.
if i > buflen-6 {
return nil, nil, fmt.Errorf("cannot decode textual bytes: %s", io.ErrShortBuffer)
}
// NOTE: Avro bytes represent binary data, and do not
// necessarily represent text. Therefore, Avro bytes are not
// encoded in UTF-16. Each \u is followed by 4 hexadecimal
// digits, the first and second of which must be 0.
v, err := parseUint64FromHexSlice(buf[i+3 : i+5])
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual bytes: %s", err)
}
i += 4 // absorb 4 characters: one 'u' and three of the digits
newBytes = append(newBytes, byte(v))
continue
}
newBytes = append(newBytes, b)
continue
}
if b == '\\' {
escaped = true
continue
}
if b == '"' {
return newBytes, buf[i+1:], nil
}
newBytes = append(newBytes, b)
}
return nil, nil, fmt.Errorf("cannot decode textual bytes: expected final \"; found: %#U", buf[buflen-1])
}
func stringNativeFromTextual(buf []byte) (interface{}, []byte, error) {
buflen := len(buf)
if buflen < 2 {
return nil, nil, fmt.Errorf("cannot decode textual string: %s", io.ErrShortBuffer)
}
if buf[0] != '"' {
return nil, nil, fmt.Errorf("cannot decode textual string: expected initial \"; found: %#U", buf[0])
}
var newBytes []byte
var escaped bool
// Loop through bytes following initial double quote, but note we will
// return immediately when find unescaped double quote.
for i := 1; i < buflen; i++ {
b := buf[i]
if escaped {
escaped = false
if b2, ok := unescapeSpecialJSON(b); ok {
newBytes = append(newBytes, b2)
continue
}
if b == 'u' {
// NOTE: Need at least 4 more bytes to read uint16, but subtract
// 1 because do not want to count the trailing quote and
// subtract another 1 because already consumed u but have yet to
// increment i.
if i > buflen-6 {
return nil, nil, fmt.Errorf("cannot decode textual string: %s", io.ErrShortBuffer)
}
v, err := parseUint64FromHexSlice(buf[i+1 : i+5])
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual string: %s", err)
}
i += 4 // absorb 4 characters: one 'u' and three of the digits
nbl := len(newBytes)
newBytes = append(newBytes, []byte{0, 0, 0, 0}...) // grow to make room for UTF-8 encoded rune
r := rune(v)
if utf16.IsSurrogate(r) {
i++ // absorb final hexadecimal digit from previous value
// Expect second half of surrogate pair
if i > buflen-6 || buf[i] != '\\' || buf[i+1] != 'u' {
return nil, nil, errors.New("cannot decode textual string: missing second half of surrogate pair")
}
v, err = parseUint64FromHexSlice(buf[i+2 : i+6])
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual string: %s", err)
}
i += 5 // absorb 5 characters: two for '\u', and 3 of the 4 digits
// Get code point by combining high and low surrogate bits
r = utf16.DecodeRune(r, rune(v))
}
width := utf8.EncodeRune(newBytes[nbl:], r) // append UTF-8 encoded version of code point
newBytes = newBytes[:nbl+width] // trim off excess bytes
continue
}
newBytes = append(newBytes, b)
continue
}
if b == '\\' {
escaped = true
continue
}
if b == '"' {
return string(newBytes), buf[i+1:], nil
}
newBytes = append(newBytes, b)
}
if escaped {
return nil, nil, fmt.Errorf("cannot decode textual string: %s", io.ErrShortBuffer)
}
return nil, nil, fmt.Errorf("cannot decode textual string: expected final \"; found: %x", buf[buflen-1])
}
func unescapeUnicodeString(some string) (string, error) {
if some == "" {
return "", nil
}
buf := []byte(some)
buflen := len(buf)
var i int
var newBytes []byte
var escaped bool
// Loop through bytes following initial double quote, but note we will
// return immediately when find unescaped double quote.
for i = 0; i < buflen; i++ {
b := buf[i]
if escaped {
escaped = false
if b == 'u' {
// NOTE: Need at least 4 more bytes to read uint16, but subtract
// 1 because do not want to count the trailing quote and
// subtract another 1 because already consumed u but have yet to
// increment i.
if i > buflen-6 {
return "", fmt.Errorf("cannot replace escaped characters with UTF-8 equivalent: %s", io.ErrShortBuffer)
}
v, err := parseUint64FromHexSlice(buf[i+1 : i+5])
if err != nil {
return "", fmt.Errorf("cannot replace escaped characters with UTF-8 equivalent: %s", err)
}
i += 4 // absorb 4 characters: one 'u' and three of the digits
nbl := len(newBytes)
newBytes = append(newBytes, []byte{0, 0, 0, 0}...) // grow to make room for UTF-8 encoded rune
r := rune(v)
if utf16.IsSurrogate(r) {
i++ // absorb final hexadecimal digit from previous value
// Expect second half of surrogate pair
if i > buflen-6 || buf[i] != '\\' || buf[i+1] != 'u' {
return "", errors.New("cannot replace escaped characters with UTF-8 equivalent: missing second half of surrogate pair")
}
v, err = parseUint64FromHexSlice(buf[i+2 : i+6])
if err != nil {
return "", fmt.Errorf("cannot replace escaped characters with UTF-8 equivalents: %s", err)
}
i += 5 // absorb 5 characters: two for '\u', and 3 of the 4 digits
// Get code point by combining high and low surrogate bits
r = utf16.DecodeRune(r, rune(v))
}
width := utf8.EncodeRune(newBytes[nbl:], r) // append UTF-8 encoded version of code point
newBytes = newBytes[:nbl+width] // trim off excess bytes
continue
}
newBytes = append(newBytes, b)
continue
}
if b == '\\' {
escaped = true
continue
}
newBytes = append(newBytes, b)
}
if escaped {
return "", fmt.Errorf("cannot replace escaped characters with UTF-8 equivalents: %s", io.ErrShortBuffer)
}
return string(newBytes), nil
}
func parseUint64FromHexSlice(buf []byte) (uint64, error) {
var value uint64
for _, b := range buf {
diff := uint64(b - '0')
if diff < 10 {
value = (value << 4) | diff
continue
}
b10 := b + 10
diff = uint64(b10 - 'A')
if diff < 10 {
return 0, hex.InvalidByteError(b)
}
if diff < 16 {
value = (value << 4) | diff
continue
}
diff = uint64(b10 - 'a')
if diff < 10 {
return 0, hex.InvalidByteError(b)
}
if diff < 16 {
value = (value << 4) | diff
continue
}
return 0, hex.InvalidByteError(b)
}
return value, nil
}
func unescapeSpecialJSON(b byte) (byte, bool) {
// NOTE: The following 8 special JSON characters must be escaped:
switch b {
case '"', '\\', '/':
return b, true
case 'b':
return '\b', true
case 'f':
return '\f', true
case 'n':
return '\n', true
case 'r':
return '\r', true
case 't':
return '\t', true
}
return b, false
}
////////////////////////////////////////
// Text Encode
////////////////////////////////////////
func bytesTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
var someBytes []byte
switch d := datum.(type) {
case []byte:
someBytes = d
case string:
someBytes = []byte(d)
default:
return nil, fmt.Errorf("cannot encode textual bytes: expected: []byte or string; received: %T", datum)
}
buf = append(buf, '"') // prefix buffer with double quote
for _, b := range someBytes {
if escaped, ok := escapeSpecialJSON(b); ok {
buf = append(buf, escaped...)
continue
}
if r := rune(b); r < utf8.RuneSelf && unicode.IsPrint(r) {
buf = append(buf, b)
continue
}
// This Code Point _could_ be encoded as a single byte, however, it's
// above standard ASCII range (b > 127), therefore must encode using its
// four-byte hexadecimal equivalent, which will always start with the
// high byte 00
buf = appendUnicodeHex(buf, uint16(b))
}
return append(buf, '"'), nil // postfix buffer with double quote
}
func stringTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
var someString string
switch d := datum.(type) {
case []byte:
someString = string(d)
case string:
someString = d
default:
return nil, fmt.Errorf("cannot encode textual string: expected: []byte or string; received: %T", datum)
}
buf = append(buf, '"') // prefix buffer with double quote
for _, r := range someString {
if r < utf8.RuneSelf {
if escaped, ok := escapeSpecialJSON(byte(r)); ok {
buf = append(buf, escaped...)
continue
}
if unicode.IsPrint(r) {
buf = append(buf, byte(r))
continue
}
}
// NOTE: Attempt to encode code point as UTF-16 surrogate pair
r1, r2 := utf16.EncodeRune(r)
if r1 != unicode.ReplacementChar || r2 != unicode.ReplacementChar {
// code point does require surrogate pair, and thus two uint16 values
buf = appendUnicodeHex(buf, uint16(r1))
buf = appendUnicodeHex(buf, uint16(r2))
continue
}
// Code Point does not require surrogate pair.
buf = appendUnicodeHex(buf, uint16(r))
}
return append(buf, '"'), nil // postfix buffer with double quote
}
func appendUnicodeHex(buf []byte, v uint16) []byte {
// Start with '\u' prefix:
buf = append(buf, sliceUnicode...)
// And tack on 4 hexadecimal digits:
buf = append(buf, hexDigits[(v&0xF000)>>12])
buf = append(buf, hexDigits[(v&0xF00)>>8])
buf = append(buf, hexDigits[(v&0xF0)>>4])
buf = append(buf, hexDigits[(v&0xF)])
return buf
}
const hexDigits = "0123456789ABCDEF"
func escapeSpecialJSON(b byte) ([]byte, bool) {
// NOTE: The following 8 special JSON characters must be escaped:
switch b {
case '"':
return sliceQuote, true
case '\\':
return sliceBackslash, true
case '/':
return sliceSlash, true
case '\b':
return sliceBackspace, true
case '\f':
return sliceFormfeed, true
case '\n':
return sliceNewline, true
case '\r':
return sliceCarriageReturn, true
case '\t':
return sliceTab, true
}
return nil, false
}
// While slices in Go are never constants, we can initialize them once and reuse
// them many times. We define these slices at library load time and reuse them
// when encoding JSON.
var (
sliceQuote = []byte("\\\"")
sliceBackslash = []byte("\\\\")
sliceSlash = []byte("\\/")
sliceBackspace = []byte("\\b")
sliceFormfeed = []byte("\\f")
sliceNewline = []byte("\\n")
sliceCarriageReturn = []byte("\\r")
sliceTab = []byte("\\t")
sliceUnicode = []byte("\\u")
)
// DEBUG -- remove function prior to committing
func decodedStringFromJSON(buf []byte) (string, []byte, error) {
fmt.Fprintf(os.Stderr, "decodedStringFromJSON(%v)\n", buf)
buflen := len(buf)
if buflen < 2 {
return "", buf, fmt.Errorf("cannot decode string: %s", io.ErrShortBuffer)
}
if buf[0] != '"' {
return "", buf, fmt.Errorf("cannot decode string: expected initial '\"'; found: %#U", buf[0])
}
var newBytes []byte
var escaped, ok bool
// Loop through bytes following initial double quote, but note we will
// return immediately when find unescaped double quote.
for i := 1; i < buflen; i++ {
b := buf[i]
if escaped {
escaped = false
if b, ok = unescapeSpecialJSON(b); ok {
newBytes = append(newBytes, b)
continue
}
if b == 'u' {
// NOTE: Need at least 4 more bytes to read uint16, but subtract
// 1 because do not want to count the trailing quote and
// subtract another 1 because already consumed u but have yet to
// increment i.
if i > buflen-6 {
return "", buf[i+1:], fmt.Errorf("cannot decode string: %s", io.ErrShortBuffer)
}
v, err := parseUint64FromHexSlice(buf[i+1 : i+5])
if err != nil {
return "", buf[i+1:], fmt.Errorf("cannot decode string: %s", err)
}
i += 4 // absorb 4 characters: one 'u' and three of the digits
nbl := len(newBytes)
newBytes = append(newBytes, 0, 0, 0, 0) // grow to make room for UTF-8 encoded rune
r := rune(v)
if utf16.IsSurrogate(r) {
i++ // absorb final hexidecimal digit from previous value
// Expect second half of surrogate pair
if i > buflen-6 || buf[i] != '\\' || buf[i+1] != 'u' {
return "", buf[i+1:], errors.New("cannot decode string: missing second half of surrogate pair")
}
v, err = parseUint64FromHexSlice(buf[i+2 : i+6])
if err != nil {
return "", buf[i+1:], fmt.Errorf("cannot decode string: cannot decode second half of surrogate pair: %s", err)
}
i += 5 // absorb 5 characters: two for '\u', and 3 of the 4 digits
// Get code point by combining high and low surrogate bits
r = utf16.DecodeRune(r, rune(v))
}
width := utf8.EncodeRune(newBytes[nbl:], r) // append UTF-8 encoded version of code point
newBytes = newBytes[:nbl+width] // trim off excess bytes
continue
}
newBytes = append(newBytes, b)
continue
}
if b == '\\' {
escaped = true
continue
}
if b == '"' {
return string(newBytes), buf[i+1:], nil
}
newBytes = append(newBytes, b)
}
return "", buf, fmt.Errorf("cannot decode string: expected final '\"'; found: %#U", buf[buflen-1])
}
vendor/github.com/linkedin/goavro/v2/canonical.go
Generated Vendored
+188
View File
@@ -0,0 +1,188 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"sort"
"strconv"
"strings"
)
// pcfProcessor is a function type that given a parsed JSON object, returns its
// Parsing Canonical Form according to the Avro specification.
type pcfProcessor func(s interface{}) (string, error)
// parsingCanonialForm returns the "Parsing Canonical Form" (pcf) for a parsed
// JSON structure of a valid Avro schema, or an error describing the schema
// error.
func parsingCanonicalForm(schema interface{}, parentNamespace string, typeLookup map[string]string) (string, error) {
switch val := schema.(type) {
case map[string]interface{}:
// JSON objects are decoded as a map of strings to empty interfaces
return pcfObject(val, parentNamespace, typeLookup)
case []interface{}:
// JSON arrays are decoded as a slice of empty interfaces
return pcfArray(val, parentNamespace, typeLookup)
case string:
// JSON string values are decoded as a Go string
return pcfString(val, typeLookup)
case float64:
// JSON numerical values are decoded as Go float64
return pcfNumber(val)
default:
return "", fmt.Errorf("cannot parse schema with invalid schema type; ought to be map[string]interface{}, []interface{}, string, or float64; received: %T: %v", schema, schema)
}
}
// pcfNumber returns the parsing canonical form for a numerical value.
func pcfNumber(val float64) (string, error) {
return strconv.FormatFloat(val, 'g', -1, 64), nil
}
// pcfString returns the parsing canonical form for a string value.
func pcfString(val string, typeLookup map[string]string) (string, error) {
if canonicalName, ok := typeLookup[val]; ok {
return `"` + canonicalName + `"`, nil
}
return `"` + val + `"`, nil
}
// pcfArray returns the parsing canonical form for a JSON array.
func pcfArray(val []interface{}, parentNamespace string, typeLookup map[string]string) (string, error) {
items := make([]string, len(val))
for i, el := range val {
p, err := parsingCanonicalForm(el, parentNamespace, typeLookup)
if err != nil {
return "", err
}
items[i] = p
}
return "[" + strings.Join(items, ",") + "]", nil
}
// pcfObject returns the parsing canonical form for a JSON object.
func pcfObject(jsonMap map[string]interface{}, parentNamespace string, typeLookup map[string]string) (string, error) {
pairs := make(stringPairs, 0, len(jsonMap))
// Remember the namespace to fully qualify names later
var namespace string
if namespaceJSON, ok := jsonMap["namespace"]; ok {
if namespaceStr, ok := namespaceJSON.(string); ok {
// and it's value is string (otherwise invalid schema)
if parentNamespace == "" {
namespace = namespaceStr
} else {
namespace = parentNamespace + "." + namespaceStr
}
parentNamespace = namespace
}
} else if objectType, ok := jsonMap["type"]; ok && objectType == "record" {
namespace = parentNamespace
}
for k, v := range jsonMap {
// Reduce primitive schemas to their simple form.
if len(jsonMap) == 1 && k == "type" {
if t, ok := v.(string); ok {
return "\"" + t + "\"", nil
}
}
// Only keep relevant attributes (strip 'doc', 'alias', 'namespace')
if _, ok := fieldOrder[k]; !ok {
continue
}
// Add namespace to a non-qualified name.
if k == "name" && namespace != "" {
// Check if the name isn't already qualified.
if t, ok := v.(string); ok && !strings.ContainsRune(t, '.') {
v = namespace + "." + t
typeLookup[t] = v.(string)
}
}
// Only fixed type allows size, and we must convert a string size to a
// float.
if k == "size" {
if s, ok := v.(string); ok {
s, err := strconv.ParseUint(s, 10, 0)
if err != nil {
// should never get here because already validated schema
return "", fmt.Errorf("Fixed size ought to be number greater than zero: %v", s)
}
v = float64(s)
}
}
pk, err := parsingCanonicalForm(k, parentNamespace, typeLookup)
if err != nil {
return "", err
}
pv, err := parsingCanonicalForm(v, parentNamespace, typeLookup)
if err != nil {
return "", err
}
pairs = append(pairs, stringPair{k, pk + ":" + pv})
}
// Sort keys by their order in specification.
sort.Sort(byAvroFieldOrder(pairs))
return "{" + strings.Join(pairs.Bs(), ",") + "}", nil
}
// stringPair represents a pair of string values.
type stringPair struct {
A string
B string
}
// stringPairs is a sortable slice of pairs of strings.
type stringPairs []stringPair
// Bs returns an array of second values of an array of pairs.
func (sp *stringPairs) Bs() []string {
items := make([]string, len(*sp))
for i, el := range *sp {
items[i] = el.B
}
return items
}
// fieldOrder defines fields that show up in canonical schema and specifies
// their precedence.
var fieldOrder = map[string]int{
"name": 1,
"type": 2,
"fields": 3,
"symbols": 4,
"items": 5,
"values": 6,
"size": 7,
}
// byAvroFieldOrder is equipped with a sort order of fields according to the
// specification.
type byAvroFieldOrder []stringPair
func (s byAvroFieldOrder) Len() int {
return len(s)
}
func (s byAvroFieldOrder) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s byAvroFieldOrder) Less(i, j int) bool {
return fieldOrder[s[i].A] < fieldOrder[s[j].A]
}
vendor/github.com/linkedin/goavro/v2/codec.go
Generated Vendored
+607
View File
@@ -0,0 +1,607 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"encoding/binary"
"encoding/json"
"fmt"
"math"
"strconv"
)
var (
// MaxBlockCount is the maximum number of data items allowed in a single
// block that will be decoded from a binary stream, whether when reading
// blocks to decode an array or a map, or when reading blocks from an OCF
// stream. This check is to ensure decoding binary data will not cause the
// library to over allocate RAM, potentially creating a denial of service on
// the system.
//
// If a particular application needs to decode binary Avro data that
// potentially has more data items in a single block, then this variable may
// be modified at your discretion.
MaxBlockCount = int64(math.MaxInt32)
// MaxBlockSize is the maximum number of bytes that will be allocated for a
// single block of data items when decoding from a binary stream. This check
// is to ensure decoding binary data will not cause the library to over
// allocate RAM, potentially creating a denial of service on the system.
//
// If a particular application needs to decode binary Avro data that
// potentially has more bytes in a single block, then this variable may be
// modified at your discretion.
MaxBlockSize = int64(math.MaxInt32)
)
// Codec supports decoding binary and text Avro data to Go native data types,
// and conversely encoding Go native data types to binary or text Avro data. A
// Codec is created as a stateless structure that can be safely used in multiple
// go routines simultaneously.
type Codec struct {
soeHeader []byte // single-object-encoding header
schemaOriginal string
schemaCanonical string
typeName *name
nativeFromTextual func([]byte) (interface{}, []byte, error)
binaryFromNative func([]byte, interface{}) ([]byte, error)
nativeFromBinary func([]byte) (interface{}, []byte, error)
textualFromNative func([]byte, interface{}) ([]byte, error)
Rabin uint64
}
// NewCodec returns a Codec used to translate between a byte slice of either
// binary or textual Avro data and native Go data.
//
// Creating a `Codec` is fast, but ought to be performed exactly once per Avro
// schema to process. Once a `Codec` is created, it may be used multiple times
// to convert data between native form and binary Avro representation, or
// between native form and textual Avro representation.
//
// A particular `Codec` can work with only one Avro schema. However,
// there is no practical limit to how many `Codec`s may be created and
// used in a program. Internally a `Codec` is merely a named tuple of
// four function pointers, and maintains no runtime state that is mutated
// after instantiation. In other words, `Codec`s may be safely used by
// many go routines simultaneously, as your program requires.
//
// codec, err := goavro.NewCodec(`
// {
// "type": "record",
// "name": "LongList",
// "fields" : [
// {"name": "next", "type": ["null", "LongList"], "default": null}
// ]
// }`)
// if err != nil {
// fmt.Println(err)
// }
func NewCodec(schemaSpecification string) (*Codec, error) {
var schema interface{}
if err := json.Unmarshal([]byte(schemaSpecification), &schema); err != nil {
return nil, fmt.Errorf("cannot unmarshal schema JSON: %s", err)
}
// bootstrap a symbol table with primitive type codecs for the new codec
st := newSymbolTable()
c, err := buildCodec(st, nullNamespace, schema)
if err != nil {
return nil, err
}
c.schemaCanonical, err = parsingCanonicalForm(schema, "", make(map[string]string))
if err != nil {
return nil, err // should not get here because schema was validated above
}
c.Rabin = rabin([]byte(c.schemaCanonical))
c.soeHeader = []byte{0xC3, 0x01, 0, 0, 0, 0, 0, 0, 0, 0}
binary.LittleEndian.PutUint64(c.soeHeader[2:], c.Rabin)
c.schemaOriginal = schemaSpecification
return c, nil
}
func newSymbolTable() map[string]*Codec {
return map[string]*Codec{
"boolean": {
typeName: &name{"boolean", nullNamespace},
schemaOriginal: "boolean",
schemaCanonical: "boolean",
binaryFromNative: booleanBinaryFromNative,
nativeFromBinary: booleanNativeFromBinary,
nativeFromTextual: booleanNativeFromTextual,
textualFromNative: booleanTextualFromNative,
},
"bytes": {
typeName: &name{"bytes", nullNamespace},
schemaOriginal: "bytes",
schemaCanonical: "bytes",
binaryFromNative: bytesBinaryFromNative,
nativeFromBinary: bytesNativeFromBinary,
nativeFromTextual: bytesNativeFromTextual,
textualFromNative: bytesTextualFromNative,
},
"double": {
typeName: &name{"double", nullNamespace},
schemaOriginal: "double",
schemaCanonical: "double",
binaryFromNative: doubleBinaryFromNative,
nativeFromBinary: doubleNativeFromBinary,
nativeFromTextual: doubleNativeFromTextual,
textualFromNative: doubleTextualFromNative,
},
"float": {
typeName: &name{"float", nullNamespace},
schemaOriginal: "float",
schemaCanonical: "float",
binaryFromNative: floatBinaryFromNative,
nativeFromBinary: floatNativeFromBinary,
nativeFromTextual: floatNativeFromTextual,
textualFromNative: floatTextualFromNative,
},
"int": {
typeName: &name{"int", nullNamespace},
schemaOriginal: "int",
schemaCanonical: "int",
binaryFromNative: intBinaryFromNative,
nativeFromBinary: intNativeFromBinary,
nativeFromTextual: intNativeFromTextual,
textualFromNative: intTextualFromNative,
},
"long": {
typeName: &name{"long", nullNamespace},
schemaOriginal: "long",
schemaCanonical: "long",
binaryFromNative: longBinaryFromNative,
nativeFromBinary: longNativeFromBinary,
nativeFromTextual: longNativeFromTextual,
textualFromNative: longTextualFromNative,
},
"null": {
typeName: &name{"null", nullNamespace},
schemaOriginal: "null",
schemaCanonical: "null",
binaryFromNative: nullBinaryFromNative,
nativeFromBinary: nullNativeFromBinary,
nativeFromTextual: nullNativeFromTextual,
textualFromNative: nullTextualFromNative,
},
"string": {
typeName: &name{"string", nullNamespace},
schemaOriginal: "string",
schemaCanonical: "string",
binaryFromNative: stringBinaryFromNative,
nativeFromBinary: stringNativeFromBinary,
nativeFromTextual: stringNativeFromTextual,
textualFromNative: stringTextualFromNative,
},
// Start of compiled logical types using format typeName.logicalType where there is
// no dependence on schema.
"long.timestamp-millis": {
typeName: &name{"long.timestamp-millis", nullNamespace},
schemaOriginal: "long",
schemaCanonical: "long",
nativeFromTextual: nativeFromTimeStampMillis(longNativeFromTextual),
binaryFromNative: timeStampMillisFromNative(longBinaryFromNative),
nativeFromBinary: nativeFromTimeStampMillis(longNativeFromBinary),
textualFromNative: timeStampMillisFromNative(longTextualFromNative),
},
"long.timestamp-micros": {
typeName: &name{"long.timestamp-micros", nullNamespace},
schemaOriginal: "long",
schemaCanonical: "long",
nativeFromTextual: nativeFromTimeStampMicros(longNativeFromTextual),
binaryFromNative: timeStampMicrosFromNative(longBinaryFromNative),
nativeFromBinary: nativeFromTimeStampMicros(longNativeFromBinary),
textualFromNative: timeStampMicrosFromNative(longTextualFromNative),
},
"int.time-millis": {
typeName: &name{"int.time-millis", nullNamespace},
schemaOriginal: "int",
schemaCanonical: "int",
nativeFromTextual: nativeFromTimeMillis(intNativeFromTextual),
binaryFromNative: timeMillisFromNative(intBinaryFromNative),
nativeFromBinary: nativeFromTimeMillis(intNativeFromBinary),
textualFromNative: timeMillisFromNative(intTextualFromNative),
},
"long.time-micros": {
typeName: &name{"long.time-micros", nullNamespace},
schemaOriginal: "long",
schemaCanonical: "long",
nativeFromTextual: nativeFromTimeMicros(longNativeFromTextual),
binaryFromNative: timeMicrosFromNative(longBinaryFromNative),
nativeFromBinary: nativeFromTimeMicros(longNativeFromBinary),
textualFromNative: timeMicrosFromNative(longTextualFromNative),
},
"int.date": {
typeName: &name{"int.date", nullNamespace},
schemaOriginal: "int",
schemaCanonical: "int",
nativeFromTextual: nativeFromDate(intNativeFromTextual),
binaryFromNative: dateFromNative(intBinaryFromNative),
nativeFromBinary: nativeFromDate(intNativeFromBinary),
textualFromNative: dateFromNative(intTextualFromNative),
},
}
}
// BinaryFromNative appends the binary encoded byte slice representation of the
// provided native datum value to the provided byte slice in accordance with the
// Avro schema supplied when creating the Codec. It is supplied a byte slice to
// which to append the binary encoded data along with the actual data to encode.
// On success, it returns a new byte slice with the encoded bytes appended, and
// a nil error value. On error, it returns the original byte slice, and the
// error message.
//
// func ExampleBinaryFromNative() {
// codec, err := goavro.NewCodec(`
// {
// "type": "record",
// "name": "LongList",
// "fields" : [
// {"name": "next", "type": ["null", "LongList"], "default": null}
// ]
// }`)
// if err != nil {
// fmt.Println(err)
// }
//
// // Convert native Go form to binary Avro data
// binary, err := codec.BinaryFromNative(nil, map[string]interface{}{
// "next": map[string]interface{}{
// "LongList": map[string]interface{}{
// "next": map[string]interface{}{
// "LongList": map[string]interface{}{
// // NOTE: May omit fields when using default value
// },
// },
// },
// },
// })
// if err != nil {
// fmt.Println(err)
// }
//
// fmt.Printf("%#v", binary)
// // Output: []byte{0x2, 0x2, 0x0}
// }
func (c *Codec) BinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
newBuf, err := c.binaryFromNative(buf, datum)
if err != nil {
return buf, err // if error, return original byte slice
}
return newBuf, nil
}
// NativeFromBinary returns a native datum value from the binary encoded byte
// slice in accordance with the Avro schema supplied when creating the Codec. On
// success, it returns the decoded datum, a byte slice containing the remaining
// undecoded bytes, and a nil error value. On error, it returns nil for
// the datum value, the original byte slice, and the error message.
//
// func ExampleNativeFromBinary() {
// codec, err := goavro.NewCodec(`
// {
// "type": "record",
// "name": "LongList",
// "fields" : [
// {"name": "next", "type": ["null", "LongList"], "default": null}
// ]
// }`)
// if err != nil {
// fmt.Println(err)
// }
//
// // Convert native Go form to binary Avro data
// binary := []byte{0x2, 0x2, 0x0}
//
// native, _, err := codec.NativeFromBinary(binary)
// if err != nil {
// fmt.Println(err)
// }
//
// fmt.Printf("%v", native)
// // Output: map[next:map[LongList:map[next:map[LongList:map[next:<nil>]]]]]
// }
func (c *Codec) NativeFromBinary(buf []byte) (interface{}, []byte, error) {
value, newBuf, err := c.nativeFromBinary(buf)
if err != nil {
return nil, buf, err // if error, return original byte slice
}
return value, newBuf, nil
}
// NativeFromSingle converts Avro data from Single-Object-Encoded format from
// the provided byte slice to Go native data types in accordance with the Avro
// schema supplied when creating the Codec. On success, it returns the decoded
// datum, along with a new byte slice with the decoded bytes consumed, and a nil
// error value. On error, it returns nil for the datum value, the original byte
// slice, and the error message.
//
// func decode(codec *goavro.Codec, buf []byte) error {
// datum, _, err := codec.NativeFromSingle(buf)
// if err != nil {
// return err
// }
// _, err = fmt.Println(datum)
// return err
// }
func (c *Codec) NativeFromSingle(buf []byte) (interface{}, []byte, error) {
fingerprint, newBuf, err := FingerprintFromSOE(buf)
if err != nil {
return nil, buf, err
}
if !bytes.Equal(buf[:len(c.soeHeader)], c.soeHeader) {
return nil, buf, ErrWrongCodec(fingerprint)
}
value, newBuf, err := c.nativeFromBinary(newBuf)
if err != nil {
return nil, buf, err // if error, return original byte slice
}
return value, newBuf, nil
}
// NativeFromTextual converts Avro data in JSON text format from the provided byte
// slice to Go native data types in accordance with the Avro schema supplied
// when creating the Codec. On success, it returns the decoded datum, along with
// a new byte slice with the decoded bytes consumed, and a nil error value. On
// error, it returns nil for the datum value, the original byte slice, and the
// error message.
//
// func ExampleNativeFromTextual() {
// codec, err := goavro.NewCodec(`
// {
// "type": "record",
// "name": "LongList",
// "fields" : [
// {"name": "next", "type": ["null", "LongList"], "default": null}
// ]
// }`)
// if err != nil {
// fmt.Println(err)
// }
//
// // Convert native Go form to text Avro data
// text := []byte(`{"next":{"LongList":{"next":{"LongList":{"next":null}}}}}`)
//
// native, _, err := codec.NativeFromTextual(text)
// if err != nil {
// fmt.Println(err)
// }
//
// fmt.Printf("%v", native)
// // Output: map[next:map[LongList:map[next:map[LongList:map[next:<nil>]]]]]
// }
func (c *Codec) NativeFromTextual(buf []byte) (interface{}, []byte, error) {
value, newBuf, err := c.nativeFromTextual(buf)
if err != nil {
return nil, buf, err // if error, return original byte slice
}
return value, newBuf, nil
}
// SingleFromNative appends the single-object-encoding byte slice representation
// of the provided native datum value to the provided byte slice in accordance
// with the Avro schema supplied when creating the Codec. It is supplied a byte
// slice to which to append the header and binary encoded data, along with the
// actual data to encode. On success, it returns a new byte slice with the
// encoded bytes appended, and a nil error value. On error, it returns the
// original byte slice, and the error message.
//
// func ExampleSingleItemEncoding() {
// codec, err := goavro.NewCodec(`"int"`)
// if err != nil {
// fmt.Fprintf(os.Stderr, "%s\n", err)
// return
// }
//
// buf, err := codec.SingleFromNative(nil, 3)
// if err != nil {
// fmt.Fprintf(os.Stderr, "%s\n", err)
// return
// }
//
// fmt.Println(buf)
// // Output: [195 1 143 92 57 63 26 213 117 114 6]
// }
func (c *Codec) SingleFromNative(buf []byte, datum interface{}) ([]byte, error) {
newBuf, err := c.binaryFromNative(append(buf, c.soeHeader...), datum)
if err != nil {
return buf, err
}
return newBuf, nil
}
// TextualFromNative converts Go native data types to Avro data in JSON text format in
// accordance with the Avro schema supplied when creating the Codec. It is
// supplied a byte slice to which to append the encoded data and the actual data
// to encode. On success, it returns a new byte slice with the encoded bytes
// appended, and a nil error value. On error, it returns the original byte
// slice, and the error message.
//
// func ExampleTextualFromNative() {
// codec, err := goavro.NewCodec(`
// {
// "type": "record",
// "name": "LongList",
// "fields" : [
// {"name": "next", "type": ["null", "LongList"], "default": null}
// ]
// }`)
// if err != nil {
// fmt.Println(err)
// }
//
// // Convert native Go form to text Avro data
// text, err := codec.TextualFromNative(nil, map[string]interface{}{
// "next": map[string]interface{}{
// "LongList": map[string]interface{}{
// "next": map[string]interface{}{
// "LongList": map[string]interface{}{
// // NOTE: May omit fields when using default value
// },
// },
// },
// },
// })
// if err != nil {
// fmt.Println(err)
// }
//
// fmt.Printf("%s", text)
// // Output: {"next":{"LongList":{"next":{"LongList":{"next":null}}}}}
// }
func (c *Codec) TextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
newBuf, err := c.textualFromNative(buf, datum)
if err != nil {
return buf, err // if error, return original byte slice
}
return newBuf, nil
}
// Schema returns the original schema used to create the Codec.
func (c *Codec) Schema() string {
return c.schemaOriginal
}
// CanonicalSchema returns the Parsing Canonical Form of the schema according to
// the Avro specification.
func (c *Codec) CanonicalSchema() string {
return c.schemaCanonical
}
// SchemaCRC64Avro returns a signed 64-bit integer Rabin fingerprint for the
// canonical schema. This method returns the signed 64-bit cast of the unsigned
// 64-bit schema Rabin fingerprint.
//
// DEPRECATED: This method has been replaced by the Rabin structure Codec field
// and is provided for backward compatibility only.
func (c *Codec) SchemaCRC64Avro() int64 {
return int64(c.Rabin)
}
// convert a schema data structure to a codec, prefixing with specified
// namespace
func buildCodec(st map[string]*Codec, enclosingNamespace string, schema interface{}) (*Codec, error) {
switch schemaType := schema.(type) {
case map[string]interface{}:
return buildCodecForTypeDescribedByMap(st, enclosingNamespace, schemaType)
case string:
return buildCodecForTypeDescribedByString(st, enclosingNamespace, schemaType, nil)
case []interface{}:
return buildCodecForTypeDescribedBySlice(st, enclosingNamespace, schemaType)
default:
return nil, fmt.Errorf("unknown schema type: %T", schema)
}
}
// Reach into the map, grabbing its "type". Use that to create the codec.
func buildCodecForTypeDescribedByMap(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
t, ok := schemaMap["type"]
if !ok {
return nil, fmt.Errorf("missing type: %v", schemaMap)
}
switch v := t.(type) {
case string:
// Already defined types may be abbreviated with its string name.
// EXAMPLE: "type":"array"
// EXAMPLE: "type":"enum"
// EXAMPLE: "type":"fixed"
// EXAMPLE: "type":"int"
// EXAMPLE: "type":"record"
// EXAMPLE: "type":"somePreviouslyDefinedCustomTypeString"
return buildCodecForTypeDescribedByString(st, enclosingNamespace, v, schemaMap)
case map[string]interface{}:
return buildCodecForTypeDescribedByMap(st, enclosingNamespace, v)
case []interface{}:
return buildCodecForTypeDescribedBySlice(st, enclosingNamespace, v)
default:
return nil, fmt.Errorf("type ought to be either string, map[string]interface{}, or []interface{}; received: %T", t)
}
}
func buildCodecForTypeDescribedByString(st map[string]*Codec, enclosingNamespace string, typeName string, schemaMap map[string]interface{}) (*Codec, error) {
isLogicalType := false
searchType := typeName
// logicalType will be non-nil for those fields without a logicalType property set
if lt := schemaMap["logicalType"]; lt != nil {
isLogicalType = true
searchType = fmt.Sprintf("%s.%s", typeName, lt)
}
// NOTE: When codec already exists, return it. This includes both primitive and
// logicalType codecs added in NewCodec, and user-defined types, added while
// building the codec.
if cd, ok := st[searchType]; ok {
return cd, nil
}
// Avro specification allows abbreviation of type name inside a namespace.
if enclosingNamespace != "" {
if cd, ok := st[enclosingNamespace+"."+typeName]; ok {
return cd, nil
}
}
// There are only a small handful of complex Avro data types.
switch searchType {
case "array":
return makeArrayCodec(st, enclosingNamespace, schemaMap)
case "enum":
return makeEnumCodec(st, enclosingNamespace, schemaMap)
case "fixed":
return makeFixedCodec(st, enclosingNamespace, schemaMap)
case "map":
return makeMapCodec(st, enclosingNamespace, schemaMap)
case "record":
return makeRecordCodec(st, enclosingNamespace, schemaMap)
case "bytes.decimal":
return makeDecimalBytesCodec(st, enclosingNamespace, schemaMap)
case "fixed.decimal":
return makeDecimalFixedCodec(st, enclosingNamespace, schemaMap)
default:
if isLogicalType {
delete(schemaMap, "logicalType")
return buildCodecForTypeDescribedByString(st, enclosingNamespace, typeName, schemaMap)
}
return nil, fmt.Errorf("unknown type name: %q", searchType)
}
}
// notion of enclosing namespace changes when record, enum, or fixed create a
// new namespace, for child objects.
func registerNewCodec(st map[string]*Codec, schemaMap map[string]interface{}, enclosingNamespace string) (*Codec, error) {
n, err := newNameFromSchemaMap(enclosingNamespace, schemaMap)
if err != nil {
return nil, err
}
c := &Codec{typeName: n}
st[n.fullName] = c
return c, nil
}
// ErrWrongCodec is returned when an attempt is made to decode a single-object
// encoded value using the wrong codec.
type ErrWrongCodec uint64
func (e ErrWrongCodec) Error() string { return "wrong codec: " + strconv.FormatUint(uint64(e), 10) }
// ErrNotSingleObjectEncoded is returned when an attempt is made to decode a
// single-object encoded value from a buffer that does not have the correct
// magic prefix.
type ErrNotSingleObjectEncoded string
func (e ErrNotSingleObjectEncoded) Error() string {
return "cannot decode buffer as single-object encoding: " + string(e)
}
vendor/github.com/linkedin/goavro/v2/debug_development.go
Generated Vendored
+13
View File
@@ -0,0 +1,13 @@
// +build goavro_debug
package goavro
import (
"fmt"
"os"
)
// debug formats and prints arguments to stderr for development builds
func debug(f string, a ...interface{}) {
os.Stderr.Write([]byte("goavro: " + fmt.Sprintf(f, a...)))
}
vendor/github.com/linkedin/goavro/v2/debug_release.go
Generated Vendored
+7
View File
@@ -0,0 +1,7 @@
// +build !goavro_debug
package goavro
// debug is a no-op for release builds, and the function call is optimized out
// by the compiler.
func debug(_ string, _ ...interface{}) {}
vendor/github.com/linkedin/goavro/v2/doc.go
Generated Vendored
+68
View File
@@ -0,0 +1,68 @@
/*
Package goavro is a library that encodes and decodes Avro data.
Goavro provides methods to encode native Go data into both binary and textual
JSON Avro data, and methods to decode both binary and textual JSON Avro data to
native Go data.
Goavro also provides methods to read and write Object Container File (OCF)
formatted files, and the library contains example programs to read and write OCF
files.
Usage Example:
package main
import (
"fmt"
"github.com/linkedin/goavro"
)
func main() {
codec, err := goavro.NewCodec(`
{
"type": "record",
"name": "LongList",
"fields" : [
{"name": "next", "type": ["null", "LongList", {"type": "long", "logicalType": "timestamp-millis"}], "default": null}
]
}`)
if err != nil {
fmt.Println(err)
}
// NOTE: May omit fields when using default value
textual := []byte(`{"next":{"LongList":{}}}`)
// Convert textual Avro data (in Avro JSON format) to native Go form
native, _, err := codec.NativeFromTextual(textual)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to binary Avro data
binary, err := codec.BinaryFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// Convert binary Avro data back to native Go form
native, _, err = codec.NativeFromBinary(binary)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to textual Avro data
textual, err = codec.TextualFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// NOTE: Textual encoding will show all fields, even those with values that
// match their default values
fmt.Println(string(textual))
// Output: {"next":{"LongList":{"next":null}}}
}
*/
package goavro
vendor/github.com/linkedin/goavro/v2/enum.go
Generated Vendored
+105
View File
@@ -0,0 +1,105 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"io"
)
// enum does not have child objects, therefore whatever namespace it defines is
// just to store its name in the symbol table.
func makeEnumCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
c, err := registerNewCodec(st, schemaMap, enclosingNamespace)
if err != nil {
return nil, fmt.Errorf("Enum ought to have valid name: %s", err)
}
// enum type must have symbols
s1, ok := schemaMap["symbols"]
if !ok {
return nil, fmt.Errorf("Enum %q ought to have symbols key", c.typeName)
}
s2, ok := s1.([]interface{})
if !ok || len(s2) == 0 {
return nil, fmt.Errorf("Enum %q symbols ought to be non-empty array of strings: %v", c.typeName, s1)
}
symbols := make([]string, len(s2))
for i, s := range s2 {
symbol, ok := s.(string)
if !ok {
return nil, fmt.Errorf("Enum %q symbol %d ought to be non-empty string; received: %T", c.typeName, i+1, symbol)
}
if err := checkString(symbol); err != nil {
return nil, fmt.Errorf("Enum %q symbol %d ought to %s", c.typeName, i+1, err)
}
symbols[i] = symbol
}
c.nativeFromBinary = func(buf []byte) (interface{}, []byte, error) {
var value interface{}
var err error
var index int64
if value, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary enum %q index: %s", c.typeName, err)
}
index = value.(int64)
if index < 0 || index >= int64(len(symbols)) {
return nil, nil, fmt.Errorf("cannot decode binary enum %q: index ought to be between 0 and %d; read index: %d", c.typeName, len(symbols)-1, index)
}
return symbols[index], buf, nil
}
c.binaryFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
someString, ok := datum.(string)
if !ok {
return nil, fmt.Errorf("cannot encode binary enum %q: expected string; received: %T", c.typeName, datum)
}
for i, symbol := range symbols {
if symbol == someString {
return longBinaryFromNative(buf, i)
}
}
return nil, fmt.Errorf("cannot encode binary enum %q: value ought to be member of symbols: %v; %q", c.typeName, symbols, someString)
}
c.nativeFromTextual = func(buf []byte) (interface{}, []byte, error) {
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, fmt.Errorf("cannot decode textual enum: %s", io.ErrShortBuffer)
}
// decode enum string
var value interface{}
var err error
value, buf, err = stringNativeFromTextual(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual enum: expected key: %s", err)
}
someString := value.(string)
for _, symbol := range symbols {
if symbol == someString {
return someString, buf, nil
}
}
return nil, nil, fmt.Errorf("cannot decode textual enum %q: value ought to be member of symbols: %v; %q", c.typeName, symbols, someString)
}
c.textualFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
someString, ok := datum.(string)
if !ok {
return nil, fmt.Errorf("cannot encode textual enum %q: expected string; received: %T", c.typeName, datum)
}
for _, symbol := range symbols {
if symbol == someString {
return stringTextualFromNative(buf, someString)
}
}
return nil, fmt.Errorf("cannot encode textual enum %q: value ought to be member of symbols: %v; %q", c.typeName, symbols, someString)
}
return c, nil
}
vendor/github.com/linkedin/goavro/v2/fixed.go
Generated Vendored
+111
View File
@@ -0,0 +1,111 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"strconv"
)
// Fixed does not have child objects, therefore whatever namespace it defines is
// just to store its name in the symbol table.
func makeFixedCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
c, err := registerNewCodec(st, schemaMap, enclosingNamespace)
if err != nil {
return nil, fmt.Errorf("Fixed ought to have valid name: %s", err)
}
size, err := sizeFromSchemaMap(c.typeName, schemaMap)
if err != nil {
return nil, err
}
c.nativeFromBinary = func(buf []byte) (interface{}, []byte, error) {
if buflen := uint(len(buf)); size > buflen {
return nil, nil, fmt.Errorf("cannot decode binary fixed %q: schema size exceeds remaining buffer size: %d > %d (short buffer)", c.typeName, size, buflen)
}
return buf[:size], buf[size:], nil
}
c.binaryFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
var someBytes []byte
switch d := datum.(type) {
case []byte:
someBytes = d
case string:
someBytes = []byte(d)
default:
return nil, fmt.Errorf("cannot encode binary fixed %q: expected []byte or string; received: %T", c.typeName, datum)
}
if count := uint(len(someBytes)); count != size {
return nil, fmt.Errorf("cannot encode binary fixed %q: datum size ought to equal schema size: %d != %d", c.typeName, count, size)
}
return append(buf, someBytes...), nil
}
c.nativeFromTextual = func(buf []byte) (interface{}, []byte, error) {
if buflen := uint(len(buf)); size > buflen {
return nil, nil, fmt.Errorf("cannot decode textual fixed %q: schema size exceeds remaining buffer size: %d > %d (short buffer)", c.typeName, size, buflen)
}
var datum interface{}
var err error
datum, buf, err = bytesNativeFromTextual(buf)
if err != nil {
return nil, buf, err
}
datumBytes := datum.([]byte)
if count := uint(len(datumBytes)); count != size {
return nil, nil, fmt.Errorf("cannot decode textual fixed %q: datum size ought to equal schema size: %d != %d", c.typeName, count, size)
}
return datum, buf, err
}
c.textualFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
var someBytes []byte
switch d := datum.(type) {
case []byte:
someBytes = d
case string:
someBytes = []byte(d)
default:
return nil, fmt.Errorf("cannot encode textual fixed %q: expected []byte or string; received: %T", c.typeName, datum)
}
if count := uint(len(someBytes)); count != size {
return nil, fmt.Errorf("cannot encode textual fixed %q: datum size ought to equal schema size: %d != %d", c.typeName, count, size)
}
return bytesTextualFromNative(buf, someBytes)
}
return c, nil
}
func sizeFromSchemaMap(typeName *name, schemaMap map[string]interface{}) (uint, error) {
// Fixed type must have size
sizeRaw, ok := schemaMap["size"]
if !ok {
return 0, fmt.Errorf("Fixed %q ought to have size key", typeName)
}
var size uint
switch val := sizeRaw.(type) {
case string:
s, err := strconv.ParseUint(val, 10, 0)
if err != nil {
return 0, fmt.Errorf("Fixed %q size ought to be number greater than zero: %v", typeName, sizeRaw)
}
size = uint(s)
case float64:
if val <= 0 {
return 0, fmt.Errorf("Fixed %q size ought to be number greater than zero: %v", typeName, sizeRaw)
}
size = uint(val)
default:
return 0, fmt.Errorf("Fixed %q size ought to be number greater than zero: %v", typeName, sizeRaw)
}
return size, nil
}
vendor/github.com/linkedin/goavro/v2/floatingPoint.go
Generated Vendored
+296
View File
@@ -0,0 +1,296 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"encoding/binary"
"fmt"
"io"
"math"
"strconv"
)
const (
doubleEncodedLength = 8 // double requires 8 bytes
floatEncodedLength = 4 // float requires 4 bytes
)
////////////////////////////////////////
// Binary Decode
////////////////////////////////////////
func doubleNativeFromBinary(buf []byte) (interface{}, []byte, error) {
if len(buf) < doubleEncodedLength {
return nil, nil, fmt.Errorf("cannot decode binary double: %s", io.ErrShortBuffer)
}
return math.Float64frombits(binary.LittleEndian.Uint64(buf[:doubleEncodedLength])), buf[doubleEncodedLength:], nil
}
func floatNativeFromBinary(buf []byte) (interface{}, []byte, error) {
if len(buf) < floatEncodedLength {
return nil, nil, fmt.Errorf("cannot decode binary float: %s", io.ErrShortBuffer)
}
return math.Float32frombits(binary.LittleEndian.Uint32(buf[:floatEncodedLength])), buf[floatEncodedLength:], nil
}
////////////////////////////////////////
// Binary Encode
////////////////////////////////////////
func doubleBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var value float64
switch v := datum.(type) {
case float64:
value = v
case float32:
value = float64(v)
case int:
if value = float64(v); int(value) != v {
return nil, fmt.Errorf("cannot encode binary double: provided Go int would lose precision: %d", v)
}
case int64:
if value = float64(v); int64(value) != v {
return nil, fmt.Errorf("cannot encode binary double: provided Go int64 would lose precision: %d", v)
}
case int32:
if value = float64(v); int32(value) != v {
return nil, fmt.Errorf("cannot encode binary double: provided Go int32 would lose precision: %d", v)
}
default:
return nil, fmt.Errorf("cannot encode binary double: expected: Go numeric; received: %T", datum)
}
buf = append(buf, 0, 0, 0, 0, 0, 0, 0, 0)
binary.LittleEndian.PutUint64(buf[len(buf)-doubleEncodedLength:], math.Float64bits(value))
return buf, nil
}
func floatBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var value float32
switch v := datum.(type) {
case float32:
value = v
case float64:
// Assume runtime can cast special floats correctly, and if there is a
// loss of precision from float64 and float32, that should be expected
// or at least understood by the client.
value = float32(v)
case int:
if value = float32(v); int(value) != v {
return nil, fmt.Errorf("cannot encode binary float: provided Go int would lose precision: %d", v)
}
case int64:
if value = float32(v); int64(value) != v {
return nil, fmt.Errorf("cannot encode binary float: provided Go int64 would lose precision: %d", v)
}
case int32:
if value = float32(v); int32(value) != v {
return nil, fmt.Errorf("cannot encode binary float: provided Go int32 would lose precision: %d", v)
}
default:
return nil, fmt.Errorf("cannot encode binary float: expected: Go numeric; received: %T", datum)
}
// return floatingBinaryEncoder(buf, uint64(math.Float32bits(value)), floatEncodedLength)
buf = append(buf, 0, 0, 0, 0)
binary.LittleEndian.PutUint32(buf[len(buf)-floatEncodedLength:], uint32(math.Float32bits(value)))
return buf, nil
}
////////////////////////////////////////
// Text Decode
////////////////////////////////////////
func doubleNativeFromTextual(buf []byte) (interface{}, []byte, error) {
return floatingTextDecoder(buf, 64)
}
func floatNativeFromTextual(buf []byte) (interface{}, []byte, error) {
return floatingTextDecoder(buf, 32)
}
func floatingTextDecoder(buf []byte, bitSize int) (interface{}, []byte, error) {
buflen := len(buf)
if buflen >= 4 {
if bytes.Equal(buf[:4], []byte("null")) {
return math.NaN(), buf[4:], nil
}
if buflen >= 5 {
if bytes.Equal(buf[:5], []byte("1e999")) {
return math.Inf(1), buf[5:], nil
}
if buflen >= 6 {
if bytes.Equal(buf[:6], []byte("-1e999")) {
return math.Inf(-1), buf[6:], nil
}
}
}
}
index, err := numberLength(buf, true) // NOTE: floatAllowed = true
if err != nil {
return nil, nil, err
}
datum, err := strconv.ParseFloat(string(buf[:index]), bitSize)
if err != nil {
return nil, nil, err
}
if bitSize == 32 {
return float32(datum), buf[index:], nil
}
return datum, buf[index:], nil
}
func numberLength(buf []byte, floatAllowed bool) (int, error) {
// ALGORITHM: increment index as long as bytes are valid for number state engine.
var index, buflen, count int
var b byte
// STATE 0: begin, optional: -
if buflen = len(buf); index == buflen {
return 0, io.ErrShortBuffer
}
if buf[index] == '-' {
if index++; index == buflen {
return 0, io.ErrShortBuffer
}
}
// STATE 1: if 0, goto 2; otherwise if 1-9, goto 3; otherwise bail
if b = buf[index]; b == '0' {
if index++; index == buflen {
return index, nil // valid number
}
} else if b >= '1' && b <= '9' {
if index++; index == buflen {
return index, nil // valid number
}
// STATE 3: absorb zero or more digits
for {
if b = buf[index]; b < '0' || b > '9' {
break
}
if index++; index == buflen {
return index, nil // valid number
}
}
} else {
return 0, fmt.Errorf("unexpected byte: %q", b)
}
if floatAllowed {
// STATE 2: if ., goto 4; otherwise goto 5
if buf[index] == '.' {
if index++; index == buflen {
return 0, io.ErrShortBuffer
}
// STATE 4: absorb one or more digits
for {
if b = buf[index]; b < '0' || b > '9' {
break
}
count++
if index++; index == buflen {
return index, nil // valid number
}
}
if count == 0 {
// did not get at least one digit
return 0, fmt.Errorf("unexpected byte: %q", b)
}
}
// STATE 5: if e|e, goto 6; otherwise goto 7
if b = buf[index]; b == 'e' || b == 'E' {
if index++; index == buflen {
return 0, io.ErrShortBuffer
}
// STATE 6: if -|+, goto 8; otherwise goto 8
if b = buf[index]; b == '+' || b == '-' {
if index++; index == buflen {
return 0, io.ErrShortBuffer
}
}
// STATE 8: absorb one or more digits
count = 0
for {
if b = buf[index]; b < '0' || b > '9' {
break
}
count++
if index++; index == buflen {
return index, nil // valid number
}
}
if count == 0 {
// did not get at least one digit
return 0, fmt.Errorf("unexpected byte: %q", b)
}
}
}
// STATE 7: end
return index, nil
}
////////////////////////////////////////
// Text Encode
////////////////////////////////////////
func floatTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
return floatingTextEncoder(buf, datum, 32)
}
func doubleTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
return floatingTextEncoder(buf, datum, 64)
}
func floatingTextEncoder(buf []byte, datum interface{}, bitSize int) ([]byte, error) {
var isFloat bool
var someFloat64 float64
var someInt64 int64
switch v := datum.(type) {
case float32:
isFloat = true
someFloat64 = float64(v)
case float64:
isFloat = true
someFloat64 = v
case int:
if someInt64 = int64(v); int(someInt64) != v {
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual double: provided Go int would lose precision: %d", v)
}
return nil, fmt.Errorf("cannot encode textual float: provided Go int would lose precision: %d", v)
}
case int64:
someInt64 = v
case int32:
if someInt64 = int64(v); int32(someInt64) != v {
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual double: provided Go int32 would lose precision: %d", v)
}
return nil, fmt.Errorf("cannot encode textual float: provided Go int32 would lose precision: %d", v)
}
default:
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual double: expected: Go numeric; received: %T", datum)
}
return nil, fmt.Errorf("cannot encode textual float: expected: Go numeric; received: %T", datum)
}
if isFloat {
if math.IsNaN(someFloat64) {
return append(buf, "null"...), nil
}
if math.IsInf(someFloat64, 1) {
return append(buf, "1e999"...), nil
}
if math.IsInf(someFloat64, -1) {
return append(buf, "-1e999"...), nil
}
return strconv.AppendFloat(buf, someFloat64, 'g', -1, bitSize), nil
}
return strconv.AppendInt(buf, someInt64, 10), nil
}
vendor/github.com/linkedin/goavro/v2/go.mod
Generated Vendored
+5
View File
@@ -0,0 +1,5 @@
module github.com/linkedin/goavro/v2
go 1.12
require github.com/golang/snappy v0.0.1
vendor/github.com/linkedin/goavro/v2/go.sum
Generated Vendored
+2
View File
@@ -0,0 +1,2 @@
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
vendor/github.com/linkedin/goavro/v2/integer.go
Generated Vendored
+199
View File
@@ -0,0 +1,199 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"io"
"strconv"
)
const (
intDownShift = uint32(31)
intFlag = byte(128)
intMask = byte(127)
longDownShift = uint32(63)
)
////////////////////////////////////////
// Binary Decode
////////////////////////////////////////
func intNativeFromBinary(buf []byte) (interface{}, []byte, error) {
var offset, value int
var shift uint
for offset = 0; offset < len(buf); offset++ {
b := buf[offset]
value |= int(b&intMask) << shift
if b&intFlag == 0 {
return (int32(value>>1) ^ -int32(value&1)), buf[offset+1:], nil
}
shift += 7
}
return nil, nil, io.ErrShortBuffer
}
func longNativeFromBinary(buf []byte) (interface{}, []byte, error) {
var offset int
var value uint64
var shift uint
for offset = 0; offset < len(buf); offset++ {
b := buf[offset]
value |= uint64(b&intMask) << shift
if b&intFlag == 0 {
return (int64(value>>1) ^ -int64(value&1)), buf[offset+1:], nil
}
shift += 7
}
return nil, nil, io.ErrShortBuffer
}
////////////////////////////////////////
// Binary Encode
////////////////////////////////////////
func intBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var value int32
switch v := datum.(type) {
case int32:
value = v
case int:
if value = int32(v); int(value) != v {
return nil, fmt.Errorf("cannot encode binary int: provided Go int would lose precision: %d", v)
}
case int64:
if value = int32(v); int64(value) != v {
return nil, fmt.Errorf("cannot encode binary int: provided Go int64 would lose precision: %d", v)
}
case float64:
if value = int32(v); float64(value) != v {
return nil, fmt.Errorf("cannot encode binary int: provided Go float64 would lose precision: %f", v)
}
case float32:
if value = int32(v); float32(value) != v {
return nil, fmt.Errorf("cannot encode binary int: provided Go float32 would lose precision: %f", v)
}
default:
return nil, fmt.Errorf("cannot encode binary int: expected: Go numeric; received: %T", datum)
}
encoded := uint64((uint32(value) << 1) ^ uint32(value>>intDownShift))
return integerBinaryEncoder(buf, encoded)
}
func longBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
var value int64
switch v := datum.(type) {
case int64:
value = v
case int:
value = int64(v)
case int32:
value = int64(v)
case float64:
if value = int64(v); float64(value) != v {
return nil, fmt.Errorf("cannot encode binary long: provided Go float64 would lose precision: %f", v)
}
case float32:
if value = int64(v); float32(value) != v {
return nil, fmt.Errorf("cannot encode binary long: provided Go float32 would lose precision: %f", v)
}
default:
return nil, fmt.Errorf("long: expected: Go numeric; received: %T", datum)
}
encoded := (uint64(value) << 1) ^ uint64(value>>longDownShift)
return integerBinaryEncoder(buf, encoded)
}
func integerBinaryEncoder(buf []byte, encoded uint64) ([]byte, error) {
// used by both intBinaryEncoder and longBinaryEncoder
if encoded == 0 {
return append(buf, 0), nil
}
for encoded > 0 {
b := byte(encoded) & intMask
encoded = encoded >> 7
if encoded != 0 {
b |= intFlag // set high bit; we have more bytes
}
buf = append(buf, b)
}
return buf, nil
}
////////////////////////////////////////
// Text Decode
////////////////////////////////////////
func longNativeFromTextual(buf []byte) (interface{}, []byte, error) {
return integerTextDecoder(buf, 64)
}
func intNativeFromTextual(buf []byte) (interface{}, []byte, error) {
return integerTextDecoder(buf, 32)
}
func integerTextDecoder(buf []byte, bitSize int) (interface{}, []byte, error) {
index, err := numberLength(buf, false) // NOTE: floatAllowed = false
if err != nil {
return nil, nil, err
}
datum, err := strconv.ParseInt(string(buf[:index]), 10, bitSize)
if err != nil {
return nil, nil, err
}
if bitSize == 32 {
return int32(datum), buf[index:], nil
}
return datum, buf[index:], nil
}
////////////////////////////////////////
// Text Encode
////////////////////////////////////////
func longTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
return integerTextEncoder(buf, datum, 64)
}
func intTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
return integerTextEncoder(buf, datum, 32)
}
func integerTextEncoder(buf []byte, datum interface{}, bitSize int) ([]byte, error) {
var someInt64 int64
switch v := datum.(type) {
case int:
someInt64 = int64(v)
case int32:
someInt64 = int64(v)
case int64:
someInt64 = v
case float32:
if someInt64 = int64(v); float32(someInt64) != v {
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual long: provided Go float32 would lose precision: %f", v)
}
return nil, fmt.Errorf("cannot encode textual int: provided Go float32 would lose precision: %f", v)
}
case float64:
if someInt64 = int64(v); float64(someInt64) != v {
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual long: provided Go float64 would lose precision: %f", v)
}
return nil, fmt.Errorf("cannot encode textual int: provided Go float64 would lose precision: %f", v)
}
default:
if bitSize == 64 {
return nil, fmt.Errorf("cannot encode textual long: expected: Go numeric; received: %T", datum)
}
return nil, fmt.Errorf("cannot encode textual int: expected: Go numeric; received: %T", datum)
}
return strconv.AppendInt(buf, someInt64, 10), nil
}
vendor/github.com/linkedin/goavro/v2/logical_type.go
Generated Vendored
+375
View File
@@ -0,0 +1,375 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"errors"
"fmt"
"math"
"math/big"
"time"
)
type toNativeFn func([]byte) (interface{}, []byte, error)
type fromNativeFn func([]byte, interface{}) ([]byte, error)
//////////////////////////////////////////////////////////////////////////////////////////////
// date logical type - to/from time.Time, time.UTC location
//////////////////////////////////////////////////////////////////////////////////////////////
func nativeFromDate(fn toNativeFn) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
l, b, err := fn(bytes)
if err != nil {
return l, b, err
}
i, ok := l.(int32)
if !ok {
return l, b, fmt.Errorf("cannot transform to native date, expected int, received %T", l)
}
t := time.Date(1970, 1, 1, 0, 0, 0, 0, time.UTC).AddDate(0, 0, int(i)).UTC()
return t, b, nil
}
}
func dateFromNative(fn fromNativeFn) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
t, ok := d.(time.Time)
if !ok {
return nil, fmt.Errorf("cannot transform to binary date, expected time.Time, received %T", d)
}
// The number of days calculation is incredibly naive we take the time.Duration
// between the given time and unix epoch and divide that by (24 * time.Hour)
// This accuracy seems acceptable given the relation to unix epoch for now
// TODO: replace with a better method
numDays := t.UnixNano() / int64(24*time.Hour)
return fn(b, numDays)
}
}
//////////////////////////////////////////////////////////////////////////////////////////////
// time-millis logical type - to/from time.Time, time.UTC location
//////////////////////////////////////////////////////////////////////////////////////////////
func nativeFromTimeMillis(fn toNativeFn) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
l, b, err := fn(bytes)
if err != nil {
return l, b, err
}
i, ok := l.(int32)
if !ok {
return l, b, fmt.Errorf("cannot transform to native time.Duration, expected int, received %T", l)
}
t := time.Duration(i) * time.Millisecond
return t, b, nil
}
}
func timeMillisFromNative(fn fromNativeFn) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
t, ok := d.(time.Duration)
if !ok {
return nil, fmt.Errorf("cannot transform to binary time-millis, expected time.Duration, received %T", d)
}
duration := int32(t.Nanoseconds() / int64(time.Millisecond))
return fn(b, duration)
}
}
//////////////////////////////////////////////////////////////////////////////////////////////
// time-micros logical type - to/from time.Time, time.UTC location
//////////////////////////////////////////////////////////////////////////////////////////////
func nativeFromTimeMicros(fn toNativeFn) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
l, b, err := fn(bytes)
if err != nil {
return l, b, err
}
i, ok := l.(int64)
if !ok {
return l, b, fmt.Errorf("cannot transform to native time.Duration, expected long, received %T", l)
}
t := time.Duration(i) * time.Microsecond
return t, b, nil
}
}
func timeMicrosFromNative(fn fromNativeFn) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
t, ok := d.(time.Duration)
if !ok {
return nil, fmt.Errorf("cannot transform to binary time-micros, expected time.Duration, received %T", d)
}
duration := t.Nanoseconds() / int64(time.Microsecond)
return fn(b, duration)
}
}
//////////////////////////////////////////////////////////////////////////////////////////////
// timestamp-millis logical type - to/from time.Time, time.UTC location
//////////////////////////////////////////////////////////////////////////////////////////////
func nativeFromTimeStampMillis(fn toNativeFn) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
l, b, err := fn(bytes)
if err != nil {
return l, b, err
}
i, ok := l.(int64)
if !ok {
return l, b, fmt.Errorf("cannot transform native timestamp-millis, expected int64, received %T", l)
}
secs := i / int64(time.Microsecond)
nanosecs := (i - secs*int64(time.Microsecond)) * int64(time.Millisecond)
return time.Unix(secs, nanosecs).UTC(), b, nil
}
}
func timeStampMillisFromNative(fn fromNativeFn) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
t, ok := d.(time.Time)
if !ok {
return nil, fmt.Errorf("cannot transform binary timestamp-millis, expected time.Time, received %T", d)
}
millisecs := t.UnixNano() / int64(time.Millisecond)
return fn(b, millisecs)
}
}
//////////////////////////////////////////////////////////////////////////////////////////////
// timestamp-micros logical type - to/from time.Time, time.UTC location
//////////////////////////////////////////////////////////////////////////////////////////////
func nativeFromTimeStampMicros(fn toNativeFn) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
l, b, err := fn(bytes)
if err != nil {
return l, b, err
}
microseconds, ok := l.(int64)
if !ok {
return l, b, fmt.Errorf("cannot transform native timestamp-micros, expected int64, received %T", l)
}
// While this code performs a few more steps than seem required, it is
// written this way to allow the best time resolution on UNIX and
// Windows without overflowing the int64 value. Windows has a zero-time
// value of 1601-01-01 UTC, and the number of nanoseconds since that
// zero-time overflows 64-bit integers.
seconds := microseconds / 1e6
nanoseconds := (microseconds - (seconds * 1e6)) * 1e3
return time.Unix(seconds, nanoseconds).UTC(), b, nil
}
}
func timeStampMicrosFromNative(fn fromNativeFn) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
t, ok := d.(time.Time)
if !ok {
return nil, fmt.Errorf("cannot transform binary timestamp-micros, expected time.Time, received %T", d)
}
// While this code performs a few more steps than seem required, it is
// written this way to allow the best time resolution on UNIX and
// Windows without overflowing the int64 value. Windows has a zero-time
// value of 1601-01-01 UTC, and the number of nanoseconds since that
// zero-time overflows 64-bit integers.
return fn(b, t.Unix()*1e6+int64(t.Nanosecond()/1e3))
}
}
/////////////////////////////////////////////////////////////////////////////////////////////
// decimal logical-type - byte/fixed - to/from math/big.Rat
// two's complement algorithm taken from:
// https://groups.google.com/d/msg/golang-nuts/TV4bRVrHZUw/UcQt7S4IYlcJ by rog
/////////////////////////////////////////////////////////////////////////////////////////////
type makeCodecFn func(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error)
func precisionAndScaleFromSchemaMap(schemaMap map[string]interface{}) (int, int, error) {
p1, ok := schemaMap["precision"]
if !ok {
return 0, 0, errors.New("cannot create decimal logical type without precision")
}
p2, ok := p1.(float64)
if !ok {
return 0, 0, fmt.Errorf("cannot create decimal logical type with wrong precision type; expected: float64; received: %T", p1)
}
p3 := int(p2)
if p3 <= 1 {
return 0, 0, fmt.Errorf("cannot create decimal logical type when precision is less than one: %d", p3)
}
var s3 int // scale defaults to 0 if not set
if s1, ok := schemaMap["scale"]; ok {
s2, ok := s1.(float64)
if !ok {
return 0, 0, fmt.Errorf("cannot create decimal logical type with wrong precision type; expected: float64; received: %T", p1)
}
s3 = int(s2)
if s3 < 0 {
return 0, 0, fmt.Errorf("cannot create decimal logical type when scale is less than zero: %d", s3)
}
if s3 > p3 {
return 0, 0, fmt.Errorf("cannot create decimal logical type when scale is larger than precision: %d > %d", s3, p3)
}
}
return p3, s3, nil
}
var one = big.NewInt(1)
func makeDecimalBytesCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
precision, scale, err := precisionAndScaleFromSchemaMap(schemaMap)
if err != nil {
return nil, err
}
if _, ok := schemaMap["name"]; !ok {
schemaMap["name"] = "bytes.decimal"
}
c, err := registerNewCodec(st, schemaMap, enclosingNamespace)
if err != nil {
return nil, fmt.Errorf("Bytes ought to have valid name: %s", err)
}
c.binaryFromNative = decimalBytesFromNative(bytesBinaryFromNative, toSignedBytes, precision, scale)
c.textualFromNative = decimalBytesFromNative(bytesTextualFromNative, toSignedBytes, precision, scale)
c.nativeFromBinary = nativeFromDecimalBytes(bytesNativeFromBinary, precision, scale)
c.nativeFromTextual = nativeFromDecimalBytes(bytesNativeFromTextual, precision, scale)
return c, nil
}
func nativeFromDecimalBytes(fn toNativeFn, precision, scale int) toNativeFn {
return func(bytes []byte) (interface{}, []byte, error) {
d, b, err := fn(bytes)
if err != nil {
return d, b, err
}
bs, ok := d.([]byte)
if !ok {
return nil, bytes, fmt.Errorf("cannot transform to native decimal, expected []byte, received %T", d)
}
i := big.NewInt(0)
fromSignedBytes(i, bs)
if i.BitLen() > 64 {
// Avro spec specifies we return underlying type if the logicalType is invalid
return d, b, err
}
r := big.NewRat(i.Int64(), int64(math.Pow10(scale)))
return r, b, nil
}
}
func decimalBytesFromNative(fromNativeFn fromNativeFn, toBytesFn toBytesFn, precision, scale int) fromNativeFn {
return func(b []byte, d interface{}) ([]byte, error) {
r, ok := d.(*big.Rat)
if !ok {
return nil, fmt.Errorf("cannot transform to bytes, expected *big.Rat, received %T", d)
}
// we reduce accuracy to precision by dividing and multiplying by digit length
num := big.NewInt(0).Set(r.Num())
denom := big.NewInt(0).Set(r.Denom())
// we get the scaled decimal representation
i := new(big.Int).Mul(num, big.NewInt(int64(math.Pow10(scale))))
// divide that by the denominator
precnum := new(big.Int).Div(i, denom)
bout, err := toBytesFn(precnum)
if err != nil {
return nil, err
}
return fromNativeFn(b, bout)
}
}
func makeDecimalFixedCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
precision, scale, err := precisionAndScaleFromSchemaMap(schemaMap)
if err != nil {
return nil, err
}
if _, ok := schemaMap["name"]; !ok {
schemaMap["name"] = "fixed.decimal"
}
c, err := makeFixedCodec(st, enclosingNamespace, schemaMap)
if err != nil {
return nil, err
}
size, err := sizeFromSchemaMap(c.typeName, schemaMap)
if err != nil {
return nil, err
}
c.binaryFromNative = decimalBytesFromNative(c.binaryFromNative, toSignedFixedBytes(size), precision, scale)
c.textualFromNative = decimalBytesFromNative(c.textualFromNative, toSignedFixedBytes(size), precision, scale)
c.nativeFromBinary = nativeFromDecimalBytes(c.nativeFromBinary, precision, scale)
c.nativeFromTextual = nativeFromDecimalBytes(c.nativeFromTextual, precision, scale)
return c, nil
}
func padBytes(bytes []byte, fixedSize uint) []byte {
s := int(fixedSize)
padded := make([]byte, s, s)
if s >= len(bytes) {
copy(padded[s-len(bytes):], bytes)
}
return padded
}
type toBytesFn func(n *big.Int) ([]byte, error)
// fromSignedBytes sets the value of n to the big-endian two's complement
// value stored in the given data. If data[0]&80 != 0, the number
// is negative. If data is empty, the result will be 0.
func fromSignedBytes(n *big.Int, data []byte) {
n.SetBytes(data)
if len(data) > 0 && data[0]&0x80 > 0 {
n.Sub(n, new(big.Int).Lsh(one, uint(len(data))*8))
}
}
// toSignedBytes returns the big-endian two's complement
// form of n.
func toSignedBytes(n *big.Int) ([]byte, error) {
switch n.Sign() {
case 0:
return []byte{0}, nil
case 1:
b := n.Bytes()
if b[0]&0x80 > 0 {
b = append([]byte{0}, b...)
}
return b, nil
case -1:
length := uint(n.BitLen()/8+1) * 8
b := new(big.Int).Add(n, new(big.Int).Lsh(one, length)).Bytes()
// When the most significant bit is on a byte
// boundary, we can get some extra significant
// bits, so strip them off when that happens.
if len(b) >= 2 && b[0] == 0xff && b[1]&0x80 != 0 {
b = b[1:]
}
return b, nil
}
return nil, fmt.Errorf("toSignedBytes: error big.Int.Sign() returned unexpected value")
}
// toSignedFixedBytes returns the big-endian two's complement
// form of n for a given length of bytes.
func toSignedFixedBytes(size uint) func(*big.Int) ([]byte, error) {
return func(n *big.Int) ([]byte, error) {
switch n.Sign() {
case 0:
return []byte{0}, nil
case 1:
b := n.Bytes()
if b[0]&0x80 > 0 {
b = append([]byte{0}, b...)
}
return padBytes(b, size), nil
case -1:
length := size * 8
b := new(big.Int).Add(n, new(big.Int).Lsh(one, length)).Bytes()
// Unlike a variable length byte length we need the extra bits to meet byte length
return b, nil
}
return nil, fmt.Errorf("toSignedBytes: error big.Int.Sign() returned unexpected value")
}
}
vendor/github.com/linkedin/goavro/v2/map.go
Generated Vendored
+307
View File
@@ -0,0 +1,307 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"errors"
"fmt"
"io"
"math"
"reflect"
)
func makeMapCodec(st map[string]*Codec, namespace string, schemaMap map[string]interface{}) (*Codec, error) {
// map type must have values
valueSchema, ok := schemaMap["values"]
if !ok {
return nil, errors.New("Map ought to have values key")
}
valueCodec, err := buildCodec(st, namespace, valueSchema)
if err != nil {
return nil, fmt.Errorf("Map values ought to be valid Avro type: %s", err)
}
return &Codec{
typeName: &name{"map", nullNamespace},
nativeFromBinary: func(buf []byte) (interface{}, []byte, error) {
var err error
var value interface{}
// block count and block size
if value, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map block count: %s", err)
}
blockCount := value.(int64)
if blockCount < 0 {
// NOTE: A negative block count implies there is a long encoded
// block size following the negative block count. We have no use
// for the block size in this decoder, so we read and discard
// the value.
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can
// never be made positive
return nil, nil, fmt.Errorf("cannot decode binary map with block count: %d", blockCount)
}
blockCount = -blockCount // convert to its positive equivalent
if _, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, nil, fmt.Errorf("cannot decode binary map when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
// NOTE: While the attempt of a RAM optimization shown below is not
// necessary, many encoders will encode all items in a single block.
// We can optimize amount of RAM allocated by runtime for the array
// by initializing the array for that number of items.
mapValues := make(map[string]interface{}, blockCount)
for blockCount != 0 {
// Decode `blockCount` datum values from buffer
for i := int64(0); i < blockCount; i++ {
// first decode the key string
if value, buf, err = stringNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map key: %s", err)
}
key := value.(string) // string decoder always returns a string
if _, ok := mapValues[key]; ok {
return nil, nil, fmt.Errorf("cannot decode binary map: duplicate key: %q", key)
}
// then decode the value
if value, buf, err = valueCodec.nativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map value for key %q: %s", key, err)
}
mapValues[key] = value
}
// Decode next blockCount from buffer, because there may be more blocks
if value, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map block count: %s", err)
}
blockCount = value.(int64)
if blockCount < 0 {
// NOTE: A negative block count implies there is a long
// encoded block size following the negative block count. We
// have no use for the block size in this decoder, so we
// read and discard the value.
if blockCount == math.MinInt64 {
// The minimum number for any signed numerical type can
// never be made positive
return nil, nil, fmt.Errorf("cannot decode binary map with block count: %d", blockCount)
}
blockCount = -blockCount // convert to its positive equivalent
if _, buf, err = longNativeFromBinary(buf); err != nil {
return nil, nil, fmt.Errorf("cannot decode binary map block size: %s", err)
}
}
// Ensure block count does not exceed some sane value.
if blockCount > MaxBlockCount {
return nil, nil, fmt.Errorf("cannot decode binary map when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
}
return mapValues, buf, nil
},
binaryFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
mapValues, err := convertMap(datum)
if err != nil {
return nil, fmt.Errorf("cannot encode binary map: %s", err)
}
keyCount := int64(len(mapValues))
var alreadyEncoded, remainingInBlock int64
for k, v := range mapValues {
if remainingInBlock == 0 { // start a new block
remainingInBlock = keyCount - alreadyEncoded
if remainingInBlock > MaxBlockCount {
// limit block count to MacBlockCount
remainingInBlock = MaxBlockCount
}
buf, _ = longBinaryFromNative(buf, remainingInBlock)
}
// only fails when given non string, so elide error checking
buf, _ = stringBinaryFromNative(buf, k)
// encode the value
if buf, err = valueCodec.binaryFromNative(buf, v); err != nil {
return nil, fmt.Errorf("cannot encode binary map value for key %q: %v: %s", k, v, err)
}
remainingInBlock--
alreadyEncoded++
}
return longBinaryFromNative(buf, 0) // append tailing 0 block count to signal end of Map
},
nativeFromTextual: func(buf []byte) (interface{}, []byte, error) {
return genericMapTextDecoder(buf, valueCodec, nil) // codecFromKey == nil
},
textualFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
return genericMapTextEncoder(buf, datum, valueCodec, nil)
},
}, nil
}
// genericMapTextDecoder decodes a JSON text blob to a native Go map, using the
// codecs from codecFromKey, and if a key is not found in that map, from
// defaultCodec if provided. If defaultCodec is nil, this function returns an
// error if it encounters a map key that is not present in codecFromKey. If
// codecFromKey is nil, every map value will be decoded using defaultCodec, if
// possible.
func genericMapTextDecoder(buf []byte, defaultCodec *Codec, codecFromKey map[string]*Codec) (map[string]interface{}, []byte, error) {
var value interface{}
var err error
var b byte
lencodec := len(codecFromKey)
mapValues := make(map[string]interface{}, lencodec)
if buf, err = advanceAndConsume(buf, '{'); err != nil {
return nil, nil, err
}
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, io.ErrShortBuffer
}
// NOTE: Special case empty map
if buf[0] == '}' {
return mapValues, buf[1:], nil
}
// NOTE: Also terminates when read '}' byte.
for len(buf) > 0 {
// decode key string
value, buf, err = stringNativeFromTextual(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual map: expected key: %s", err)
}
key := value.(string)
// Is key already used?
if _, ok := mapValues[key]; ok {
return nil, nil, fmt.Errorf("cannot decode textual map: duplicate key: %q", key)
}
// Find a codec for the key
fieldCodec := codecFromKey[key]
if fieldCodec == nil {
fieldCodec = defaultCodec
}
if fieldCodec == nil {
return nil, nil, fmt.Errorf("cannot decode textual map: cannot determine codec: %q", key)
}
// decode colon
if buf, err = advanceAndConsume(buf, ':'); err != nil {
return nil, nil, err
}
// decode value
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, io.ErrShortBuffer
}
value, buf, err = fieldCodec.nativeFromTextual(buf)
if err != nil {
return nil, nil, err
}
// set map value for key
mapValues[key] = value
// either comma or closing curly brace
if buf, _ = advanceToNonWhitespace(buf); len(buf) == 0 {
return nil, nil, io.ErrShortBuffer
}
switch b = buf[0]; b {
case '}':
return mapValues, buf[1:], nil
case ',':
// no-op
default:
return nil, nil, fmt.Errorf("cannot decode textual map: expected ',' or '}'; received: %q", b)
}
// NOTE: consume comma from above
if buf, _ = advanceToNonWhitespace(buf[1:]); len(buf) == 0 {
return nil, nil, io.ErrShortBuffer
}
}
return nil, nil, io.ErrShortBuffer
}
// genericMapTextEncoder encodes a native Go map to a JSON text blob, using the
// codecs from codecFromKey, and if a key is not found in that map, from
// defaultCodec if provided. If defaultCodec is nil, this function returns an
// error if it encounters a map key that is not present in codecFromKey. If
// codecFromKey is nil, every map value will be encoded using defaultCodec, if
// possible.
func genericMapTextEncoder(buf []byte, datum interface{}, defaultCodec *Codec, codecFromKey map[string]*Codec) ([]byte, error) {
mapValues, err := convertMap(datum)
if err != nil {
return nil, fmt.Errorf("cannot encode textual map: %s", err)
}
var atLeastOne bool
buf = append(buf, '{')
for key, value := range mapValues {
atLeastOne = true
// Find a codec for the key
fieldCodec := codecFromKey[key]
if fieldCodec == nil {
fieldCodec = defaultCodec
}
if fieldCodec == nil {
return nil, fmt.Errorf("cannot encode textual map: cannot determine codec: %q", key)
}
// Encode key string
buf, err = stringTextualFromNative(buf, key)
if err != nil {
return nil, err
}
buf = append(buf, ':')
// Encode value
buf, err = fieldCodec.textualFromNative(buf, value)
if err != nil {
// field was specified in datum; therefore its value was invalid
return nil, fmt.Errorf("cannot encode textual map: value for %q does not match its schema: %s", key, err)
}
buf = append(buf, ',')
}
if atLeastOne {
return append(buf[:len(buf)-1], '}'), nil
}
return append(buf, '}'), nil
}
// convertMap converts datum to map[string]interface{} if possible.
func convertMap(datum interface{}) (map[string]interface{}, error) {
mapValues, ok := datum.(map[string]interface{})
if ok {
return mapValues, nil
}
// NOTE: When given a map of any other type, zip values to items as a
// convenience to client.
v := reflect.ValueOf(datum)
if v.Kind() != reflect.Map {
return nil, fmt.Errorf("cannot create map[string]interface{}: expected map[string]...; received: %T", datum)
}
// NOTE: Two better alternatives to the current algorithm are:
// (1) mutate the reflection tuple underneath to convert the
// map[string]int, for example, to map[string]interface{}, with
// O(1) complexity.
// (2) use copy builtin to zip the data items over with O(n) complexity,
// but more efficient than what's below.
mapValues = make(map[string]interface{}, v.Len())
for _, key := range v.MapKeys() {
k, ok := key.Interface().(string)
if !ok {
// bail when map key type is not string
return nil, fmt.Errorf("cannot create map[string]interface{}: expected map[string]...; received: %T", datum)
}
mapValues[string(k)] = v.MapIndex(key).Interface()
}
return mapValues, nil
}
vendor/github.com/linkedin/goavro/v2/name.go
Generated Vendored
+142
View File
@@ -0,0 +1,142 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"errors"
"fmt"
"strings"
)
const nullNamespace = ""
// ErrInvalidName is the error returned when one or more parts of an Avro name
// is invalid.
type ErrInvalidName struct {
Message string
}
func (e ErrInvalidName) Error() string {
return "schema name ought to " + e.Message
}
// NOTE: This function designed to work with name components, after they have
// been split on the period rune.
func isRuneInvalidForFirstCharacter(r rune) bool {
return (r < 'A' || r > 'Z') && (r < 'a' || r > 'z') && r != '_'
}
func isRuneInvalidForOtherCharacters(r rune) bool {
return isRuneInvalidForFirstCharacter(r) && (r < '0' || r > '9')
}
func checkNameComponent(s string) error {
err := checkString(s)
if err != nil {
return &ErrInvalidName{err.Error()}
}
return err
}
func checkString(s string) error {
if len(s) == 0 {
return errors.New("be non-empty string")
}
if strings.IndexFunc(s[:1], isRuneInvalidForFirstCharacter) != -1 {
return errors.New("start with [A-Za-z_]: " + s)
}
if strings.IndexFunc(s[1:], isRuneInvalidForOtherCharacters) != -1 {
return errors.New("have second and remaining characters contain only [A-Za-z0-9_]: " + s)
}
return nil
}
// name describes an Avro name in terms of its full name and namespace.
type name struct {
fullName string // the instance's Avro name
namespace string // for use when building new name from existing one
}
// newName returns a new Name instance after first ensuring the arguments do not
// violate any of the Avro naming rules.
func newName(n, ns, ens string) (*name, error) {
var nn name
if index := strings.LastIndexByte(n, '.'); index > -1 {
// inputName does contain a dot, so ignore everything else and use it as the full name
nn.fullName = n
nn.namespace = n[:index]
} else {
// inputName does not contain a dot, therefore is not the full name
if ns != nullNamespace {
// if namespace provided in the schema in the same schema level, use it
nn.fullName = ns + "." + n
nn.namespace = ns
} else if ens != nullNamespace {
// otherwise if enclosing namespace provided, use it
nn.fullName = ens + "." + n
nn.namespace = ens
} else {
// otherwise no namespace, so use null namespace, the empty string
nn.fullName = n
}
}
// verify all components of the full name for adherence to Avro naming rules
for i, component := range strings.Split(nn.fullName, ".") {
if i == 0 && RelaxedNameValidation && component == "" {
continue
}
if err := checkNameComponent(component); err != nil {
return nil, err
}
}
return &nn, nil
}
var (
// RelaxedNameValidation causes name validation to allow the first component
// of an Avro namespace to be the empty string.
RelaxedNameValidation bool
)
func newNameFromSchemaMap(enclosingNamespace string, schemaMap map[string]interface{}) (*name, error) {
var nameString, namespaceString string
name, ok := schemaMap["name"]
if !ok {
return nil, errors.New("schema ought to have name key")
}
nameString, ok = name.(string)
if !ok || nameString == nullNamespace {
return nil, fmt.Errorf("schema name ought to be non-empty string; received: %T: %v", name, name)
}
if namespace, ok := schemaMap["namespace"]; ok {
namespaceString, ok = namespace.(string)
if !ok {
return nil, fmt.Errorf("schema namespace, if provided, ought to be a string; received: %T: %v", namespace, namespace)
}
}
return newName(nameString, namespaceString, enclosingNamespace)
}
func (n *name) String() string {
return n.fullName
}
// short returns the name without the prefixed namespace.
func (n *name) short() string {
if index := strings.LastIndexByte(n.fullName, '.'); index > -1 {
return n.fullName[index+1:]
}
return n.fullName
}
vendor/github.com/linkedin/goavro/v2/null.go
Generated Vendored
+45
View File
@@ -0,0 +1,45 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"errors"
"fmt"
"io"
)
var nullBytes = []byte("null")
func nullNativeFromBinary(buf []byte) (interface{}, []byte, error) { return nil, buf, nil }
func nullBinaryFromNative(buf []byte, datum interface{}) ([]byte, error) {
if datum != nil {
return nil, fmt.Errorf("cannot encode binary null: expected: Go nil; received: %T", datum)
}
return buf, nil
}
func nullNativeFromTextual(buf []byte) (interface{}, []byte, error) {
if len(buf) < 4 {
return nil, nil, fmt.Errorf("cannot decode textual null: %s", io.ErrShortBuffer)
}
if bytes.Equal(buf[:4], nullBytes) {
return nil, buf[4:], nil
}
return nil, nil, errors.New("cannot decode textual null: expected: null")
}
func nullTextualFromNative(buf []byte, datum interface{}) ([]byte, error) {
if datum != nil {
return nil, fmt.Errorf("cannot encode textual null: expected: Go nil; received: %T", datum)
}
return append(buf, nullBytes...), nil
}
vendor/github.com/linkedin/goavro/v2/ocf.go
Generated Vendored
+240
View File
@@ -0,0 +1,240 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"crypto/rand"
"errors"
"fmt"
"io"
)
const (
// CompressionNullLabel is used when OCF blocks are not compressed.
CompressionNullLabel = "null"
// CompressionDeflateLabel is used when OCF blocks are compressed using the
// deflate algorithm.
CompressionDeflateLabel = "deflate"
// CompressionSnappyLabel is used when OCF blocks are compressed using the
// snappy algorithm.
CompressionSnappyLabel = "snappy"
)
// compressionID are values used to specify compression algorithm used to compress
// and decompress Avro Object Container File (OCF) streams.
type compressionID uint8
const (
compressionNull compressionID = iota
compressionDeflate
compressionSnappy
)
const (
ocfBlockConst = 24 // Each OCF block has two longs prefix, and sync marker suffix
ocfHeaderSizeConst = 48 // OCF header is usually about 48 bytes longer than its compressed schema
ocfMagicString = "Obj\x01"
ocfMetadataSchema = `{"type":"map","values":"bytes"}`
ocfSyncLength = 16
)
var (
ocfMagicBytes = []byte(ocfMagicString)
ocfMetadataCodec *Codec
)
func init() {
ocfMetadataCodec, _ = NewCodec(ocfMetadataSchema)
}
type ocfHeader struct {
codec *Codec
compressionID compressionID
syncMarker [ocfSyncLength]byte
metadata map[string][]byte
}
func newOCFHeader(config OCFConfig) (*ocfHeader, error) {
var err error
header := new(ocfHeader)
//
// avro.codec
//
switch config.CompressionName {
case "":
header.compressionID = compressionNull
case CompressionNullLabel:
header.compressionID = compressionNull
case CompressionDeflateLabel:
header.compressionID = compressionDeflate
case CompressionSnappyLabel:
header.compressionID = compressionSnappy
default:
return nil, fmt.Errorf("cannot create OCF header using unrecognized compression algorithm: %q", config.CompressionName)
}
//
// avro.schema
//
if config.Codec != nil {
header.codec = config.Codec
} else if config.Schema == "" {
return nil, fmt.Errorf("cannot create OCF header without either Codec or Schema specified")
} else {
if header.codec, err = NewCodec(config.Schema); err != nil {
return nil, fmt.Errorf("cannot create OCF header: %s", err)
}
}
header.metadata = config.MetaData
//
// The 16-byte, randomly-generated sync marker for this file.
//
_, err = rand.Read(header.syncMarker[:])
if err != nil {
return nil, err
}
return header, nil
}
func readOCFHeader(ior io.Reader) (*ocfHeader, error) {
//
// magic bytes
//
magic := make([]byte, 4)
_, err := io.ReadFull(ior, magic)
if err != nil {
return nil, fmt.Errorf("cannot read OCF header magic bytes: %s", err)
}
if !bytes.Equal(magic, ocfMagicBytes) {
return nil, fmt.Errorf("cannot read OCF header with invalid magic bytes: %#q", magic)
}
//
// metadata
//
metadata, err := metadataBinaryReader(ior)
if err != nil {
return nil, fmt.Errorf("cannot read OCF header metadata: %s", err)
}
//
// avro.codec
//
// NOTE: Avro specification states that `null` cID is used by
// default when "avro.codec" was not included in the metadata header. The
// specification does not talk about the case when "avro.codec" was included
// with the empty string as its value. I believe it is an error for an OCF
// file to provide the empty string as the cID algorithm. While it
// is trivially easy to gracefully handle here, I'm not sure whether this
// happens a lot, and don't want to accept bad input unless we have
// significant reason to do so.
var cID compressionID
value, ok := metadata["avro.codec"]
if ok {
switch avroCodec := string(value); avroCodec {
case CompressionNullLabel:
cID = compressionNull
case CompressionDeflateLabel:
cID = compressionDeflate
case CompressionSnappyLabel:
cID = compressionSnappy
default:
return nil, fmt.Errorf("cannot read OCF header using unrecognized compression algorithm from avro.codec: %q", avroCodec)
}
}
//
// create goavro.Codec from specified avro.schema
//
value, ok = metadata["avro.schema"]
if !ok {
return nil, errors.New("cannot read OCF header without avro.schema")
}
codec, err := NewCodec(string(value))
if err != nil {
return nil, fmt.Errorf("cannot read OCF header with invalid avro.schema: %s", err)
}
header := &ocfHeader{codec: codec, compressionID: cID, metadata: metadata}
//
// read and store sync marker
//
if n, err := io.ReadFull(ior, header.syncMarker[:]); err != nil {
return nil, fmt.Errorf("cannot read OCF header without sync marker: only read %d of %d bytes: %s", n, ocfSyncLength, err)
}
//
// header is valid
//
return header, nil
}
func writeOCFHeader(header *ocfHeader, iow io.Writer) (err error) {
//
// avro.codec
//
var avroCodec string
switch header.compressionID {
case compressionNull:
avroCodec = CompressionNullLabel
case compressionDeflate:
avroCodec = CompressionDeflateLabel
case compressionSnappy:
avroCodec = CompressionSnappyLabel
default:
return fmt.Errorf("should not get here: cannot write OCF header using unrecognized compression algorithm: %d", header.compressionID)
}
//
// avro.schema
//
// Create buffer for OCF header. The first four bytes are magic, and we'll
// use copy to fill them in, so initialize buffer's length with 4, and its
// capacity equal to length of avro schema plus a constant.
schema := header.codec.Schema()
buf := make([]byte, 4, len(schema)+ocfHeaderSizeConst)
_ = copy(buf, ocfMagicBytes)
//
// file metadata, including the schema
//
meta := make(map[string]interface{})
for k, v := range header.metadata {
meta[k] = v
}
meta["avro.schema"] = []byte(schema)
meta["avro.codec"] = []byte(avroCodec)
buf, err = ocfMetadataCodec.BinaryFromNative(buf, meta)
if err != nil {
return fmt.Errorf("should not get here: cannot write OCF header: %s", err)
}
//
// 16-byte sync marker
//
buf = append(buf, header.syncMarker[:]...)
// emit OCF header
_, err = iow.Write(buf)
if err != nil {
return fmt.Errorf("cannot write OCF header: %s", err)
}
return nil
}
vendor/github.com/linkedin/goavro/v2/ocf_reader.go
Generated Vendored
+263
View File
@@ -0,0 +1,263 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"compress/flate"
"encoding/binary"
"errors"
"fmt"
"hash/crc32"
"io"
"io/ioutil"
"github.com/golang/snappy"
)
// OCFReader structure is used to read Object Container Files (OCF).
type OCFReader struct {
header *ocfHeader
block []byte // buffer from which decoding takes place
rerr error // most recent error that took place while reading bytes (unrecoverable)
ior io.Reader
readReady bool // true after Scan and before Read
remainingBlockItems int64 // count of encoded data items remaining in block buffer to be decoded
}
// NewOCFReader initializes and returns a new structure used to read an Avro
// Object Container File (OCF).
//
// func example(ior io.Reader) error {
// // NOTE: Wrap provided io.Reader in a buffered reader, which improves the
// // performance of streaming file data.
// br := bufio.NewReader(ior)
// ocfr, err := goavro.NewOCFReader(br)
// if err != nil {
// return err
// }
// for ocfr.Scan() {
// datum, err := ocfr.Read()
// if err != nil {
// return err
// }
// fmt.Println(datum)
// }
// return ocfr.Err()
// }
func NewOCFReader(ior io.Reader) (*OCFReader, error) {
header, err := readOCFHeader(ior)
if err != nil {
return nil, fmt.Errorf("cannot create OCFReader: %s", err)
}
return &OCFReader{header: header, ior: ior}, nil
}
//MetaData returns the file metadata map found within the OCF file
func (ocfr *OCFReader) MetaData() map[string][]byte {
return ocfr.header.metadata
}
// Codec returns the codec found within the OCF file.
func (ocfr *OCFReader) Codec() *Codec {
return ocfr.header.codec
}
// CompressionName returns the name of the compression algorithm found within
// the OCF file.
func (ocfr *OCFReader) CompressionName() string {
switch ocfr.header.compressionID {
case compressionNull:
return CompressionNullLabel
case compressionDeflate:
return CompressionDeflateLabel
case compressionSnappy:
return CompressionSnappyLabel
default:
return "should not get here: unrecognized compression algorithm"
}
}
// Err returns the last error encountered while reading the OCF file. See
// `NewOCFReader` documentation for an example.
func (ocfr *OCFReader) Err() error {
return ocfr.rerr
}
// Read consumes one datum value from the Avro OCF stream and returns it. Read
// is designed to be called only once after each invocation of the Scan method.
// See `NewOCFReader` documentation for an example.
func (ocfr *OCFReader) Read() (interface{}, error) {
// NOTE: Test previous error before testing readReady to prevent overwriting
// previous error.
if ocfr.rerr != nil {
return nil, ocfr.rerr
}
if !ocfr.readReady {
ocfr.rerr = errors.New("Read called without successful Scan")
return nil, ocfr.rerr
}
ocfr.readReady = false
// decode one datum value from block
var datum interface{}
datum, ocfr.block, ocfr.rerr = ocfr.header.codec.NativeFromBinary(ocfr.block)
if ocfr.rerr != nil {
return false, ocfr.rerr
}
ocfr.remainingBlockItems--
return datum, nil
}
// RemainingBlockItems returns the number of items remaining in the block being
// processed.
func (ocfr *OCFReader) RemainingBlockItems() int64 {
return ocfr.remainingBlockItems
}
// Scan returns true when there is at least one more data item to be read from
// the Avro OCF. Scan ought to be called prior to calling the Read method each
// time the Read method is invoked. See `NewOCFReader` documentation for an
// example.
func (ocfr *OCFReader) Scan() bool {
ocfr.readReady = false
if ocfr.rerr != nil {
return false
}
// NOTE: If there are no more remaining data items from the existing block,
// then attempt to slurp in the next block.
if ocfr.remainingBlockItems <= 0 {
if count := len(ocfr.block); count != 0 {
ocfr.rerr = fmt.Errorf("extra bytes between final datum in previous block and block sync marker: %d", count)
return false
}
// Read the block count and update the number of remaining items for
// this block
ocfr.remainingBlockItems, ocfr.rerr = longBinaryReader(ocfr.ior)
if ocfr.rerr != nil {
if ocfr.rerr == io.EOF {
ocfr.rerr = nil // merely end of file, rather than error
} else {
ocfr.rerr = fmt.Errorf("cannot read block count: %s", ocfr.rerr)
}
return false
}
if ocfr.remainingBlockItems <= 0 {
ocfr.rerr = fmt.Errorf("cannot decode when block count is not greater than 0: %d", ocfr.remainingBlockItems)
return false
}
if ocfr.remainingBlockItems > MaxBlockCount {
ocfr.rerr = fmt.Errorf("cannot decode when block count exceeds MaxBlockCount: %d > %d", ocfr.remainingBlockItems, MaxBlockCount)
}
var blockSize int64
blockSize, ocfr.rerr = longBinaryReader(ocfr.ior)
if ocfr.rerr != nil {
ocfr.rerr = fmt.Errorf("cannot read block size: %s", ocfr.rerr)
return false
}
if blockSize <= 0 {
ocfr.rerr = fmt.Errorf("cannot decode when block size is not greater than 0: %d", blockSize)
return false
}
if blockSize > MaxBlockSize {
ocfr.rerr = fmt.Errorf("cannot decode when block size exceeds MaxBlockSize: %d > %d", blockSize, MaxBlockSize)
return false
}
// read entire block into buffer
ocfr.block = make([]byte, blockSize)
_, ocfr.rerr = io.ReadFull(ocfr.ior, ocfr.block)
if ocfr.rerr != nil {
ocfr.rerr = fmt.Errorf("cannot read block: %s", ocfr.rerr)
return false
}
switch ocfr.header.compressionID {
case compressionNull:
// no-op
case compressionDeflate:
// NOTE: flate.NewReader wraps with io.ByteReader if argument does
// not implement that interface.
rc := flate.NewReader(bytes.NewBuffer(ocfr.block))
ocfr.block, ocfr.rerr = ioutil.ReadAll(rc)
if ocfr.rerr != nil {
_ = rc.Close()
return false
}
if ocfr.rerr = rc.Close(); ocfr.rerr != nil {
return false
}
case compressionSnappy:
index := len(ocfr.block) - 4 // last 4 bytes is crc32 of decoded block
if index <= 0 {
ocfr.rerr = fmt.Errorf("cannot decompress snappy without CRC32 checksum: %d", len(ocfr.block))
return false
}
decoded, err := snappy.Decode(nil, ocfr.block[:index])
if err != nil {
ocfr.rerr = fmt.Errorf("cannot decompress: %s", err)
return false
}
actualCRC := crc32.ChecksumIEEE(decoded)
expectedCRC := binary.BigEndian.Uint32(ocfr.block[index : index+4])
if actualCRC != expectedCRC {
ocfr.rerr = fmt.Errorf("snappy CRC32 checksum mismatch: %x != %x", actualCRC, expectedCRC)
return false
}
ocfr.block = decoded
default:
ocfr.rerr = fmt.Errorf("should not get here: cannot compress block using unrecognized compression: %d", ocfr.header.compressionID)
return false
}
// read and ensure sync marker matches
sync := make([]byte, ocfSyncLength)
var n int
if n, ocfr.rerr = io.ReadFull(ocfr.ior, sync); ocfr.rerr != nil {
ocfr.rerr = fmt.Errorf("cannot read sync marker: read %d out of %d bytes: %s", n, ocfSyncLength, ocfr.rerr)
return false
}
if !bytes.Equal(sync, ocfr.header.syncMarker[:]) {
ocfr.rerr = fmt.Errorf("sync marker mismatch: %v != %v", sync, ocfr.header.syncMarker)
return false
}
}
ocfr.readReady = true
return true
}
// SkipThisBlockAndReset can be called after an error occurs while reading or
// decoding datum values from an OCF stream. OCF specifies each OCF stream
// contain one or more blocks of data. Each block consists of a block count, the
// number of bytes for the block, followed be the possibly compressed
// block. Inside each decompressed block is all of the binary encoded datum
// values concatenated together. In other words, OCF framing is at a block level
// rather than a datum level. If there is an error while reading or decoding a
// datum, the reader is not able to skip to the next datum value, because OCF
// does not have any markers for where each datum ends and the next one
// begins. Therefore, the reader is only able to skip this datum value and all
// subsequent datum values in the current block, move to the next block and
// start decoding datum values there.
func (ocfr *OCFReader) SkipThisBlockAndReset() {
// ??? is it an error to call method unless the reader has had an error
ocfr.remainingBlockItems = 0
ocfr.block = ocfr.block[:0]
ocfr.rerr = nil
}
vendor/github.com/linkedin/goavro/v2/ocf_writer.go
Generated Vendored
+253
View File
@@ -0,0 +1,253 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"compress/flate"
"encoding/binary"
"errors"
"fmt"
"hash/crc32"
"io"
"io/ioutil"
"os"
"github.com/golang/snappy"
)
// OCFConfig is used to specify creation parameters for OCFWriter.
type OCFConfig struct {
// W specifies the `io.Writer` to which to send the encoded data,
// (required). If W is `*os.File`, then creating an OCF for writing will
// attempt to read any existing OCF header and use the schema and
// compression codec specified by the existing header, then advance the file
// position to the tail end of the file for appending.
W io.Writer
// Codec specifies the Codec to use for the new OCFWriter, (optional). If
// the W parameter above is an `*os.File` which contains a Codec, the Codec
// in the existing file will be used instead. Otherwise if this Codec
// parameter is specified, it will be used. If neither the W parameter above
// is an `*os.File` with an existing Codec, nor this Codec parameter is
// specified, the OCFWriter will create a new Codec from the schema string
// specified by the Schema parameter below.
Codec *Codec
// Schema specifies the Avro schema for the data to be encoded, (optional).
// If neither the W parameter above is an `*os.File` with an existing Codec,
// nor the Codec parameter above is specified, the OCFWriter will create a
// new Codec from the schema string specified by this Schema parameter.
Schema string
// CompressionName specifies the compression codec used, (optional). If
// omitted, defaults to "null" codec. When appending to an existing OCF,
// this field is ignored.
CompressionName string
//MetaData specifies application specific meta data to be added to
//the OCF file. When appending to an existing OCF, this field
//is ignored
MetaData map[string][]byte
}
// OCFWriter is used to create a new or append to an existing Avro Object
// Container File (OCF).
type OCFWriter struct {
header *ocfHeader
iow io.Writer
}
// NewOCFWriter returns a new OCFWriter instance that may be used for appending
// binary Avro data, either by appending to an existing OCF file or creating a
// new OCF file.
func NewOCFWriter(config OCFConfig) (*OCFWriter, error) {
var err error
ocf := &OCFWriter{iow: config.W}
switch config.W.(type) {
case nil:
return nil, errors.New("cannot create OCFWriter when W is nil")
case *os.File:
file := config.W.(*os.File)
stat, err := file.Stat()
if err != nil {
return nil, fmt.Errorf("cannot create OCFWriter: %s", err)
}
// NOTE: When upstream provides a new file, it will already exist but
// have a size of 0 bytes.
if stat.Size() > 0 {
// attempt to read existing OCF header
if ocf.header, err = readOCFHeader(file); err != nil {
return nil, fmt.Errorf("cannot create OCFWriter: %s", err)
}
// prepare for appending data to existing OCF
if err = ocf.quickScanToTail(file); err != nil {
return nil, fmt.Errorf("cannot create OCFWriter: %s", err)
}
return ocf, nil // happy case for appending to existing OCF
}
}
// create new OCF header based on configuration parameters
if ocf.header, err = newOCFHeader(config); err != nil {
return nil, fmt.Errorf("cannot create OCFWriter: %s", err)
}
if err = writeOCFHeader(ocf.header, config.W); err != nil {
return nil, fmt.Errorf("cannot create OCFWriter: %s", err)
}
return ocf, nil // another happy case for creation of new OCF
}
// quickScanToTail advances the stream reader to the tail end of the
// file. Rather than reading each encoded block, optionally decompressing it,
// and then decoding it, this method reads the block count, ignoring it, then
// reads the block size, then skips ahead to the followig block. It does this
// repeatedly until attempts to read the file return io.EOF.
func (ocfw *OCFWriter) quickScanToTail(ior io.Reader) error {
sync := make([]byte, ocfSyncLength)
for {
// Read and validate block count
blockCount, err := longBinaryReader(ior)
if err != nil {
if err == io.EOF {
return nil // merely end of file, rather than error
}
return fmt.Errorf("cannot read block count: %s", err)
}
if blockCount <= 0 {
return fmt.Errorf("cannot read when block count is not greater than 0: %d", blockCount)
}
if blockCount > MaxBlockCount {
return fmt.Errorf("cannot read when block count exceeds MaxBlockCount: %d > %d", blockCount, MaxBlockCount)
}
// Read block size
blockSize, err := longBinaryReader(ior)
if err != nil {
return fmt.Errorf("cannot read block size: %s", err)
}
if blockSize <= 0 {
return fmt.Errorf("cannot read when block size is not greater than 0: %d", blockSize)
}
if blockSize > MaxBlockSize {
return fmt.Errorf("cannot read when block size exceeds MaxBlockSize: %d > %d", blockSize, MaxBlockSize)
}
// Advance reader to end of block
if _, err = io.CopyN(ioutil.Discard, ior, blockSize); err != nil {
return fmt.Errorf("cannot seek to next block: %s", err)
}
// Read and validate sync marker
var n int
if n, err = io.ReadFull(ior, sync); err != nil {
return fmt.Errorf("cannot read sync marker: read %d out of %d bytes: %s", n, ocfSyncLength, err)
}
if !bytes.Equal(sync, ocfw.header.syncMarker[:]) {
return fmt.Errorf("sync marker mismatch: %v != %v", sync, ocfw.header.syncMarker)
}
}
}
// Append appends one or more data items to an OCF file in a block. If there are
// more data items in the slice than MaxBlockCount allows, the data slice will
// be chunked into multiple blocks, each not having more than MaxBlockCount
// items.
func (ocfw *OCFWriter) Append(data interface{}) error {
arrayValues, err := convertArray(data)
if err != nil {
return err
}
// Chunk data so no block has more than MaxBlockCount items.
for int64(len(arrayValues)) > MaxBlockCount {
if err := ocfw.appendDataIntoBlock(arrayValues[:MaxBlockCount]); err != nil {
return err
}
arrayValues = arrayValues[MaxBlockCount:]
}
return ocfw.appendDataIntoBlock(arrayValues)
}
func (ocfw *OCFWriter) appendDataIntoBlock(data []interface{}) error {
var block []byte // working buffer for encoding data values
var err error
// Encode and concatenate each data item into the block
for _, datum := range data {
if block, err = ocfw.header.codec.BinaryFromNative(block, datum); err != nil {
return fmt.Errorf("cannot translate datum to binary: %v; %s", datum, err)
}
}
switch ocfw.header.compressionID {
case compressionNull:
// no-op
case compressionDeflate:
// compress into new bytes buffer.
bb := bytes.NewBuffer(make([]byte, 0, len(block)))
cw, _ := flate.NewWriter(bb, flate.DefaultCompression)
// writing bytes to cw will compress bytes and send to bb.
if _, err := cw.Write(block); err != nil {
return err
}
if err := cw.Close(); err != nil {
return err
}
block = bb.Bytes()
case compressionSnappy:
compressed := snappy.Encode(nil, block)
// OCF requires snappy to have CRC32 checksum after each snappy block
compressed = append(compressed, 0, 0, 0, 0) // expand slice by 4 bytes so checksum will fit
binary.BigEndian.PutUint32(compressed[len(compressed)-4:], crc32.ChecksumIEEE(block)) // checksum of decompressed block
block = compressed
default:
return fmt.Errorf("should not get here: cannot compress block using unrecognized compression: %d", ocfw.header.compressionID)
}
// create file data block
buf := make([]byte, 0, len(block)+ocfBlockConst) // pre-allocate block bytes
buf, _ = longBinaryFromNative(buf, len(data)) // block count (number of data items)
buf, _ = longBinaryFromNative(buf, len(block)) // block size (number of bytes in block)
buf = append(buf, block...) // serialized objects
buf = append(buf, ocfw.header.syncMarker[:]...) // sync marker
_, err = ocfw.iow.Write(buf)
return err
}
// Codec returns the codec used by OCFWriter. This function provided because
// upstream may be appending to existing OCF which uses a different schema than
// requested during instantiation.
func (ocfw *OCFWriter) Codec() *Codec {
return ocfw.header.codec
}
// CompressionName returns the name of the compression algorithm used by
// OCFWriter. This function provided because upstream may be appending to
// existing OCF which uses a different compression algorithm than requested
// during instantiation. the OCF file.
func (ocfw *OCFWriter) CompressionName() string {
switch ocfw.header.compressionID {
case compressionNull:
return CompressionNullLabel
case compressionDeflate:
return CompressionDeflateLabel
case compressionSnappy:
return CompressionSnappyLabel
default:
return "should not get here: unrecognized compression algorithm"
}
}
vendor/github.com/linkedin/goavro/v2/rabin.go
Generated Vendored
+336
View File
@@ -0,0 +1,336 @@
package goavro
import (
"encoding/binary"
"fmt"
"io"
)
// rabinEmpty is a constant used to initialize the crc64Table, and to compute
// the CRC-64-AVRO fingerprint of every object schema.
const rabinEmpty = uint64(0xc15d213aa4d7a795)
// rabinTable is never modified after initialization but its values are read to
// compute the CRC-64-AVRO fingerprint of every schema its given.
var rabinTable = [256]uint64{
0,
3238593523956797946,
6477187047913595892,
8435907220062204430,
12954374095827191784,
11472609148414072338,
16871814440124408860,
14327483619285186022,
16515860097293205755,
14539261057490653441,
13607494391182877455,
10387063993012335349,
6265406319754774291,
8791864835633305321,
1085550678754862311,
2585467722461443357,
5247393906202824413,
7215812591205457703,
1239030555549527337,
4449591751341063379,
18092457712352332085,
15556728100436498639,
11742789833002527425,
10234164645493242683,
12530812639509548582,
9302088354573213660,
17583729671266610642,
15633189885995973672,
2171101357509724622,
3661574416647526452,
5170935444922886714,
7724537325157989312,
10494787812405648826,
13642865964979244096,
14431625182410915406,
16480541316673728436,
2478061111099054674,
1049933365183482792,
8899183502682126758,
6300970840149272668,
8399466921467862337,
6368420890995002555,
3275086581351513781,
108854135608684367,
14364169659802000041,
16980263386864569171,
11435870349096892765,
12845837170396948647,
15669858317114364775,
17692196227407282845,
9265331945857609875,
12422293323479818601,
7688114635962061967,
5062151678603773301,
3698085083440658299,
2279937883717887617,
4342202715019449244,
1203395666939462246,
7323148833295052904,
5282940851558637970,
10341870889845773428,
11778178981837571470,
15449074650315978624,
18057156506771531386,
11669866394404287583,
10160817855121008037,
17874829710049597355,
15339802717267265105,
1311848476550706103,
4523114428088083021,
5464845951130112067,
7432843562972398009,
4956122222198109348,
7509300761534850398,
2099866730366965584,
3591042414950500010,
17798367005364253516,
15848531969535615670,
12601941680298545336,
9372796311334617410,
16798933842935724674,
14253900473960229752,
12736841781990005110,
11255500115345754252,
6550173162703027562,
8509314479008689296,
217708271217368734,
3455596968422674276,
870833084869474937,
2370047569572014979,
6194214610827729293,
8721096401170761847,
13822387873690697105,
10602378625989962859,
16587157392570359397,
14609853536892473247,
3483332339477899749,
2064482512161650719,
7616958077116566033,
4991418462803860459,
9480190278288059917,
12637572737790640119,
15741190762473065977,
17762823925471730691,
15376229271924123934,
17983608511393921252,
10124303357207546602,
11561034798826117904,
7396170166881316598,
5356383260452470540,
4559875767435775234,
1420363961462201592,
8684405430038898488,
6085769495188764354,
2406791333878924492,
979366144819647798,
14646297666590105808,
16695918618875998506,
10565881703117275940,
13713538703073841886,
11362911691697612739,
12772455230081578553,
14146576876296094775,
16763373153642681805,
3347869283551649835,
182341662412566993,
8616954185191982047,
6585487012709290533,
13933329357911598997,
17126321439046432367,
11006435164953838689,
12992741788688209307,
8257930048646602877,
6803747195591438727,
3132703159877387145,
542775339377431155,
2623696953101412206,
619515277774763668,
9046228856176166042,
5871394916501263712,
10929691902260224134,
13501751302614184316,
14865687125944796018,
16338017159720129160,
9912244444396218696,
11925134239902742706,
15018601523069700796,
18202706530865158982,
4199733460733931168,
1637543290675756890,
7182084829901000020,
5717935174548446382,
7834929158557182387,
4632665972928804937,
3844057317981030983,
1849042541720329149,
16103865201353027163,
17549867708331900833,
9700748483321744815,
12280807109898935381,
5834933197202143791,
8937414855024798677,
655924238275353051,
2732422975565056033,
16374796089197559239,
14974255385173568573,
13465025131935292979,
10821211621719183305,
13100346325406055124,
11041713811386575662,
17018628958017378592,
13897997918303815898,
435416542434737468,
3097107305413864646,
6911193936845348552,
8293578696285179698,
1741666169738949874,
3808479038558283016,
4740095139144029958,
7870595381236532988,
12388429221655458586,
9736009554713699040,
17442192802341523694,
16068516186704462100,
18239503069743100937,
15127152172900050419,
11888425678624364541,
9803746554456753671,
5681455845848806369,
7073288438148047387,
1673934641775824917,
4308477092595991023,
6966664678955799498,
5503217582476919344,
4128965024323301438,
1566351579938693572,
15233916154233132066,
18417600011429070296,
9982836925607720918,
11996431537128302124,
9627165335515697969,
12207926510359495371,
15886756170769674437,
17332335396841578815,
3917464579278591193,
1922028658990515491,
8051932600676513581,
4850374241660872407,
2917466598601071895,
327962119137676525,
8187398044598779619,
6732512565967646489,
11221777246008269567,
13207379120439233285,
14004037317153847563,
17197450482186430705,
14792340333762633196,
16265093719173729302,
10712766520904941080,
13284123302255603682,
9119751534871550468,
5944212839312182270,
2840727922924403184,
836967320887912458,
17368810860077796976,
15995557527495450506,
12171538990377528708,
9518416773021940862,
4813582667757848984,
7943378085384837218,
1958732289639295596,
4025966300338256790,
1458733299300535947,
4093699022299389809,
5610888623004134783,
7002018658576923781,
12103802978479819107,
10018419036150929561,
18310175810188503703,
15198246066092718957,
13391477134206599341,
10748366240846565719,
16157651908532642649,
14756687855020634787,
729366649650267973,
2805444311502067391,
6051901489239909553,
9155087905094251851,
6695738567103299670,
8078825954266321324,
364683324825133986,
3025950744619954776,
17233908370383964094,
14112856248920397380,
13170974025418581066,
11113046258555286960,
}
// rabin returns an unsigned 64-bit integer Rabin fingerprint for buf. NOTE:
// This is only used during Codec instantiation to calculate the Rabin
// fingerprint of the canonical schema.
func rabin(buf []byte) uint64 {
fp := rabinEmpty
for i := 0; i < len(buf); i++ {
fp = (fp >> 8) ^ rabinTable[(byte(fp)^buf[i])&0xff] // unsigned right shift >>>
}
return fp
}
const soeMagicPrefix = 2 // 2-byte prefix for SOE encoded data
const soeHeaderLen = soeMagicPrefix + 8 // 2-byte prefix plus 8-byte fingerprint
// FingerprintFromSOE returns the unsigned 64-bit Rabin fingerprint from the
// header of a buffer that encodes a Single-Object Encoded datum. This function
// is designed to be used to lookup a Codec that can decode the contents of the
// buffer. Once a Codec is found that has the matching Rabin fingerprint, its
// NativeFromBinary method may be used to decode the remaining bytes returned as
// the second return value. On failure this function returns an
// ErrNotSingleObjectEncoded error.
//
// func decode(codex map[uint64]*goavro.Codec, buf []byte) error {
// // Perform a sanity check on the buffer, then return the Rabin fingerprint
// // of the schema used to encode the data.
// fingerprint, newBuf, err := goavro.FingerprintFromSOE(buf)
// if err != nil {
// return err
// }
//
// // Get a previously stored Codec from the codex map.
// codec, ok := codex[fingerprint]
// if !ok {
// return fmt.Errorf("unknown codec: %#x", fingerprint)
// }
//
// // Use the fetched Codec to decode the buffer as a SOE.
// //
// // Faster because SOE magic prefix and schema fingerprint already
// // checked and used to fetch the Codec. Just need to decode the binary
// // bytes remaining after the prefix were removed.
// datum, _, err := codec.NativeFromBinary(newBuf)
// if err != nil {
// return err
// }
//
// _, err = fmt.Println(datum)
// return err
// }
func FingerprintFromSOE(buf []byte) (uint64, []byte, error) {
if len(buf) < soeHeaderLen {
// Not enough bytes to encode schema fingerprint.
return 0, nil, ErrNotSingleObjectEncoded(io.ErrShortBuffer.Error())
}
if buf[0] != 0xC3 || buf[1] != 0x01 {
// Currently only one SOE prefix is recognized.
return 0, nil, ErrNotSingleObjectEncoded(fmt.Sprintf("unknown SOE prefix: %#x", buf[:soeMagicPrefix]))
}
// Only recognizes single-object encodings format version 1.
return binary.LittleEndian.Uint64(buf[soeMagicPrefix:]), buf[soeHeaderLen:], nil
}
vendor/github.com/linkedin/goavro/v2/record.go
Generated Vendored
+233
View File
@@ -0,0 +1,233 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
)
func makeRecordCodec(st map[string]*Codec, enclosingNamespace string, schemaMap map[string]interface{}) (*Codec, error) {
// NOTE: To support recursive data types, create the codec and register it
// using the specified name, and fill in the codec functions later.
c, err := registerNewCodec(st, schemaMap, enclosingNamespace)
if err != nil {
return nil, fmt.Errorf("Record ought to have valid name: %s", err)
}
fields, ok := schemaMap["fields"]
if !ok {
return nil, fmt.Errorf("Record %q ought to have fields key", c.typeName)
}
fieldSchemas, ok := fields.([]interface{})
if !ok || fieldSchemas == nil {
return nil, fmt.Errorf("Record %q fields ought to be non-nil array: %v", c.typeName, fields)
}
codecFromFieldName := make(map[string]*Codec)
codecFromIndex := make([]*Codec, len(fieldSchemas))
nameFromIndex := make([]string, len(fieldSchemas))
defaultValueFromName := make(map[string]interface{}, len(fieldSchemas))
for i, fieldSchema := range fieldSchemas {
fieldSchemaMap, ok := fieldSchema.(map[string]interface{})
if !ok {
return nil, fmt.Errorf("Record %q field %d ought to be valid Avro named type; received: %v", c.typeName, i+1, fieldSchema)
}
// NOTE: field names are not registered in the symbol table, because
// field names are not individually addressable codecs.
fieldCodec, err := buildCodecForTypeDescribedByMap(st, c.typeName.namespace, fieldSchemaMap)
if err != nil {
return nil, fmt.Errorf("Record %q field %d ought to be valid Avro named type: %s", c.typeName, i+1, err)
}
// However, when creating a full name for the field name, be sure to use
// record's namespace
n, err := newNameFromSchemaMap(c.typeName.namespace, fieldSchemaMap)
if err != nil {
return nil, fmt.Errorf("Record %q field %d ought to have valid name: %v", c.typeName, i+1, fieldSchemaMap)
}
fieldName := n.short()
if _, ok := codecFromFieldName[fieldName]; ok {
return nil, fmt.Errorf("Record %q field %d ought to have unique name: %q", c.typeName, i+1, fieldName)
}
if defaultValue, ok := fieldSchemaMap["default"]; ok {
typeNameShort := fieldCodec.typeName.short()
switch typeNameShort {
case "boolean":
v, ok := defaultValue.(bool)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = bool(v)
case "bytes":
v, ok := defaultValue.(string)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = []byte(v)
case "double":
v, ok := defaultValue.(float64)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = float64(v)
case "float":
v, ok := defaultValue.(float64)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = float32(v)
case "int":
v, ok := defaultValue.(float64)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = int32(v)
case "long":
v, ok := defaultValue.(float64)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = int64(v)
case "string":
v, ok := defaultValue.(string)
if !ok {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValue = string(v)
case "union":
// When codec is union, then default value ought to encode using
// first schema in union. NOTE: To support a null default
// value, the string literal "null" must be coerced to a `nil`
if defaultValue == "null" {
defaultValue = nil
}
// NOTE: To support record field default values, union schema
// set to the type name of first member
// TODO: change to schemaCanonical below
defaultValue = Union(fieldCodec.schemaOriginal, defaultValue)
default:
debug("fieldName: %q; type: %q; defaultValue: %T(%#v)\n", fieldName, c.typeName, defaultValue, defaultValue)
}
// attempt to encode default value using codec
_, err = fieldCodec.binaryFromNative(nil, defaultValue)
if err != nil {
return nil, fmt.Errorf("Record %q field %q: default value ought to encode using field schema: %s", c.typeName, fieldName, err)
}
defaultValueFromName[fieldName] = defaultValue
}
nameFromIndex[i] = fieldName
codecFromIndex[i] = fieldCodec
codecFromFieldName[fieldName] = fieldCodec
}
c.binaryFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
valueMap, ok := datum.(map[string]interface{})
if !ok {
return nil, fmt.Errorf("cannot encode binary record %q: expected map[string]interface{}; received: %T", c.typeName, datum)
}
// records encoded in order fields were defined in schema
for i, fieldCodec := range codecFromIndex {
fieldName := nameFromIndex[i]
// NOTE: If field value was not specified in map, then set
// fieldValue to its default value (which may or may not have been
// specified).
fieldValue, ok := valueMap[fieldName]
if !ok {
if fieldValue, ok = defaultValueFromName[fieldName]; !ok {
return nil, fmt.Errorf("cannot encode binary record %q field %q: schema does not specify default value and no value provided", c.typeName, fieldName)
}
}
var err error
buf, err = fieldCodec.binaryFromNative(buf, fieldValue)
if err != nil {
return nil, fmt.Errorf("cannot encode binary record %q field %q: value does not match its schema: %s", c.typeName, fieldName, err)
}
}
return buf, nil
}
c.nativeFromBinary = func(buf []byte) (interface{}, []byte, error) {
recordMap := make(map[string]interface{}, len(codecFromIndex))
for i, fieldCodec := range codecFromIndex {
name := nameFromIndex[i]
var value interface{}
var err error
value, buf, err = fieldCodec.nativeFromBinary(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode binary record %q field %q: %s", c.typeName, name, err)
}
recordMap[name] = value
}
return recordMap, buf, nil
}
c.nativeFromTextual = func(buf []byte) (interface{}, []byte, error) {
var mapValues map[string]interface{}
var err error
// NOTE: Setting `defaultCodec == nil` instructs genericMapTextDecoder
// to return an error when a field name is not found in the
// codecFromFieldName map.
mapValues, buf, err = genericMapTextDecoder(buf, nil, codecFromFieldName)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual record %q: %s", c.typeName, err)
}
if actual, expected := len(mapValues), len(codecFromFieldName); actual != expected {
// set missing field keys to their respective default values, then
// re-check number of keys
for fieldName, defaultValue := range defaultValueFromName {
if _, ok := mapValues[fieldName]; !ok {
mapValues[fieldName] = defaultValue
}
}
if actual, expected = len(mapValues), len(codecFromFieldName); actual != expected {
return nil, nil, fmt.Errorf("cannot decode textual record %q: only found %d of %d fields", c.typeName, actual, expected)
}
}
return mapValues, buf, nil
}
c.textualFromNative = func(buf []byte, datum interface{}) ([]byte, error) {
// NOTE: Ensure only schema defined field names are encoded; and if
// missing in datum, either use the provided field default value or
// return an error.
sourceMap, ok := datum.(map[string]interface{})
if !ok {
return nil, fmt.Errorf("cannot encode textual record %q: expected map[string]interface{}; received: %T", c.typeName, datum)
}
destMap := make(map[string]interface{}, len(codecFromIndex))
for fieldName := range codecFromFieldName {
fieldValue, ok := sourceMap[fieldName]
if !ok {
defaultValue, ok := defaultValueFromName[fieldName]
if !ok {
return nil, fmt.Errorf("cannot encode textual record %q field %q: schema does not specify default value and no value provided", c.typeName, fieldName)
}
fieldValue = defaultValue
}
destMap[fieldName] = fieldValue
}
datum = destMap
// NOTE: Setting `defaultCodec == nil` instructs genericMapTextEncoder
// to return an error when a field name is not found in the
// codecFromFieldName map.
return genericMapTextEncoder(buf, datum, nil, codecFromFieldName)
}
return c, nil
}
vendor/github.com/linkedin/goavro/v2/text.go
Generated Vendored
+41
View File
@@ -0,0 +1,41 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"fmt"
"io"
"unicode"
)
// advanceAndConsume advances to non whitespace and returns an error if the next
// non whitespace byte is not what is expected.
func advanceAndConsume(buf []byte, expected byte) ([]byte, error) {
var err error
if buf, err = advanceToNonWhitespace(buf); err != nil {
return nil, err
}
if actual := buf[0]; actual != expected {
return nil, fmt.Errorf("expected: %q; actual: %q", expected, actual)
}
return buf[1:], nil
}
// advanceToNonWhitespace consumes bytes from buf until non-whitespace character
// is found. It returns error when no more bytes remain, because its purpose is
// to scan ahead to the next non-whitespace character.
func advanceToNonWhitespace(buf []byte) ([]byte, error) {
for i, b := range buf {
if !unicode.IsSpace(rune(b)) {
return buf[i:], nil
}
}
return nil, io.ErrShortBuffer
}
vendor/github.com/linkedin/goavro/v2/union.go
Generated Vendored
+179
View File
@@ -0,0 +1,179 @@
// Copyright [2019] LinkedIn Corp. Licensed under the Apache License, Version
// 2.0 (the "License"); you may not use this file except in compliance with the
// License. You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
package goavro
import (
"bytes"
"errors"
"fmt"
)
// Union wraps a datum value in a map for encoding as a Union, as required by
// Union encoder.
//
// When providing a value for an Avro union, the encoder will accept `nil` for a
// `null` value. If the value is non-`nil`, it must be a
// `map[string]interface{}` with a single key-value pair, where the key is the
// Avro type name and the value is the datum's value. As a convenience, the
// `Union` function wraps any datum value in a map as specified above.
//
// func ExampleUnion() {
// codec, err := goavro.NewCodec(`["null","string","int"]`)
// if err != nil {
// fmt.Println(err)
// }
// buf, err := codec.TextualFromNative(nil, goavro.Union("string", "some string"))
// if err != nil {
// fmt.Println(err)
// }
// fmt.Println(string(buf))
// // Output: {"string":"some string"}
// }
func Union(name string, datum interface{}) interface{} {
if datum == nil && name == "null" {
return nil
}
return map[string]interface{}{name: datum}
}
func buildCodecForTypeDescribedBySlice(st map[string]*Codec, enclosingNamespace string, schemaArray []interface{}) (*Codec, error) {
if len(schemaArray) == 0 {
return nil, errors.New("Union ought to have one or more members")
}
allowedTypes := make([]string, len(schemaArray)) // used for error reporting when encoder receives invalid datum type
codecFromIndex := make([]*Codec, len(schemaArray))
codecFromName := make(map[string]*Codec, len(schemaArray))
indexFromName := make(map[string]int, len(schemaArray))
for i, unionMemberSchema := range schemaArray {
unionMemberCodec, err := buildCodec(st, enclosingNamespace, unionMemberSchema)
if err != nil {
return nil, fmt.Errorf("Union item %d ought to be valid Avro type: %s", i+1, err)
}
fullName := unionMemberCodec.typeName.fullName
if _, ok := indexFromName[fullName]; ok {
return nil, fmt.Errorf("Union item %d ought to be unique type: %s", i+1, unionMemberCodec.typeName)
}
allowedTypes[i] = fullName
codecFromIndex[i] = unionMemberCodec
codecFromName[fullName] = unionMemberCodec
indexFromName[fullName] = i
}
return &Codec{
// NOTE: To support record field default values, union schema set to the
// type name of first member
// TODO: add/change to schemaCanonical below
schemaOriginal: codecFromIndex[0].typeName.fullName,
typeName: &name{"union", nullNamespace},
nativeFromBinary: func(buf []byte) (interface{}, []byte, error) {
var decoded interface{}
var err error
decoded, buf, err = longNativeFromBinary(buf)
if err != nil {
return nil, nil, err
}
index := decoded.(int64) // longDecoder always returns int64, so elide error checking
if index < 0 || index >= int64(len(codecFromIndex)) {
return nil, nil, fmt.Errorf("cannot decode binary union: index ought to be between 0 and %d; read index: %d", len(codecFromIndex)-1, index)
}
c := codecFromIndex[index]
decoded, buf, err = c.nativeFromBinary(buf)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode binary union item %d: %s", index+1, err)
}
if decoded == nil {
// do not wrap a nil value in a map
return nil, buf, nil
}
// Non-nil values are wrapped in a map with single key set to type name of value
return Union(allowedTypes[index], decoded), buf, nil
},
binaryFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
switch v := datum.(type) {
case nil:
index, ok := indexFromName["null"]
if !ok {
return nil, fmt.Errorf("cannot encode binary union: no member schema types support datum: allowed types: %v; received: %T", allowedTypes, datum)
}
return longBinaryFromNative(buf, index)
case map[string]interface{}:
if len(v) != 1 {
return nil, fmt.Errorf("cannot encode binary union: non-nil Union values ought to be specified with Go map[string]interface{}, with single key equal to type name, and value equal to datum value: %v; received: %T", allowedTypes, datum)
}
// will execute exactly once
for key, value := range v {
index, ok := indexFromName[key]
if !ok {
return nil, fmt.Errorf("cannot encode binary union: no member schema types support datum: allowed types: %v; received: %T", allowedTypes, datum)
}
c := codecFromIndex[index]
buf, _ = longBinaryFromNative(buf, index)
return c.binaryFromNative(buf, value)
}
}
return nil, fmt.Errorf("cannot encode binary union: non-nil Union values ought to be specified with Go map[string]interface{}, with single key equal to type name, and value equal to datum value: %v; received: %T", allowedTypes, datum)
},
nativeFromTextual: func(buf []byte) (interface{}, []byte, error) {
if len(buf) >= 4 && bytes.Equal(buf[:4], []byte("null")) {
if _, ok := indexFromName["null"]; ok {
return nil, buf[4:], nil
}
}
var datum interface{}
var err error
datum, buf, err = genericMapTextDecoder(buf, nil, codecFromName)
if err != nil {
return nil, nil, fmt.Errorf("cannot decode textual union: %s", err)
}
return datum, buf, nil
},
textualFromNative: func(buf []byte, datum interface{}) ([]byte, error) {
switch v := datum.(type) {
case nil:
_, ok := indexFromName["null"]
if !ok {
return nil, fmt.Errorf("cannot encode textual union: no member schema types support datum: allowed types: %v; received: %T", allowedTypes, datum)
}
return append(buf, "null"...), nil
case map[string]interface{}:
if len(v) != 1 {
return nil, fmt.Errorf("cannot encode textual union: non-nil Union values ought to be specified with Go map[string]interface{}, with single key equal to type name, and value equal to datum value: %v; received: %T", allowedTypes, datum)
}
// will execute exactly once
for key, value := range v {
index, ok := indexFromName[key]
if !ok {
return nil, fmt.Errorf("cannot encode textual union: no member schema types support datum: allowed types: %v; received: %T", allowedTypes, datum)
}
buf = append(buf, '{')
var err error
buf, err = stringTextualFromNative(buf, key)
if err != nil {
return nil, fmt.Errorf("cannot encode textual union: %s", err)
}
buf = append(buf, ':')
c := codecFromIndex[index]
buf, err = c.textualFromNative(buf, value)
if err != nil {
return nil, fmt.Errorf("cannot encode textual union: %s", err)
}
return append(buf, '}'), nil
}
}
return nil, fmt.Errorf("cannot encode textual union: non-nil values ought to be specified with Go map[string]interface{}, with single key equal to type name, and value equal to datum value: %v; received: %T", allowedTypes, datum)
},
}, nil
}