Reflection (Rich Descriptors) for Go Protocol Buffers
This release adds support for Protobuf Editions to many of the packages. Note, however, that the desc/protoparse
package does not yet work to parse Protobuf Editions source files. This release also includes a handful of bug-fixes.
Additions:
Edition
method to *desc.FileDescriptor
, which returns non-zero for files that use Protobuf Editions (instead of syntax "proto2" or "proto3").Changes/fixes:
Wrap*
functions, that could result in a descriptor hierarchy where the same descriptors were incorrectly present multiple times. This could happen if a file in the transitive dependency graph being wrapped had more than one incoming edge (e.g. same file appearing in import statements from two or more other files).Additions:
Editions
field and SetEditions
method to *builder.FileBuilder
, for creating files that use Protobuf Editions.Changes/fixes:
Printer.CustomSortFunction
was set wherein enum values could be printed in incorrect order, resulting in invalid source output. In syntax "proto3" and in open enums in Protobuf Editions, the first value must be the one with a zero numeric value.Additions:
AllowMissingFileDescriptors
method to *grpcreflect.Client
that allows users to opt-in to a lenient mode that will try to return schemas even when some files are unavailable. This will only be successful in cases where the missing files only provide custom options or are unused imports. This restores some lenience that was inadvertently provided (due to bugs and missing validation checks) when versions of this repo prior to v1.15 were combined with versions of the google.golang.org/grpc
module prior to v1.61.This is a bugfix release to address recently identified regressions in the desc/protoparse
package -- things that worked in v1.14.1 but did not work in v1.15.x.
Changes/fixes:
Parser.ImportPaths
field specifies that the field is not used for calls to the Parser.ParseFilesButDoNotLink
method. This is because import paths do not need to be resolved if not linking. But as of v1.15.0, the ImportPaths
field was being used for that method, and the difference in behavior could break existing programs. This has been fixed, and the behavior once again matches the spec.Parser.ParseFiles
were resolved as descriptors (via the Parser.LookupImport
or Parser.LookupImportProto
functions) instead of as source code (via the Parser.Accessor
function or the default action of os.Open
), the parse operation would fail with a "no such file" error. This would succeed with v1.14.1, with the descriptor being used as the input for the file instead of the file being parsed from source. So working programs could have been broken by an upgrade to v1.15.x. The package has been changed back to the v1.14.1 behavior.This is a bugfix release to address bugs in the desc/builder
, desc/protoparse
, and grpcreflect
packages and a compatibility issue with the most recent version of github.com/bufbuild/protocompile
.
Changes/fixes:
Changes/fixes:
github.com/bufbuild/protocompile
package under the hood as the parser. But that package is still using major version zero, since it's API has not completely stabilized. A recent release (v0.8.0) included backwards-incompatible changes that caused this repo to no longer compile. This release addresses the compile errors: this repo now uses that latest release of protocompile
and correctly compiles against it.desc/protoparse
package that could cause it to incorrectly accept or reject some Protobuf source files. See the release notes for protocompile v0.8.0 for more details.Changes/fixes:
grpcreflect.NewClientAuto
function (added in v1.14.0) returns a client that automatically falls back to using v1alpha of the reflection protocol if the server does not support v1. However, it relied on the server returning an "unimplemented" error to decide to fall back, but this condition has been observed to sometimes instead manifest as an "unavailable" error (in proxies that fail to write a 404 error before closing the connection). In those cases, the client was not correctly falling back to v1alpha. This release remedies that, so it can fallback to v1alpha when this category of error occurs.This is a bugfix release to address bugs in the desc/protoprint
package and a compatibility issue with the most recent version of github.com/bufbuild/protocompile
.
Changes/fixes:
github.com/bufbuild/protocompile
package under the hood as the parser. But that package is still using major version zero, since it's API has not completely stabilized. A recent release (v0.7.0) included backwards-incompatible changes that caused this repo to no longer compile. This release addresses the compile errors: this repo now uses that latest release of protocompile
and correctly compiles against it.Changes/fixes:
"
), it would get printed incorrectly leading to either an incorrect string literal in some cases or an invalid and uncompilable source file in others. This has been fixed.FileDescriptorProto
but never linked, the descriptor may be missing the type
field of a FieldDescriptorProto
(since it is not known whether a named type refers to a message, group, or enum), and it may have options left in the uninterpreted_options
field of the various options messages, which has a peculiar representation for aggregate values. When printing a descriptor that had these characteristics, they were not correctly handled, so the output of the printer would be an invalid and uncompilable source file. Both of these cases have been fixed.This is a bugfix release to address a panics in the desc/builder
and desc/protoparse
packages.
Changes/fixes:
EnumBuilder
had numbers explicitly assigned and others did not, a panic could occur. This panic has now been fixed. When there is a mix of explicit value numbers and implicit numbers, the implicit values will be auto-assigned available numbers (starting at zero) while the explicit values will use the configured number.Changes/fixes:
protoparse
package was overhauled in v1.15.0, a bug was inadvertently introduced to the functionality related to the Parser.InferImportPaths
flag. If the files being processed imported a file for which an AST was not available (which is actually quite common, since they may import a standard file such as "google/protobuf/empty.proto"), a panic would occur. This panic has been fixed.Parser.InferImportPaths
flag which now allow it to better detect file layouts and remedy import mismatches in more scenarios.This is a bugfix release to address issues found in the desc/protoparse
package.
Changes/fixes:
Parser
were updated in this fashion. Other files in the transitive dependency (i.e. imports) did not have options represented this way. As of this release, the fix is complete and all descriptors returned by this package should have the same representation for custom options as in v1.14 and earlier.github.com/bufbuild/protocompile
package under the hood as the parser. But that package is still using major version zero, since it's API has not completely stabilized. A recent release (v0.6.0) included backwards-incompatible changes that caused this repo to no longer compile. This release addresses the compile errors: this repo now uses that latest release of protocompile
and correctly compiles against it.This is small bugfix release to address regressions reported in v1.15.0.
It also fixes a panic bug that can occur when a program includes both the grpcreflect package from this module and v1.52.0 or newer of the "google.golang.org/grpc" module.
Changes/fixes:
Parser.LookupImport
or Parser.LookupImportProto
field to supply some dependencies as descriptors instead of as source. The race occurred when the same descriptor was referenced from different, concurrent parse operations: one goroutine would modify the descriptor during linking, while another goroutine was trying to access it (without synchronization). It has now been fixed (via a fix in the underlying protocompile
library: https://github.com/bufbuild/protocompile/pull/103).Changes/fixes:
grpc/reflection/v1/reflection.proto
. This was because, historically, there were no released Go packages that included it. But as of v1.52.0 of the grpc-go runtime, another package now provides this. If both packages are linked into the same program, a panic will occur during program initialization due to there being multiple, conflicting definitions of these types. As of v1.57.0 of grpc-go, this init-time panic is guaranteed to happen. This panic is resolved in this version (the internal copy is now registered using a non-conflicting name). An example panic stack-trace follows:
panic: proto: file "reflection.proto" has a name conflict over grpc.reflection.v1.ErrorResponse
previously from: "google.golang.org/grpc/reflection/grpc_reflection_v1"
currently from: "github.com/jhump/protoreflect/grpcreflect/internal/grpc_reflection_v1"
See https://protobuf.dev/reference/go/faq#namespace-conflict
goroutine 1 [running]:
google.golang.org/protobuf/reflect/protoregistry.glob..func1({0x1075580e0?, 0xc0005e2800?}, {0x1075580e0?, 0xc0005e2840})
/.../go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:56 +0x264
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile.func1({0x107569940, 0xc0006fc5b0})
/.../go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:154 +0x2ec
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x10756d6c8, 0xc0004e76c0}, 0xc00071ce28)
/.../go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:417 +0x148
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0005ce258, {0x10756d6c8?, 0xc0004e76c0?})
/.../go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:149 +0x708
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1072414ee, 0x45}, {0x108070e00, 0x6bd, 0x6bd}, 0x0, 0x8, 0x0, 0x1, {0x10755ec98, ...}, ...})
/.../go/pkg/mod/google.golang.org/[email protected]/internal/filedesc/build.go:112 +0x22c
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1072414ee, 0x45}, {0x108070e00, 0x6bd, 0x6bd}, 0x0, 0x8, 0x0, 0x1, {0x0, ...}, ...}, ...})
/.../go/pkg/mod/google.golang.org/[email protected]/internal/filetype/build.go:138 +0x22c
github.com/jhump/protoreflect/grpcreflect/internal/grpc_reflection_v1.file_reflection_proto_init()
/.../go/pkg/mod/github.com/jhump/[email protected]/grpcreflect/internal/grpc_reflection_v1/reflection.pb.go:947 +0x558
github.com/jhump/protoreflect/grpcreflect/internal/grpc_reflection_v1.init.0()
/.../go/pkg/mod/github.com/jhump/[email protected]/grpcreflect/internal/grpc_reflection_v1/reflection.pb.go:818 +0x24
This is a significant and admittedly long overdue release. This release finally provides a bridge for interoperability between packages and functionality in this repo and the new reflection packages and APIs added in the "v2 API" of the Protobuf runtime for Go (google.golang.org/protobuf
), particularly the protoreflect, protodesc, protoregistry, and dynamicpb packages.
This repo began because the Protobuf runtime (at the time, the "v1 API" in github.com/golang/protobuf
) had no support for Protobuf reflection and dynamic messages. The functionality slowly grew to include dynamic gRPC, interesting meta features like parsing Protobuf sources or printing Protobuf source from a descriptor, etc.
But the v2 API was released nearly three years ago. And it does provide some functionality that this repo provided, but now in the core runtime library. And yet this repo still refers to the now-deprecated v1 API.
Under the hood, this is a very big change, mainly due to the big changes (described below) in the github.com/jhump/protoreflect/desc
and github.com/jhump/protoreflect/desc/protoparse
packages.
This release is planned to be the final v1 release. Any other changes significant enough to warrant a new minor version will instead be added in an upcoming v2 version. A new v2 version will have zero references to the v1 API of the Protobuf runtime, and it will not duplicate any of the functionality that now exists in the v2 API.
The changes are all detailed below and include all of the changes described in the past two release candidates (v1.15.0-rc1 and v1.15.0-rc2).
NOTE: Zero defects were reported in the nearly six weeks since the first release candidate was created. It is entirely possible that very few users tried it, so there still could be some material bugs and compatibility issues lurking in this release. If you encounter any marked changes in behavior between the previous release (v1.14.1) and this one, please file an issue as soon as possible! However, if the issue is a performance degradation, it may not be addressed until a v2. So If your use case is very sensitive to performance, you may need to stay on v1.14.1. The under-the-hood changes in this repo for this release are expected to have some performance impact, though it will hopefully be minor or even negligible for most users.
Changes/fixes:
google.golang.org/protobuf/reflect/protoreflect
package.
protoreflect.Descriptor
to a desc.Descriptor
(via the various Wrap*
functions in this package). It is similarly easy to unwrap a desc.Descriptor
value, to recover the underlying protoreflectDescriptor
.protoreflect
and accompanying packages. You can take a protoreflect.Descriptor
and then easily use it with the desc/builder
and desc/protoprint
packages in this repo. Similarly, you can create a desc.Descriptor
using the desc/protoparse
package and easily turn that into a protoreflect.Descriptor
, for use with the google.golang.org/protobuf/...
packages.Additions:
DescriptorWrapper
interface, which is implemented by all descriptor implementations in this package. It contains an Unwrap() protoreflect.Descriptor
function, to recover the underlying protoreflect.Descriptor
.*desc.FileDescriptor
has a method UnwrapFile() protoreflect.FileDescriptor
.Wrap*
functions, which accept a protoreflect.Descriptor
and wrap it, returning a desc.Descriptor
. There is one function for each concrete type, for example for messages there is WrapMessage(d protoreflect.MessageDescriptor) (*MessageDescriptor, error)
Changes/fixes:
desc.Descriptor
values returned by this package are now backed by protoreflect.Descriptor
values, more rules are enforced. This is because the implementation of descriptors in the protoreflect
package does perform all of those validation checks. The new checks that are now enforced that previously were not:
Changes/fixes:
desc
package. The implementation in this package has been completely replaced with the functionality of the github.com/bufbuild/protocompile
package. So the exported APIs in this package are now just adapters. The actual parser/compiler is implemented in this other dependency. Most of this adaptation logic is trivial with the exception being for the Parser.ParseToAST
method, which must convert from protocompile
's AST model to the model defined in desc/protoparse/ast
.Changes/fixes:
{
and }
) instead of angle brackets (<
and >
), which is better aligned with the commonly-used and preferred syntax style.Printer.MessageLiteralExpansionThresholdLength
, instead of this threshold only being applied for a top-level message. So small nested messages can be emitted in compact form, even if enclosed within a message literal that has been expanded.Printer
, then elements with multiple option declarations would have those options printed in seemingly random order. The actual order was based on a map iteration, and was thus non-deterministic. This has been fixed. In such a case, options will now be ordered by name.Additions:
TypeResolver
interface has been added, as well as a GlobalTypes
package variable that implements that interface. This is analogous to protoregistry.GlobalTypes
, just as the existing sourceinfo.GlobalFiles
corresponds to protoregistry.GlobalFiles
.Wrap*
functions, that accept various protoreflect
types and return values that are identical to the input values except that they include source code info that was registered with this package.Changes/fixes:
protoreflect
APIs were used to store a value in an extension field, using a protoreflect.FieldDescriptor
that was not a generated extension (i.e. not known at compile-time), trying to convert the resulting message to a *dynamic.Message
would overlook that field; it would be absent from the result, not even appearing in the unrecognized fields. This has been fixed. Such extension values will be in the resulting *dynamic.Message
and appear as recognized/known fields.This is still a release candidate due to the significance of changes to under-the-hood machinery. But we expect to cut a proper v1.15.0 soon.
In addition to all of the many changes in v1.15.0-rc1, this release includes the following:
Changes/fixes:
protocompile
package implementation. These were innocuous-looking issues that could become extreme if given the right kind of pathological input. These issues were fixed, and the fixed version of protocompile
has been pulled into this repo.Changes/fixes:
Printer
, then elements with multiple option declarations would have those options printed in seemingly random order. The actual order was based on a map iteration, and was thus non-deterministic. This has been fixed. In such a case, options will now be ordered by name.This is a very big change under-the-hood, which is why this is a release candidate. Unfortunately, go get -u ...
and tools like Depend-a-bot will not take release candidates/pre-release versions. So this candidate is a sort of shot in the dark, hoping that some brave souls might try it out and provide feedback before I stamp it as an official release.
All of CI is green, I've run apidiff
to make sure there are no backwards-incompatible changes in the exported API, and I've done some other exploratory testing to make sure this is all good. My hope is that no one will notice a thing. If they do notice something, it might be a slight performance degradation, but hopefully not consequential enough to be a problem.
I expect v1.15 to be the last v1 version. Any other changes significant enough to warrant a new minor version will instead be added in an upcoming v2 version. This version is intended to be a stop-gap between v1 and v2 as it provides a long-needed bridge between the functionality in this repo and the newer google.golang.org/protobuf/reflect/protoreflect package (and its accompanying packages, protodesc, protoregistry, and dynamicpb).
Changes/fixes:
google.golang.org/protobuf/reflect/protoreflect
package.
protoreflect.Descriptor
to a desc.Descriptor
(via the various Wrap*
functions in this package). It is similarly easy to unwrap a desc.Descriptor
value, to recover the underlying protoreflectDescriptor
.protoreflect
and accompanying packages. You can take a protoreflect.Descriptor
and then easily use it with the desc/builder
and desc/protoprint
packages in this repo. Similarly, you can create a desc.Descriptor
using the desc/protoparse
package and easily turn that into a protoreflect.Descriptor
, for use with the google.golang.org/protobuf/...
packages.Additions:
DescriptorWrapper
interface, which is implemented by all descriptor implementations in this package. It contains an Unwrap() protoreflect.Descriptor
function, to recover the underlying protoreflect.Descriptor
.*desc.FileDescriptor
has a method UnwrapFile() protoreflect.FileDescriptor
.Wrap*
functions, which accept a protoreflect.Descriptor
and wrap it, returning a desc.Descriptor
. There is one function for each concrete type, for example for messages there is WrapMessage(d protoreflect.MessageDescriptor) (*MessageDescriptor, error)
Changes/fixes:
desc.Descriptor
values returned by this package are now backed by protoreflect.Descriptor
values, more rules are enforced. This is because the implementation of descriptors in the protoreflect
package does perform all of those validation checks. The new checks that are now enforced that previously were not:
Changes/fixes:
desc
package. The implementation in this package has been completely replaced with the functionality of the github.com/bufbuild/protocompile
package. So the exported APIs in this package are now just adapters. The actual parser/compiler is implemented in this other dependency. Most of this adaptation logic is trivial with the exception being for the Parser.ParseToAST
method, which must convert from protocompile
's AST model to the model defined in desc/protoparse/ast
.Changes/fixes:
{
and }
) instead of angle brackets (<
and >
), which is better aligned with the commonly-used and preferred syntax style.Printer.MessageLiteralExpansionThresholdLength
, instead of this threshold only being applied for a top-level message. So small nested messages can be emitted in compact form, even if enclosed within a message literal that has been expanded.Additions:
TypeResolver
interface has been added, as well as a GlobalTypes
package variable that implements that interface. This is analogous to protoregistry.GlobalTypes
, just as the existing sourceinfo.GlobalFiles
corresponds to protoregistry.GlobalFiles
.Wrap*
functions, that accept various protoreflect
types and return values that are identical to the input values except that they include source code info that was registered with this package.Changes/fixes:
protoreflect
APIs were used to store a value in an extension field, using a protoreflect.FieldDescriptor
that was not a generated extension (i.e. not known at compile-time), trying to convert the resulting message to a *dynamic.Message
would overlook that field; it would be absent from the result, not even appearing in the unrecognized fields. This has been fixed. Such extension values will be in the resulting *dynamic.Message
and appear as recognized/known fields.