Metadata key contains illegal characters

We've started noticing a new error in production (and in staging env also) that happens ~50 times a day for us (for many thousands or maybe even millions of calls per day, don't have exact number). It looks like this in a stacktrace:


unhandledRejection:  { 
Error: Metadata key "a" contains illegal characters     
at Http2CallStream.call.on (/ncbackend3/node_modules/@grpc/grpc-js/build/src/client.js:102:45)     
at Http2CallStream.emit (events.js:194:15)     
at Http2CallStream.EventEmitter.emit (domain.js:459:23)     
at Http2CallStream.endCall (/ncbackend3/node_modules/@grpc/grpc-js/build/src/call-stream.js:74:18)     
at ClientHttp2Stream.stream.on (/ncbackend3/node_modules/@grpc/grpc-js/build/src/call-stream.js:205:30)     
at ClientHttp2Stream.emit (events.js:189:13)     
at ClientHttp2Stream.EventEmitter.emit (domain.js:459:23)     at emit (internal/http2/core.js:236:8)     
at process._tickCallback (internal/process/next_tick.js:63:19)  
 code: 2,   
details:    'Metadata key "a\u0000\u0000\u0000" contains illegal characters',   
metadata: Metadata { internalRepr: Map {} },   
note:    'Exception occurred in retry method that was not classified as transient' }

I've even seen this error myself once while calling a certain endpoint that was doing 1 query and 2-3 getById calls to Datastore. So, no saves was made, only querying.

This error is not reproducible all the time, as I mentioned, rather "rare", but still noticeable (~50 hard errors every day, that end up in http 500 errors for our api users).

I don't know how to dig further and provide other helpful info. Important to note that it only happened after updating from ^3.0.0 (latest version in 3.x.x) to 4.0.0. I'm assuming the error could somewhere downstream (in grpc-js itself) that was updated as a sub-dependency. We're on the latest version of everything as of today (yarn upgrade was done and we have no deps pinned).

ps: forgot to add that the "key" in the metadata has different values in each other error (we see it in Sentry), examples (err.message here):

Metadata key "a" contains illegal characters
Metadata key " �û·" contains illegal characters
Metadata key " �`ñ" contains illegal characters
Metadata key " r�ð" contains illegal characters
Metadata key " òe×" contains illegal characters
Metadata key " �wâ" contains illegal characters
...

pps: we were using .runQuery() method with 2 simple filters by number fields and .get(key) (that's what I called getById). Same code in 3.x.x version worked just fine.

Noticed that a occured more often than others.

Environment details

OS: node:10-alpine official docker image
Node.js version: 10.15.3
npm version: not applicable
@google-cloud/datastore version: 4.0.0

Just confirming I'm seeing this frequently as well, but also not reproducibly. Retrying the request in question almost always succeeds. It happens regardless of whether there are simultaneous parallel requests.

Our project contains get, runQuery, and upsert calls, and I've seen it happen with all of them.

I think we'll be forced to downgrade to 3.x due to this soon. Too many errors that affect users...

@alexander-fenster

@alexander-fenster I wonder if this could relate to our upgrade to grpc-js?

@alexander-fenster

@alexander-fenster I wonder if this could relate to our upgrade to grpc-js?

It's probably not, cause we did this sequence of things:

Downgraded to @google-cloud/datastore@^3.0.0. Error was gone.
Upgraded to @grpc/grpc-js@0.4.1 (the one that was unpublished later) and it was working fine for 2 days (no "metadata errors" or new errors).

ps: After I've just written this, I realized I'm not 100% sure. Tried to trace changes via yarn.lock commit history, but the dependency tree between google libs is more complicated to grasp it in few seconds. If it's important - I can try to trace more carefully. But I see multiple of versions of grpc-js in my lock due to different google libs installed.

@kirillgroshkov

@kirillgroshkov @kmontag The main difference between v3.x and v4.x under the hood is the replacement of the original grpc (which uses C++ code compiled into Node.js binary) with the new @grpc/grpc-js. The error you folks see comes from grpc-js so it might be totally possible that the upgrade is causing this. Similar metadata check exists in grpc as well but it has completely different logic.

The good news is that you can keep using the C++ grpc for the time being by passing it to the client constructor:

npm install grpc

const grpc = require('grpc'); // not '@grpc/grpc-js'
const {Datastore} = require('@google-cloud/datastore');
const projectId = 'YOUR_PROJECT_ID';
const datastore = new Datastore({
  projectId,
  grpc   // <-- passing it down to the client
});

I would really appreciate it if you folks try that and update this issue with the results (e.g. if the errors disappear or not). This will help us report a bug to gRPC developers.

Thanks!

@murgatroid99

@murgatroid99 @nicolasnoble This is FYI for now but will likely be upgraded to a real @grpc/grpc-js issue :)

A quick clarification: please upgrade back to v4.0.0 before substituting grpc.

At the moment, the best possible thing to help figure this out would be a tcpdump of a stream that has this error, preferably not all encryped with TLS. If anyone could manage to get one of those and share it here, that would be very helpful.

@kirillgroshkov

@kirillgroshkov Technically you're right and we need to bump it here as well (@bcoe can you please do it?), but if you check how it is used, you'll see it's here just for TypeScript types and no actual code from @grpc/grpc-js is used directly in datastore library. It should be fine just with the google-gax update that would pull @grpc/grpc-js v0.5.2 for itself.

Deployed it yesterday. So far 0 errors, looks promising.

@kirillgroshkov

@kirillgroshkov Are you able to see any warnings (console.warn) in your setup? If so, feel free to update the issue with the warnings printed. We still need to collect some data to have more information for the upstream issue.

This issue has been hitting us too for the past 20 days (around 2500 occurrences of that error so far that I can tell), so we'll try to update and see if it fixes the issue at our side too.

@kirillgroshkov

@kirillgroshkov Are you able to see any warnings (console.warn) in your setup? If so, feel free to update the issue with the warnings printed. We still need to collect some data to have more information for the upstream issue.

I'm unable to check console logs in our current system. But 8 days since my last post - it runs stable, original issue is not reproduced since.

It is fixed in out application too. We see no more of these:

Error: Metadata key "a???" contains illegal characters     
at Http2CallStream.call.on (/ncbackend3/node_modules/@grpc/grpc-js/build/src/client.js:102:45)

however, we apparently traded it for a different Error now:

Error
    at Http2CallStream.call.on (/app/heimdall/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:45)

which is happening since that update.

@hermanbanken

Seeing the same. It happens very frequently now too so really unsure what to do. Testing with @google-cloud/datastore@3 again. All I get is the same Http2Callstream.. related stacktrace so I am flying a bit blind.

@hermanbanken any luck since your last comment?

@JustinBeckwith @schmidt-sebastian is this related to: firebase/firebase-functions#536 (comment)

The amount of occurrences those new errors is much lower, so it is kind of an improvement.

The console.warn in the original 'fix' is kinda annoying though, because it ends up in stderr as text, which also ends up in StackDriver. At least the function is no longer terminated by the throw...

Your fix was to bump @grpc/grpc-js to v0.5.2?

$ npm ls @grpc/grpc-js
myapp
├─┬ @google-cloud/datastore@3.1.2
│ └─┬ google-gax@0.25.6
│   └── @grpc/grpc-js@0.3.6
└─┬ @google-cloud/logging-bunyan@0.10.1
  └─┬ @google-cloud/logging@4.5.2
    └─┬ google-gax@0.25.6
      └── @grpc/grpc-js@0.3.6

I dropped back down to 3 and am not seeing issues in the last hour or so

This sounds like the same issue that we fixed in googleapis/nodejs-firestore#697. The fix was to update google-gax to 1.1.2. I am however not certain that it will be a drop in replacement here since your dependencies still shows gax at 0.25.6.

chore(deps): update dependency @grpc/grpc-js to ^0.5.2 (#532)

Navigation Menu

Repository navigation

Metadata key contains illegal characters #415

Environment details

40 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Metadata key contains illegal characters #415

Description

Environment details

Activity

kmontag commented on May 24, 2019

kirillgroshkov commented on May 27, 2019

bcoe commented on Jun 5, 2019

kirillgroshkov commented on Jun 5, 2019

alexander-fenster commented on Jun 5, 2019

alexander-fenster commented on Jun 5, 2019

alexander-fenster commented on Jun 5, 2019

murgatroid99 commented on Jun 6, 2019

40 remaining items

alexander-fenster commented on Jul 16, 2019

kirillgroshkov commented on Jul 18, 2019

alexander-fenster commented on Jul 18, 2019

hermanbanken commented on Jul 26, 2019

kirillgroshkov commented on Jul 26, 2019

hermanbanken commented on Jul 30, 2019

shaunwarman commented on Aug 12, 2019

hermanbanken commented on Aug 12, 2019

shaunwarman commented on Aug 12, 2019

schmidt-sebastian commented on Aug 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions