Repository navigation
BigTable: Table.read_row: StatusCode.INTERNAL, GOAWAY received #4766
Description
Hey, I have application running on Google App Engine. When executing read_row
It raises grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, GOAWAY received).
Full stack trace:
Traceback (most recent call last):
File "/home/vmagent/app/server.py", line 186, in alias
user_alias_id = api.handler.alias(user_id, user_alias_id)
File "/home/vmagent/app/handler.py", line 97, in alias
existing_alias = self.bt.get_identifier(user_id)
File "/home/vmagent/app/bigtable.py", line 99, in get_identifier
row = self.identifier_table.read_row(row_key)
File "/env/lib/python3.6/site-packages/google/cloud/bigtable/table.py", line 267, in read_row
rows_data.consume_all()
File "/env/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 327, in consume_all
self.consume_next()
File "/env/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 261, in consume_next
response = six.next(self._response_iterator)
File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 344, in __next__
return self._next()
File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 338, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, GOAWAY received)>
Any idea why this is happen? I can't reproduce the exception (the exception happen only once).
another question: Is it okay if I set GOOGLE_CLOUD_DISABLE_GRPC=true in production environment?
Thanks!
OS type and version
Google app engine custom runtime
FROM gcr.io/google-appengine/python
Python version
python3.6
Packages
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
google-api-core==0.1.3
google-api-python-client==1.6.4
google-auth==1.2.1
google-cloud==0.32.0
google-cloud-bigquery==0.28.0
google-cloud-bigquery-datatransfer==0.1.0
google-cloud-bigtable==0.28.1
google-cloud-container==0.1.0
google-cloud-core==0.28.0
google-cloud-datastore==1.4.0
google-cloud-dns==0.28.0
google-cloud-error-reporting==0.28.0
google-cloud-firestore==0.28.0
google-cloud-language==1.0.0
google-cloud-logging==1.4.0
google-cloud-monitoring==0.28.0
google-cloud-pubsub==0.30.1
google-cloud-resource-manager==0.28.0
google-cloud-runtimeconfig==0.28.0
google-cloud-spanner==0.29.0
google-cloud-speech==0.30.0
google-cloud-storage==1.6.0
google-cloud-trace==0.17.0
google-cloud-translate==1.3.0
google-cloud-videointelligence==1.0.0
google-cloud-vision==0.29.0
google-gax==0.15.16
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
grpcio==1.8.3
Activity
sduskis commentedon Jan 24, 2018
The server-side disconnects every so often and sends a GOAWAY. I'm quite surprised that it results in a
StatusCode.INTERNAL. I'll have to ask about that.The solution here is automatic retries by the client, which is a long standing Feature Request: #2255. We are actively working on this issue.
pyk commentedon Jan 26, 2018
@sduskis thanks for the explanation, i hope the feature is implemented soon 👍