This repository was archived by the owner on Nov 29, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 29, 2023. It is now read-only.
default deadline too short to retry based on default timeout #128
Copy link
Copy link
Closed
Labels
api: bigqueryreservationIssues related to the googleapis/python-bigquery-reservation API.Issues related to the googleapis/python-bigquery-reservation API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.
Description
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/python-bigquery-reservation/issues
- Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python
If you are still having issues, please be sure to include as much information as possible:
Environment details
- OS type and version:
- Python version: Python 3.6.9
- pip version: pip 21.1.3
- google-cloud-bigquery-reservation version: 1.2.0
Steps to reproduce
- run the python script below
- crontab schedule the python script every 10 mins to repeat the "FLEX slots create-assign-delete" process
- check the result and found most of the scheduled process completed with success but a few failed with error stack below.
Code example
from google.cloud import bigquery
#from google.cloud.bigquery.bigquery_reservation_v1 import *
from google.cloud import bigquery_reservation_v1
import time
bq_client = bigquery.Client(project='project ID')
res_client = bigquery_reservation_v1.ReservationServiceClient()
parent = 'projects/{}/locations/{}'.format('project ID', 'US')
# commit to a flex slots purchase, billing begins
commit_config = bigquery_reservation_v1.CapacityCommitment(plan='FLEX', slot_count=200)
commit = res_client.create_capacity_commitment(parent=parent,
capacity_commitment=commit_config)
print('commitment of flex slots created')
time.sleep(100)
# create a slot reservation with a descriptive name
res_config = bigquery_reservation_v1.Reservation(slot_capacity=200, ignore_idle_slots=False)
res = res_client.create_reservation(parent=parent,
reservation_id='zhangsteven200',
reservation=res_config)
# assign the reservation to the project executing the BigQuery jobs
assign_config = bigquery_reservation_v1.Assignment(job_type='QUERY',
assignee='projects/{}'.format('prorject ID'))
assign = res_client.create_assignment(parent=res.name, assignment=assign_config)
# Assignments can take up to three minutes to take effect
time.sleep(180)
print('assigned has taken effect')
# execute large jobs at flat rate pricing
query='SELECT * FROM `projectID.zhangsteven.case_support` '
job = bq_client.query(query)
job.result(max_results=1)
print('job has done successfully...')
# remove the assignment, reservation, and capacity commitment
time.sleep(3000)
res_client.delete_assignment(name=assign.name)
print('assigment deleted...')
time.sleep(20)
res_client.delete_reservation(name=res.name)
print('reservation deleted...')
time.sleep(20)
res_client.delete_capacity_commitment(name=commit.name)
print('commitment deleted...')
# example
Stack trace
Traceback (most recent call last):
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "502:Bad Gateway"
debug_error_string = "{"created":"@1625222974.547476662","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"502:Bad Gateway","grpc_status":14}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "bq_reservation_test2.py", line 13, in <module>
capacity_commitment=commit_config)
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/cloud/bigquery_reservation_v1/services/reservation_service/client.py", line 911, in create_capacity_commitment
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 502:Bad Gateway
Customer pain:
client's issue is "I want all commands to succeed without having to add exception handling logic to my python script"
Metadata
Metadata
Assignees
Labels
api: bigqueryreservationIssues related to the googleapis/python-bigquery-reservation API.Issues related to the googleapis/python-bigquery-reservation API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.
Activity
tswast commentedon Jul 19, 2021
I believe UNAVAILABLE should be possible to add to default Retry. I'm not 100% certain where this needs to be added in generated clients.
As a workaround, delete_assignment and all other client methods take a
retryargument of type Retry.tswast commentedon Jul 19, 2021
This is where Retry is currently constructed. Unavailable is not listed as a retryable error there at the moment.
python-bigquery-reservation/google/cloud/bigquery_reservation_v1/services/reservation_service/transports/base.py
Lines 167 to 178 in 32b279f
stevenzhang-support commentedon Jul 19, 2021
Thanks for replying to me on this, I appreciate your kindness to consider this to be added.
tswast commentedon Nov 2, 2021
On second look,
ServiceUnavailableis listed in the allowed exception types and also here:https://github.com/googleapis/googleapis/blob/bcfd386c224fded5f5c5be7a17bc5884a2a52f59/google/cloud/bigquery/reservation/v1/bigqueryreservation_grpc_service_config.json#L95
That said, the deadline of 60 seconds is not long enough to retry if the original request takes 60 seconds (per the default timeout)
python-bigquery-reservation/google/cloud/bigquery_reservation_v1/services/reservation_service/transports/base.py
Line 225 in 9105981
[-]Python script cannot always guarantee deletion of FLEX slots with error: google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.[/-][+]default deadline too short to retry based on default timeout[/+]tswast commentedon Nov 2, 2021
I've mailed internal change 407167373 to update the default deadline to 5 minutes, which should give the client a chance to retry.
tswast commentedon Dec 20, 2021
Change 407167373 has been merged and committed here: googleapis/googleapis@489719a
We should be getting a PR with the updated client soon.