Skip to content
This repository was archived by the owner on Nov 29, 2023. It is now read-only.
This repository was archived by the owner on Nov 29, 2023. It is now read-only.

default deadline too short to retry based on default timeout #128

Closed

Description

@stevenzhang-support

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

If you are still having issues, please be sure to include as much information as possible:

Environment details

  • OS type and version:
  • Python version: Python 3.6.9
  • pip version: pip 21.1.3
  • google-cloud-bigquery-reservation version: 1.2.0

Steps to reproduce

  1. run the python script below
  2. crontab schedule the python script every 10 mins to repeat the "FLEX slots create-assign-delete" process
  3. check the result and found most of the scheduled process completed with success but a few failed with error stack below.

Code example

from google.cloud import bigquery
#from google.cloud.bigquery.bigquery_reservation_v1 import *
from google.cloud import bigquery_reservation_v1
import time

bq_client = bigquery.Client(project='project ID')
res_client = bigquery_reservation_v1.ReservationServiceClient()
parent = 'projects/{}/locations/{}'.format('project ID', 'US')

# commit to a flex slots purchase, billing begins
commit_config = bigquery_reservation_v1.CapacityCommitment(plan='FLEX', slot_count=200)
commit = res_client.create_capacity_commitment(parent=parent,
                                               capacity_commitment=commit_config)
print('commitment of flex slots created')
time.sleep(100)
# create a slot reservation with a descriptive name
res_config = bigquery_reservation_v1.Reservation(slot_capacity=200, ignore_idle_slots=False)
res = res_client.create_reservation(parent=parent,
                                    reservation_id='zhangsteven200',
                                    reservation=res_config)

# assign the reservation to the project executing the BigQuery jobs
assign_config = bigquery_reservation_v1.Assignment(job_type='QUERY',
                           assignee='projects/{}'.format('prorject ID'))
assign = res_client.create_assignment(parent=res.name, assignment=assign_config)

# Assignments can take up to three minutes to take effect
time.sleep(180)
print('assigned has taken effect')
# execute large jobs at flat rate pricing
query='SELECT * FROM `projectID.zhangsteven.case_support` '
job = bq_client.query(query)
job.result(max_results=1)
print('job has done successfully...')
# remove the assignment, reservation, and capacity commitment
time.sleep(3000)
res_client.delete_assignment(name=assign.name)
print('assigment deleted...')
time.sleep(20)
res_client.delete_reservation(name=res.name)
print('reservation deleted...')
time.sleep(20)
res_client.delete_capacity_commitment(name=commit.name)
print('commitment deleted...')
# example

Stack trace

Traceback (most recent call last):
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "502:Bad Gateway"
        debug_error_string = "{"created":"@1625222974.547476662","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"502:Bad Gateway","grpc_status":14}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "bq_reservation_test2.py", line 13, in <module>
    capacity_commitment=commit_config)
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/cloud/bigquery_reservation_v1/services/reservation_service/client.py", line 911, in create_capacity_commitment
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/zhangsteven/google-cloud/bq_reservation/myvenv/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 502:Bad Gateway

Customer pain:

client's issue is "I want all commands to succeed without having to add exception handling logic to my python script"

Activity

  1. added
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    and removed
    🚨This issue needs some love.
    triage meI really want to be triaged.
    on Jul 19, 2021
  2. tswast commented on Jul 19, 2021

    @tswast
    Contributor

    I believe UNAVAILABLE should be possible to add to default Retry. I'm not 100% certain where this needs to be added in generated clients.

    As a workaround, delete_assignment and all other client methods take a retry argument of type Retry.

  3. tswast commented on Jul 19, 2021

    @tswast
    Contributor

    This is where Retry is currently constructed. Unavailable is not listed as a retryable error there at the moment.

    default_retry=retries.Retry(
    initial=0.1,
    maximum=60.0,
    multiplier=1.3,
    predicate=retries.if_exception_type(
    core_exceptions.DeadlineExceeded,
    core_exceptions.ServiceUnavailable,
    ),
    deadline=60.0,
    ),
    default_timeout=60.0,
    client_info=client_info,

  4. stevenzhang-support commented on Jul 19, 2021

    @stevenzhang-support
    Author

    Thanks for replying to me on this, I appreciate your kindness to consider this to be added.

  5. tswast commented on Nov 2, 2021

    @tswast
    Contributor

    On second look, ServiceUnavailable is listed in the allowed exception types and also here:

    https://github.com/googleapis/googleapis/blob/bcfd386c224fded5f5c5be7a17bc5884a2a52f59/google/cloud/bigquery/reservation/v1/bigqueryreservation_grpc_service_config.json#L95

    That said, the deadline of 60 seconds is not long enough to retry if the original request takes 60 seconds (per the default timeout)

  6. changed the title [-]Python script cannot always guarantee deletion of FLEX slots with error: google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.[/-] [+]default deadline too short to retry based on default timeout[/+] on Nov 2, 2021
  7. tswast commented on Nov 2, 2021

    @tswast
    Contributor

    I've mailed internal change 407167373 to update the default deadline to 5 minutes, which should give the client a chance to retry.

  8. tswast commented on Dec 20, 2021

    @tswast
    Contributor

    Change 407167373 has been merged and committed here: googleapis/googleapis@489719a

    We should be getting a PR with the updated client soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: bigqueryreservationIssues related to the googleapis/python-bigquery-reservation API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions