Skip to content

catch pyarrow.lib.ArrowTypeError for augment_schema #15698

Closed as not planned
Closed as not planned

Description

@j-blackwell

Is your feature request related to a problem? Please describe.
The problematic field name is not returned in the error for the augment_schema function like it is elsewhere. This can then show up in places like load_table_from_dataframe since augment_schema is called.

pyarrow.lib.ArrowTypeError: Expected bytes, got a 'float' object

  File "...", line 57, in load_table_bq
    job = client.load_table_from_dataframe(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/client.py", line 2781, in load_table_from_dataframe
    new_job_config.schema = _pandas_helpers.dataframe_to_bq_schema(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 491, in dataframe_to_bq_schema
    bq_schema_out = augment_schema(dataframe, bq_schema_out)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 520, in augment_schema
    arrow_table = pyarrow.array(dataframe.reset_index()[field.name])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/array.pxi", line 360, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 87, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status

Describe the solution you'd like
Fix could be similar to googleapis/python-bigquery#1836

def augment_schema(dataframe, current_bq_schema):
    ...
    for field in current_bq_schema:
        if field.field_type is not None:
            augmented_schema.append(field)
            continue
        try:
            arrow_table = pyarrow.array(dataframe.reset_index()[field.name])
        except ArrowTypeError:
            msg = f"""Error converting Pandas column with name: "{field.name}" and datatype: "{field.dtype}" to an appropriate pyarrow datatype: ..."""
        _LOGGER.error(msg)
        raise ArrowTypeError(msg)

Happy to submit a PR if this would be approved?

Activity

  1. tswast commented on Feb 18, 2025

    @tswast
    Contributor

    Could you confirm that this also happens in pandas-gbq? I'm working on googleapis/python-bigquery#2095 to align google-cloud-bigquery and pandas-gbq so we only have to make such fixes in one place in future.

  2. j-blackwell commented on Mar 3, 2025

    @j-blackwell
    Author

    Ah nice one - can see that this has been merged now. Am I right in thinking that this issue has been fixed by that merge (or it will become a pandas-gbq issue if not)?

  3. parthea commented on Jan 12, 2026

    @parthea
    Contributor

    Transferring to google-cloud-python as the code for python-bigquery will be migrated there in 1-3 weeks.

  4. parthea commented on Jan 12, 2026

    @parthea
    Contributor

    Transferring this issue back to python-bigquery.

  5. chalmerlowe commented on Mar 2, 2026

    @chalmerlowe
    Contributor

    This issue was moved from the following repo as part of the monorepo migration: googleapis/python-bigquery

  6. removed their assignment
    on Mar 27, 2026
  7. daniel-sanche commented on May 28, 2026

    @daniel-sanche
    Contributor

    This issue is being bulk closed due to inactivity. Please re-open if you think this still requires attention, and we'll take another look. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: bigqueryIssues related to the BigQuery API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions