Repository navigation
Closed as not planned
Labels
api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.
Description
Is your feature request related to a problem? Please describe.
The problematic field name is not returned in the error for the augment_schema function like it is elsewhere. This can then show up in places like load_table_from_dataframe since augment_schema is called.
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'float' object
File "...", line 57, in load_table_bq
job = client.load_table_from_dataframe(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/client.py", line 2781, in load_table_from_dataframe
new_job_config.schema = _pandas_helpers.dataframe_to_bq_schema(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 491, in dataframe_to_bq_schema
bq_schema_out = augment_schema(dataframe, bq_schema_out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../.venv/lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 520, in augment_schema
arrow_table = pyarrow.array(dataframe.reset_index()[field.name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 360, in pyarrow.lib.array
File "pyarrow/array.pxi", line 87, in pyarrow.lib._ndarray_to_array
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
Describe the solution you'd like
Fix could be similar to googleapis/python-bigquery#1836
def augment_schema(dataframe, current_bq_schema):
...
for field in current_bq_schema:
if field.field_type is not None:
augmented_schema.append(field)
continue
try:
arrow_table = pyarrow.array(dataframe.reset_index()[field.name])
except ArrowTypeError:
msg = f"""Error converting Pandas column with name: "{field.name}" and datatype: "{field.dtype}" to an appropriate pyarrow datatype: ..."""
_LOGGER.error(msg)
raise ArrowTypeError(msg)
Happy to submit a PR if this would be approved?
Metadata
Metadata
Assignees
Labels
api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.
Activity
tswast commentedon Feb 18, 2025
Could you confirm that this also happens in
pandas-gbq? I'm working on googleapis/python-bigquery#2095 to aligngoogle-cloud-bigqueryandpandas-gbqso we only have to make such fixes in one place in future.j-blackwell commentedon Mar 3, 2025
Ah nice one - can see that this has been merged now. Am I right in thinking that this issue has been fixed by that merge (or it will become a
pandas-gbqissue if not)?parthea commentedon Jan 12, 2026
Transferring to google-cloud-python as the code for python-bigquery will be migrated there in 1-3 weeks.
parthea commentedon Jan 12, 2026
Transferring this issue back to python-bigquery.
chalmerlowe commentedon Mar 2, 2026
This issue was moved from the following repo as part of the monorepo migration: googleapis/python-bigquery
daniel-sanche commentedon May 28, 2026
This issue is being bulk closed due to inactivity. Please re-open if you think this still requires attention, and we'll take another look. Thanks!