BigQuery DataFrames (also known as BigFrames) provides a Pythonic DataFrame and machine learning (ML) API powered by the BigQuery engine. It provides modules for many use cases, including:

• bigframes.pandas is a pandas API for analytics. Many workloads can be migrated from pandas to bigframes by just changing a few imports.
• bigframes.bigquery.ai provides powerful AI functions, text generation, vector embeddings, and semantic search powered by Gemini.
• bigframes.ml is a scikit-learn-like API for ML.

BigQuery DataFrames is an open-source package.

BigQuery DataFrames eliminates the "data movement bottleneck." Instead of downloading large datasets to a local environment, BigFrames translates your Python code into optimized SQL executed across BigQuery's distributed compute engine.

• Petabyte-Scale Scalability: Effortlessly process datasets exceeding local memory constraints without provisioning or managing Spark or Ray clusters.
• Familiar Python Ecosystem: Use the pandas operations you already know (read_gbq, groupby, merge, pivot_table) directly on BigQuery tables.
• Generative AI & Multimodality: Seamlessly leverage Gemini models, AI functions, multimodal processing (audio, video, PDF), vector embeddings, and vector search with bigframes.bigquery.ai.
• Optimized Execution: Take advantage of BigFrames partial ordering mode (bpd.options.bigquery.ordering_mode = "partial") for considerable performance gains on analytical workloads.
• Enterprise Security: Keep your data securely governed within the BigQuery security perimeter.
• Hybrid Flexibility: Seamlessly transition between distributed BigQuery processing and local pandas analysis with to_pandas() and read_pandas().

The easiest way to get started is to open a sample notebook in Colab or BigQuery Studio. More details can be found in the BigFrames quickstart and in the introduction to notebooks.

To use BigFrames in your local development environment:

1. Run pip install --upgrade bigframes to install the latest version.
2. Set up Application Default Credentials for your local development environment.
3. Create a Google Cloud project with the BigQuery API enabled.
4. Use the bigframes package to query data.

import bigframes.pandas as bpd

bpd.options.bigquery.project = your_gcp_project_id  # Optional in BQ Studio.
bpd.options.bigquery.ordering_mode = "partial"  # Recommended for performance.
df = bpd.read_gbq("bigquery-public-data.usa_names.usa_1910_2013")
print(
    df.groupby("name")
    .agg({"number": "sum"})
    .sort_values("number", ascending=False)
    .head(10)
    .to_pandas()
)

To learn more about BigQuery DataFrames, visit these pages

• Introduction to BigQuery DataFrames (BigFrames)
• Sample notebooks
• API reference
• Source code (GitHub)

BigQuery DataFrames is distributed with the Apache-2.0 license.

It also contains code derived from the following third-party packages:

• Ibis
• pandas
• Python
• scikit-learn
• XGBoost
• SQLGlot

For details, see the third_party directory.

For further help and provide feedback, you can email us at [email protected].