Why Choose BigQuery DataFrames?#

BigFrames eliminates the “data movement bottleneck.” Instead of downloading large datasets to a local environment, BigFrames translates your Python code into optimized SQL, executing complex transformations across the BigQuery fleet.

• Petabyte-Scale Scalability: Effortlessly process datasets that far exceed local memory limits.

• Familiar Python Ecosystem: Use the same read_gbq, groupby, merge, and pivot_table functions you already know from pandas.

• Generative AI and Machine Learning: Seamlessly leverage Gemini models, AI functions, and vector search with bigframes.bigquery.ai, alongside BigQuery ML’s powerful algorithms via a scikit-learn-compatible interface (bigframes.ml).

• Enterprise-Grade Security: Maintain data governance and security by keeping your data within the BigQuery perimeter.

• Hybrid Flexibility: Easily move between distributed BigQuery processing and local pandas analysis with to_pandas().

Core Components of BigFrames#

BigQuery DataFrames is organized into specialized modules designed for the modern data stack:

1. bigframes.pandas: A high-performance, pandas-compatible API for scalable data exploration, cleaning, and transformation.

2. bigframes.bigquery: Specialized utilities for direct BigQuery resource management, including integrations with Gemini and other AI models in the bigframes.bigquery.ai submodule.

Quickstart: Scalable Data Analysis in Seconds#

Install BigQuery DataFrames via pip:

pip install --upgrade bigframes

The following example demonstrates how to perform a distributed aggregation on a public dataset with millions of rows using just a few lines of Python:

import bigframes.pandas as bpd

# If running in your local environment or Colab, uncomment these lines and add your GCP project ID
# PROJECT_ID = "bigframes-dev"
# bpd.options.bigquery.project = PROJECT_ID

# Initialize BigFrames and load a public dataset
df = bpd.read_gbq("bigquery-public-data.usa_names.usa_1910_2013")

# Perform familiar pandas operations that execute in the cloud
top_names = (
    df.groupby("name")
    .agg({"number": "sum"})
    .sort_values("number", ascending=False)
    .head(10)
)

# Bring the final, aggregated results back to local memory if needed
print(top_names.to_pandas())

Sample Notebooks and Interactive Demos#

Explore sample notebooks demonstrating end-to-end workflows across analytics, GenAI, and machine learning. Each notebook can be launched directly in BigQuery Studio or Consumer Colab:

Browse the complete notebooks catalog on GitHub for additional tutorials and community examples.

Articles and Guides#

To learn more about BigQuery DataFrames architecture, features, and best practices, check out the following articles and documentation guides:

• BigQuery DataFrames: Generally Available: Official announcement detailing petabyte-scale pandas and scikit-learn on BigQuery.

• Generative AI in BigQuery with DataFrames: How to leverage Gemini models and AI functions directly in Python.

• Analyze Multimodal Data with BigQuery DataFrames: End-to-end unstructured data analysis across images, audio, and PDF documents.

• Scalable Data Science with BigQuery DataFrames: Best practices for scaling analytics and machine learning workflows in the cloud.

• BigQuery DataFrames Introduction (Cloud Docs): Official documentation covering data manipulation, sessions, and deployment.

• Try BigQuery DataFrames (Quickstart Guide): Step-by-step tutorial in BigQuery Studio and Colab.

Explore the Documentation#