Streamlining Your Analytics Pipeline: Exporting GA4 Data to BigQuery, Running DBT Models, and Visualizing with Looker Studio

Photo by Wynand Uys on Unsplash

Streamlining Your Analytics Pipeline: Exporting GA4 Data to BigQuery, Running DBT Models, and Visualizing with Looker Studio

GA4 has issues, luckily there's a straightforward solution

·

4 min read

Problem

GA4 is great until it isn't. There are 2 big issues with standard (free) GA4, especially if you use Looker Studio:

  • GA4 API Quota Limits

  • 14-Month Data Retention Policy


GA4 API Quota Limits

If your organization accesses GA4 data through Looker Studio you have probably encountered this error:

Accessing GA4's API via Looker Studio is limited by tokens, there is a certain allowable amount. Tokens are consumed each time a chart or visualization is loaded, pivoted or filtered.

There is some caching that is happening behind the scenes but if you are a power user of Looker Studio and or have multiple stakeholders accessing reports, you can very quickly run into these limits, halting your analytic analyses.

Standard GA4 Limits

FrequencyQuota NameAllowed Tokens
HourlyPer Property Per Project1,750
HourlyPer Property5,000
DailyPer Property25,000

In other words, a single GA4 property is only allowed 5,000 tokens per hour. Loading 1 page of 5 visualizations with basic complexity in Looker consumes about ~500 tokens. Your token usage will vary greatly with the number of page objects and the complexity of those queries.


14-Month Data Retention Policy

Standard GA4 has a data retention policy of 14 months. That's to say only 14 months' worth of data is stored on your GA4 instance.

After that period, unless exported, the data is deleted forever...


What About Analytics 360..

Upgrading to 360 has several advantages but in regards to these 2 issues:

  • All Quotas are increased by 10x (1,750 tokens per hour to 17,500)

  • Data Retention is increased from 14 to 50 months

However.. Analytics 360 is a whopping $50,000 per year. So onto another option for most of us..


Solution - High-Level Steps

Exporting GA4 data to BigQuery

  • Set up a BigQuery project and dataset.

  • Link your GA4 property to your BigQuery project.

  • Choose which GA4 data you want to export to BigQuery.

  • Schedule regular data exports to keep your data up-to-date.

Running DBT Models

  • Create a Cloud DBT account and connect it to your BigQuery project.

  • Copy or Write DBT models in SQL to transform your GA4 data into a format that is more conducive to analysis.

  • Use DBT to automate the data transformation process.

  • Test your DBT models to ensure they are working correctly.

Visualizing with Looker Studio

  • Connect Looker Studio to your BigQuery project.

  • Create custom dashboards and visualizations to display your data.


Disclaimers

1 Million Event Export Limit

Google allows standard GA4 properties to export 1M events per day, if you are below this threshold, the export process is free.

Costs

There will be incremental storage costs and querying costs associated with Big Query. However, costs should be fairly nominal, a few hundred dollars a month, for your average-sized organization. See Google Cloud's Cost Estimator and my other [future] article about estimating Big Query costs.

Google - Subject to Change

A lot of folks are upset about the token limitation. Google is subject to change its terms and product offerings at any time. They have already increased the token limit once with the initial outcry and I believe their native solution to this, the Extract Data Source, could get better in time.

But.. not a bad idea to get your exports going regardless so you can retain a longer history of your web analytics.

Other Solutions

There are certainly other solutions to GA4's quota limits. Some involve paying another vendor like Power My Analytics, using Looker's Extract layer or transitioning to native GA4 reports.

All are valid solutions. However, unless you pay for 360, there is no other solution to retain your data. With that, adding a DBT layer isn't too much work. If you're already exporting data to Big Query you might as well do something with it.

All that said, I do believe this may not fit everyone's use case or technical aptitude. I plan on doing another article that lays out other options.


Why DBT?

The reason we need to use DBT is because the native export that comes out of GA4 is incredibly nested. The data needs to go through another step to be usable in analysis.

There can be costs associated with DBT, but you can get away with the free tier depending on your needs. The process can all be done within the free DBT tier, the advantages to an upgraded plan are more organizationally related (more user accounts, access control etc.) at least for this process specifically.


Conclusion

We didn't get too into the weeds on setting up DBT or a GA4 export. Those detailed steps deserve another dedicated write-up. However, I hope this enlightened you and at the very least, got you thinking about taking action with your GA4 data.

More to come! Please leave a comment if you found this helpful or would like to know more!