# BigQuery: Integrating and streaming from Firebase

**Channel:** Firebase
**Source:** https://www.youtube.com/watch?v=W9t3QupnqPA
**Transcript page:** https://www.withtranscript.ai/video/W9t3QupnqPA

## Chapters

- 0:03 — What BigQuery Is and Why Integrate
- 2:31 — Supported Products, Pricing, and Sandbox
- 4:43 — Linking Firebase to BigQuery Steps
- 6:33 — Streaming Analytics and Crashlytics Data
- 9:23 — Closing Thoughts and Upcoming Content

## Transcript

**[0:03]:** hi today i'm going to tell you about bigquery specifically integrating firebase with bigquery and streaming data from firebase to bigquery wow i just said bigquery a lot of times what is it bigquery is google cloud's enterprise data warehouse and it's designed to help ingest store analyze and visualize a bunch of data it's often used to quickly get insights from that data in firebase there's a lot of data across a lot of the products and sometimes you might have some questions about your app or your user's experiences that can be answered by looking more closely at this data maybe you're using google analytics and wants to see the retention rate by country or maybe you want to check crashlytics to look at the number of crashes per day for the past month to ensure a bug batch you just had made your app more stable over time all of these questions can be answered by integrating

**[1:01]:** firebase with bigquery where you can perform custom analysis on your data set per product but wait all these questions can be answered from within the analytics dashboards too so why integrate firebase with bigquery well when you integrate firebase with bigquery you can dig into the data to see if there are any patterns and trends or perhaps you have questions that aren't already answered in the firebase console you can use bigquery queries to answer those questions to get more actionable insights and get into much deeper exploration of your data in ways that are far more customized than what's available in the console for example maybe you don't just want to see user retention bank country but also want to compare different user groups on a regular basis based on factors like when they first started using the app or their total app engagement time or maybe you want to do something even more customized like cross

**[2:00]:** referencing that data with your own marketing and other backend data to match analytics data with other marketing campaigns you recently ran another benefit of exporting firebase data to bigquery is that you can decide how long you want to keep your raw data history since the console doesn't keep your data around forever there are literally endless possibilities and this is where using bigquery can become a big help currently five firebase products integrate with bigquery analytics cloud messaging

**[2:31]:** crashlytics performance monitoring and a b testing after integrating with bigquery you can export data from each of these products and use sql to query the data to gain insights there's also a firebase extension for exporting cloud firestore collections to bigquery as well and there's a link below if you want to learn more about that you may be wondering whether bitquery is an entirely no cost service and the answer to that question is that some features are free of charge and some features aren't while loading and exporting data have no cost bigquery does charge for storing data streaming inserts and querying data there's more information in the link provided in the description what if you wanted to try out bigquery to make sure it fits your needs bigquery sandbox is a way to try out bigquery without entering a credit card and it lets you explore bigquery capabilities at no cost to confirm that it does

**[3:30]:** indeed fit your needs there are some limits though including a default table expiration time of 60 days meaning the tables will get deleted in 60 days you also won't have access to some features such as streaming to bigquery which we'll talk about in a bit data transfer service and data manipulation language statements if you do want those features you can upgrade at any point from the sandbox to use bigquery without these limits and when you upgrade you'll still have access to the no cost tiers included in the sandbox but operating beyond those limits can incur charges and those charges depend on how much data you store in bigquery and the queries you make although loading and exporting data is free to check out more details on bigquery pricing there's a link in the description as a reminder with bigquery sandbox there's a 60 day expiry set on tables used for exporting and while upgrading bigquery

**[4:30]:** means you no longer have the 60-day limit you'll have to manually set the expiration as upgrading doesn't automatically reset it take a look at the link in the description for how to do so to export data to bigquery

**[4:43]:** from firebase first go to the integrations page in the firebase console in the bigquery card click link and then you'll see a page with some information about linking firebase to bigquery as a side note you'll have to have the required permissions to view or manage settings for data export to bigquery and you'll find a link with more information about this in the description in firebase console you can click next to choose which firebase products you'd like to integrate with bigquery and you can click on link to bigquery from there if at any time you'd like to integrate more firebase products or unlink from bigquery overall you can come back to this page once you link a firebase product you can choose which apps in your project will send data to bigquery by clicking on the edit button here next to apps exporting if you create a new app for that project the new apps data

**[5:38]:** will automatically be exported but you can come back to this page at any time to manage which apps and data the details of bitcoin integration with each of the five firebase products are slightly different and i'm going to cover the ones that have specific bigquery features enabled for them like google analytics and crashlytics the other products are fairly similar in how they integrate with bigquery but still each have their own nuances these are more minor differences though so i won't cover them here but in case you're curious you can check out the docs linked in the description for more details on each of them in general linking firebase products to bigquery enables you to gain insight into data for the previous day that is today's data will not be available today and you'll typically have to wait until tomorrow to be able to query today's data but what if you wanted to check data for

**[6:33]:** today to get a sense of what's happening right now for that google analytics and crashlytics support streaming which makes data for the current date available by publishing periodic updates to intraday tables throughout the day for google analytics streaming bitquery has more recent information you can analyze about your users and their traffic on your property for each day streaming export creates one new table and it's named events intraday and then the date which is an internal staging table that includes records of session activity that took place during the day streaming export is a best effort operation and may omit all data for reasons such as the processing of late events or field uploads data is exported continuously throughout the day and this table can include records of a session when that session spans multiple export operations the table is eventually deleted when the daily export is complete it's important to note that no backfill for ga

**[7:36]:** is available meaning that until you link ga2 bigquery there will be no ga data available in bigquery so if you plan on using bigquery or even if you just suspect that you might use it it's probably worth linking it sooner rather than later so you have data available to query also as a side note while there is no limit on streaming export standard ga properties do have a daily bigquery export limit of 1 million events if your property consistently exceeds the export limit the daily bigquery export will be paused and the previous day's exports will not be reprocessed to get rid of this limitation you can upgrade to ga360 and to help operate under that limit you can use data filtering to determine which events get exported see the link in the description to learn more about how to set this up you can also include streaming for crashlytics which will result in a real-time

**[8:36]:** table in addition to the batch table you'd have for integrating crashlytics with bigquery the real-time table will have data exported in real time but with no backfill available whereas the batch table has data exported once daily and can provide backfill data up to 90 days prior the batch table is ideal for long-term analysis and identifying trends over time because firebase durably stores events before writing them when data gets written to your real-time table the data gets immediately written to bigquery and so is ideal for live dashboards and custom alerts these two tables can be combined with the stitching query to get the benefits of both streaming google analytics and crashlytics data to bigquery as well

**[9:23]:** as integrating those in a b testing cloud messaging and performance monitoring with firebase can be quite helpful as you'll be able to gain more insights into all of the data that you have hopefully giving you a better understanding of your users as you dive deeper into bigquery you might be wondering about what sort of data is being exported to bigquery and what you're actually able to query and it might also be helpful to take a look at some examples of queries as if i read your mind there will be a video coming soon about this thanks for watching and happy querying

**[10:10]:** you
