Firestore for Rust

Library provides a simple API for Google Firestore based on the official gRPC API:

• Fluent high-level and strongly typed API, the only public API of this library;
• Create or update documents using Rust structures and Serde;
• Support for:
  • Querying/streaming docs/objects;
  • Listing documents/objects (stream_all follows the page tokens for you);
  • Listening changes from Firestore, with targets added and removed on a running listener and an opt-in idle reset for silent streams;
  • Transactions;
  • Aggregated queries: count, sum, avg;
  • Streaming batch writes with automatic throttling to stay under the Firestore write rate limits;
  • K-nearest neighbour (KNN) vector search with Euclidean, Cosine and Dot Product measures;
  • Query cursors and partition queries;
  • Collection group queries and listing collection IDs;
  • Server side document transforms: increment, array append/remove, server timestamp;
  • Update/delete preconditions;
  • Consistency selectors to read at a given time or inside a transaction;
  • Explaining queries: Core queries on the Standard edition, pipelines and SQL on the Enterprise edition;
  • Request tags to attribute Firestore usage;
  • Opt-in request timeouts;
• On the Enterprise edition of Firestore:
  • Pipelines: filters, projections, aggregations, joins and subqueries, unions, nearest neighbours and writes;
  • Full-text and geospatial search in pipelines;
  • SQL statements in the BigQuery dialect, translated into pipelines, behind an opt-in sql feature;
  • Unique, dense and one-field indexes, and text and geo search indexes, through index management;
• Full async based on Tokio runtime;
• Macros that help you use your structure fields as Firestore field paths;
• Implements own Serde serializer to Firestore protobuf values;
• Support for multiple database IDs;
• Support for extended datatypes:
  • Firestore timestamp as a FirestoreTimestamp type or with #[serde(with)] attributes (based on jiff);
  • Lat/Lng;
  • References;
  • Explicit nulls;
• Caching support for collections and documents, for Core reads and for pipeline and SQL reads:
  • In-memory cache;
  • Persistent cache;
• Declarative index management behind an opt-in admin feature: composite indexes (including one-field, unique and sparse ones), vector indexes, text and geo search indexes, single-field overrides and TTL policy, planned and synced with one call, following the edition of the database;
• Bulk delete behind the same admin feature: removes every document in named collection groups, at any depth across the whole database, in the background;
• Works with the Firestore emulator through FIRESTORE_EMULATOR_HOST, no credentials needed;
• Google client based on gcloud-sdk library that automatically detects GCE environment or application default accounts for local development;

Quick start

Cargo.toml:

[dependencies]
firestore = "0.63"

Pipelines and SQL need a database of the Enterprise edition, see Standard and Enterprise editions.

Crypto provider error

Depends on your other dependencies you may see the error like:

no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point

This is because the TLS providers are not installed by default and you can choose different. The easiest way to fix is just to include one of the provider, for example:

[dependencies]
rustls = "0.23"

If you have multiple you may need to call CryptoProvider::install_default() before using the Firestore client, e.g.:

rustls::crypto::ring::default_provider().install_default().expect("Failed to install rustls crypto provider");

Running the examples

All examples available in the examples directory.

To run an example with environment variables:

PROJECT_ID=<your-google-project-id> cargo run --example crud

Client instance and lifecycle

To create a new instance of Firestore client you need to provide at least a GCP project ID. It is not recommended creating a new client for each request, so it is recommended to create a client once and reuse it whenever possible. Cloning instances is much cheaper than creating a new one.

The client is created using the FirestoreDb::new method:

fn config_env_var(name: &str) -> Result<String, String> {
    std::env::var(name).map_err(|e| format!("{name}: {e}"))
}
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use firestore::*;

// Create an instance
let db = FirestoreDb::new(&config_env_var("PROJECT_ID")?).await?;
Ok(())
}

This is the recommended way to create a new instance of the client, since it automatically detects the environment and uses credentials, service accounts, Workload Identity on GCP, etc. Look at the section below Google authentication for more details.

In cases if you need to create a new instance explicitly specifying a key file, you can use:

use firestore::*;
fn config_env_var(name: &str) -> Result<String, String> {
    std::env::var(name).map_err(|e| format!("{name}: {e}"))
}
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
FirestoreDb::with_options_service_account_key_file(
    FirestoreDbOptions::new(config_env_var("PROJECT_ID")?.to_string()),
    "/tmp/key.json".into(),
)
.await?
;
Ok(())
}

The key file is read as a service account key.

For any other credentials use FirestoreDb::with_options_auth, which takes everything that converts into gcloud-sdk’s GoogleAuthHeaders:

• Credentials from the builders of firestore::gcloud_sdk::google_cloud_auth::credentials: service account keys, user credentials, impersonation, workload identity federation, the metadata server, etc.;
• a CredentialsProvider of your own, passed to Credentials::from;
• GoogleAuthHeaders::from_adc_with_scopes for the Application Default Credentials with other scopes;
• GoogleAuthHeaders::from_service_account_key for a service account key with other scopes.

use firestore::*;
fn config_env_var(name: &str) -> Result<String, String> {
    std::env::var(name).map_err(|e| format!("{name}: {e}"))
}
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
FirestoreDb::with_options_auth(
    FirestoreDbOptions::new(config_env_var("PROJECT_ID")?.to_string()),
    firestore::gcloud_sdk::GoogleAuthHeaders::from_adc_with_scopes(vec![
        "https://www.googleapis.com/auth/datastore".to_string(),
    ])
    .await?,
)
.await?
;
Ok(())
}

These types come from gcloud-sdk, which the library re-exports as firestore::gcloud_sdk, so your project needs no gcloud-sdk dependency of its own.

Full example with a custom CredentialsProvider available here.

Multiple databases

Firestore supports multiple databases per project, so you can specify the database ID in the options:

use firestore::*;
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
FirestoreDb::with_options(
    FirestoreDbOptions::new("your-project-id".to_string())
        .with_database_id("your-database-id".to_string()),
)
.await?
;
Ok(())
}

The edition, Standard or Enterprise, belongs to a database, so the database ID chooses it too. A project can hold databases of both editions, and a client per database ID reaches each. See Standard and Enterprise editions for what each edition runs.

Standard and Enterprise editions

Firestore databases come in two editions, Standard and Enterprise. The library supports both, and some features need Enterprise. Every page of this book applies to both editions, unless its title, or the part of the book it sits in, names one edition, or a section “On the Enterprise edition” lists what differs.

The edition belongs to a database, not to the project: one project can hold databases of both editions, and the client picks one by its ID, see Client instance and lifecycle.

What each edition runs

The API column names the API a row is about: Core queries, pipelines, SQL, Admin for index management, TTL and bulk delete, or Any for a row that belongs to no one API. “Pipeline, SQL” marks a row where both behave the same. A number points to the notes under the table.

Telling which edition a database is

gcloud firestore databases describe prints the settings of a database, among them its edition, its real-time updates mode and its concurrency mode:

gcloud firestore databases describe --database=my-database

These lines of its output name the three settings:

concurrencyMode: OPTIMISTIC
databaseEdition: ENTERPRISE
realtimeUpdatesMode: REALTIME_UPDATES_MODE_ENABLED

The Firestore page of the Google Cloud console lists the databases of a project with the edition of each.

With the admin feature, FirestoreDb::read_database_edition() returns the edition of the database the client is connected to. A database that reports no edition reads as Standard, the edition Firestore applies at creation. An edition this version of the crate does not know reads as Unknown, with the value Firestore reported. Firestore may add editions, so FirestoreDatabaseEdition is non-exhaustive. The emulator does not serve the Admin API, so the call fails there with UNIMPLEMENTED.

use firestore::*;
async fn example(db: FirestoreDb) -> FirestoreResult<()> {
match db.read_database_edition().await? {
    FirestoreDatabaseEdition::Standard => println!("Standard edition"),
    FirestoreDatabaseEdition::Enterprise => println!("Enterprise edition"),
    other => println!("an edition this version does not know: {other:?}"),
}
Ok(())
}

Two errors also tell the editions apart: a pipeline sent to a Standard edition database fails with FAILED_PRECONDITION, and a Core .explain() sent to an Enterprise edition database fails with INVALID_ARGUMENT, “Explain options are not supported in RunQuery API for Enterprise edition”.

Index management reads the edition itself, the same way, on every .plan() and .sync(), and follows the rules of that edition.

Settings fixed at creation

The edition and the real-time updates setting are immutable: to change either, create a new database with the wanted settings and move the data. Both are chosen when the database is created:

gcloud firestore databases create --database=my-database --edition=enterprise \
  --location=nam5 --enable-firestore-data-access --enable-realtime-updates

Google’s documentation does not agree on the default of real-time updates for an Enterprise edition database, so set --enable-realtime-updates explicitly when the database needs listeners or a cache.

Limits

• a pipeline or SQL statement sent to a Standard edition database fails at once with FAILED_PRECONDITION, and the library does not retry it;
• a search without a search index that covers it fails with FAILED_PRECONDITION, see Pipeline search;
• joins and subqueries are in Preview, see Pipeline aggregations and joins;
• Firestore rejects an atomic pipeline or SQL write that is too big, see Write limits;
• Firestore has no setting that rejects a table scan, see Query plans and cost;
• a Core .explain() is rejected on Enterprise. The same query written as a pipeline or SQL returns its plan through .explain(), see Explaining a pipeline.

Query and write semantics

A pipeline and a SQL statement keep Firestore’s pipeline semantics, which differ from a Core query in a few places: not_equal matches a document that lacks the field, sort keeps such documents, and the stages run in the order they are written. Query and write semantics lists every difference, with the SQL spelling of each.

Indexes

An Enterprise edition database builds no index on its own, takes indexes of one field, unique and dense indexes, and search indexes. A Standard edition database indexes every field automatically, and takes single-field overrides. See Composite and vector indexes, Search indexes and TTL and the Enterprise edition for the rules, and Waiting for changes to finish for the build times to expect on each edition.

Cost

Standard bills each document read, written and deleted; Enterprise bills read units (4 KiB) and write units (1 KiB, one more per indexed field) instead, and real-time updates as a separate charge. A read unit is 4 KiB of data, documents or index entries, that a read processes, so an Enterprise read costs by the data it scans. The rates are on Google’s pricing pages, for Standard and Enterprise. Query plans and cost shows how to read what a query costs.

Which API to use

Choosing Core queries, pipelines or SQL lists when each API fits, on each edition.

1. A pipeline or SQL statement sent to a Standard edition database fails with FAILED_PRECONDITION, unless a cache session answers it. ↩

2. FAILED_PRECONDITION, “The query requires an index”. ↩

3. The query reads every document of the collection. Without order_by the order of its results is unspecified. ↩

4. A pipeline or SQL statement that no index covers runs as a table scan. A search is the exception: it needs a search index and fails without one, see Known table scans. ↩

5. Optimistic by default, so under contention more attempts abort and are retried. ↩

6. An Enterprise edition database rejects a Core .explain() with INVALID_ARGUMENT, so explain the same query as a pipeline or SQL, see Explaining a pipeline. ↩ ↩2

7. Without a vector index the query fails with FAILED_PRECONDITION. ↩

8. A vector index can still be created on Enterprise, but whether a Core query uses it cannot be seen, since Core .explain() is rejected there. ↩

9. The find_nearest stage reads every document with or without a vector index, and a filter before it does not narrow the read. Google lists this as a known issue of pipelines. Use a Core find_nearest where the index should be used, see Vector search. ↩

10. An .exempt() is already satisfied. ↩

11. The density of an index is which documents it holds an entry for. ↩

12. A search index is SparseAny. ↩

13. Real-time updates are billed separately. ↩

Query and write semantics

A pipeline runs with Firestore’s pipeline semantics, and a SQL statement runs as a pipeline, so it keeps them too. A Core query has semantics of its own. This page lists where the two differ, and where the pipeline semantics differ from what a SQL database does.

Null and absent

A document either has a field, holding a value that can be null, or lacks it. SQL has no absent field: the column holds NULL. A Core filter or order_by on a field never returns a document that lacks the field.

The pipeline column names Firestore’s pipeline functions, with the name of the filter builder method in brackets where it differs:

A row that lacks a key deserialises into an Option field as None.

COUNT(x) counts documents where x has a value that is not null. Over 11 documents, one holding null and one lacking the field, COUNT(*) is 11 and COUNT(v) is 9; COUNT(DISTINCT v) counts each distinct value once.

The SQL results are written in SQL, using the pipeline function is_absent, which passes through:

• v IS NULL OR is_absent(v) matches null and absent;
• v != 1 AND v IS NOT NULL leaves them out;
• if_absent(v, NULL) selects an absent field as null.

The filters differ in cost. rating != 4.2 AND rating IS NOT NULL keeps the index ranges of rating != 4.2, and genre IS NULL OR is_absent(genre) loses the index range of genre IS NULL, so it scans every index entry of the field. See Query plans and cost.

Whether a field is stored as null or left absent is decided when it is written, see Explicit null value serialisation.

Types

A condition earlier in the same AND that excludes the other types avoids the failure: with k IN ('pct', 'pcx') AND v LIKE '100%', the LIKE runs only on strings.

A Core range filter matches values of its own type only, and never NaN. A Core order_by over mixed types follows Firestore’s order of types: null, booleans, numbers, timestamps, strings, bytes, references, geo points, arrays, vectors and maps.

Numbers

• v = 1 and v = 1.0 match the same documents, and DISTINCT and GROUP BY collapse an integer and a double of the same value. They collapse null and absent into one group too;
• two integers divide as integers: 7 / 2 is 3, -7 / 2 is -3 and 7 % 3 is 1, where 7.0 / 2 is 3.5. A SQL dialect whose / returns a float differs from this;
• a Core query compares an integer and a double by value too, so a filter on 1 matches the double 1.0, and order_by sorts them together.

Names and patterns

• field names are case-sensitive. SELECT Title for a field title returns rows without a title, and WHERE Genre = 'Dystopian' matches nothing. A misspelled field is an absent field, with no error;
• LIKE is case-sensitive, % and _ are wildcards, and . and ( are literal. A backslash escapes a wildcard, written '100\\%' in the statement since a backslash escapes in a string literal too;
• __name__ compared with a string is read as a document name, and the translator makes it a reference. __name__ in INSERT takes an ID or a path, see Writes and ON CONFLICT.

Stage order

The stages of a pipeline run in the order they are written. A limit before a sort limits the documents first and sorts what is left, which is not what the same two calls mean in a Core query, where limit always applies to the ordered results.

Order without a sort

• a pipeline without a sort returns its documents in an order of its own;
• on a Standard edition database, a Core query without order_by returns its documents in the order of the fields it compares with <, <=, >, >=, != or not-in, and then by document name;
• on an Enterprise edition database the order of a Core query without order_by is unspecified.

Results without a document name

A stored document carries its name, so _firestore_id is filled for it in Core and pipeline results. A Core aggregation result, and the result of a pipeline stage that builds new documents, such as .fields() or .aggregate(), has no name. A SQL row holds only the columns the statement selects, so it has no name unless the statement selects __name__. A field of the result type that reads _firestore_id must be an Option, see Typed results and metadata and Document metadata as struct fields.

Cached results

A read that a cache session answers follows the same semantics, with these differences:

• a cached Core query follows the order of the Standard edition, and on an Enterprise edition database documents with equal values can come back in another order, see What the cache answers;
• a cached pipeline returns documents that tie on every sort key in document-name order, where Firestore’s order of them is unspecified, see Caching pipelines;
• both are eventually consistent, see How the cache is updated.

Writes

• a Core write in a transaction is buffered in the library and sent with the commit. A pipeline or SQL write is sent when it runs, and its documents change at the commit. A transaction takes pipeline writes or buffered writes: Firestore rejects a commit that mixes them with INVALID_ARGUMENT, see Writes in a transaction;
• a pipeline or SQL write is sent once and never retried. Simple batch writes and transactions retry, see Request timeouts;
• a pipeline upsert and SQL INSERT OR REPLACE replace the stored document, so its fields that the write does not set are gone;
• a write that a unique index refuses returns DataConflictError, see TTL and the Enterprise edition;
• Core transactions behave the same on both editions; under contention an Enterprise database, which is optimistic by default, aborts and retries transactions more often than a Standard one, see Transactions.

Query plans and cost

What a read costs depends on the edition of the database and on whether an index serves it. This page holds the billing model of each edition, how to read the plan Firestore chose for a query, and the reads that are known to scan.

Two terms are used throughout:

• a table scan reads every document of the collection from primary storage, instead of reading an index;
• a read unit is the billing unit of reads on the Enterprise edition: 4 KiB of data, documents or index entries, that a read processes.

Cost per edition

Standard bills each document read, written and deleted; Enterprise bills read units (4 KiB) and write units (1 KiB, one more per indexed field) instead, and real-time updates as a separate charge. The rates differ per region, and are on Google’s pricing pages for Standard and Enterprise.

Standard edition

Google’s pricing page bills:

• each document read, written and deleted, the index entries a query reads, storage and network bandwidth;
• at least one document read for each query, even one that returns nothing;
• one read for each batch of up to 1000 index entries a query reads; a query with at most one range field is not charged for index entries;
• an aggregation (count, sum, avg) by the index entries it reads, and at least one document read;
• a listener one read each time a document is added to its results or updated in them, and when a document leaves them. A deleted document is not charged as a read.

Enterprise edition

Google’s pricing page bills:

• read units for the data a read processes, documents or index entries, counted in 4 KiB. Its example: a collection scan over 20 documents of 1 KiB each costs 5 read units, and an indexed scan of the same data costs 21, one for each document and one for the index read;
• write units counted in 1 KiB, and one more for each indexed field, so a 5 KiB document with 5 single-field indexes costs 10 write units. A delete is billed as write units;
• real-time updates as a charge of their own: one unit per 4 KiB of an updated document, plus one when a document leaves the results.

A Core query that no index covers runs as a table scan on Enterprise, and is billed by the data it reads: a scan of the whole collection reads every 4 KiB of it. On Standard the same query is either refused or answered from an index, and billed per returned document.

A SQL statement that reads a document by ID, with = or IN, costs 1 read unit for each document on the Enterprise edition, with or without a further condition such as stock >= 0. Firestore’s published pricing charges a point read of the Core get in 4 KiB tranches, rounded up, so a document of up to 4 KiB costs 1 read unit through get, as it does through SQL. See Reads by ID.

Both editions

A read that a cache session answers sends no request, so Firestore charges nothing for it. Query Explain is never answered from the cache.

Explaining a Core query (Standard edition)

Query Explain returns the plan Firestore chose for a Core query, in the metadata of the results:

use firestore::*;
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .explain()
    // or explain_with_options for options such as analyze, which runs the query to gather
    // additional statistics:
    // .explain_with_options(FirestoreExplainOptions::new().with_analyze(true))
    .stream_query_with_metadata()
    .await?;
Ok(())
}

Full example available here.

On a Standard edition database, a Core .explain() of a query that needs a missing index fails with FAILED_PRECONDITION and the message “no matching index found.”

An Enterprise edition database rejects a Core .explain() with INVALID_ARGUMENT, “Explain options are not supported in RunQuery API for Enterprise edition. Please use the ExecutePipeline API instead.” There, write the query as a pipeline to see its plan.

Explaining a pipeline

Query Explain returns the plan Firestore chose for a pipeline. .explain() runs the pipeline for its plan only: no document is read or returned. .explain_with_options(..) with analyze set also returns the documents, and adds the execution profile to the plan. The plan arrives as text in the metadata of the last item:

use firestore::*;
use futures::TryStreamExt;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let items: Vec<_> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .filter(|q| q.field("genre").eq("Dystopian"))
    .explain()
    .stream_query_with_metadata()
    .await?
    .try_collect()
    .await?;

for item in items {
    if let Some(stats) = item.metadata.explain_stats {
        println!("{}", stats.text);
    }
}
Ok(())
}

The text is Firestore’s own format. The library does not parse it, since Firestore can change it at any time.

A SQL SELECT returns its plan the same way, through explain() and explain_with_options(..) of the SQL builder, or with the EXPLAIN and EXPLAIN ANALYZE prefix in the text, which is the SQL spelling of the same two calls:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let items = db
    .fluent()
    .sql("EXPLAIN SELECT title FROM books WHERE genre = 'Dystopian'")
    .stream_query_with_metadata()
    .await?;
let _ = items;
Ok(())
}

EXPLAIN takes a SELECT and no other option than ANALYZE, and fails together with .explain() on the builder, since the text and the builder would both set the options. The Enterprise emulator answers an EXPLAIN with the rows and no plan, see Firestore emulator.

Recognising a full scan

The plan names how the pipeline reads its documents:

• a plan that ends in a TableScan node reads every document of the collection from primary storage;
• a plan that ends in a SequentialScan node with an index: line reads through that index, such as index: /books (genre ASC, rating DESC)@[id = CICAgJim14AK];
• a search ends in a StIndexSearch node.

Run .explain() on a pipeline before it reaches production data, and add an index when it shows a TableScan over a large collection. Firestore has no setting that rejects a table scan, and the library adds no check of its own.

Choosing an index

.force_index(..) on a from(..) source makes the pipeline read through the index with a given ID, the one Firestore assigned to it. It is available on a collection source and on a collection-group source, and nowhere else:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let by_genre = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .force_index(FirestoreIndexHint::Index("CICAgJim14AK".parse()?))
    .filter(|q| q.field("genre").eq("Dystopian"))
    .query()
    .await?;

let scanned = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .force_index(FirestoreIndexHint::TableScan)
    .filter(|q| q.field("genre").eq("Dystopian"))
    .query()
    .await?;
let _ = (by_genre, scanned);
Ok(())
}

FirestoreIndexHint::TableScan forces a table scan. An ID that names no index of that collection fails the request at once with INVALID_ARGUMENT. An index that cannot serve the filter is still used, and the pipeline then reads all of it. A vector index cannot be forced.

Known table scans

• A pipeline without an index. Indexes are optional for a pipeline. A pipeline without an index that covers its filter still runs, as a table scan. The cost of a pipeline follows the amount of data it scans, so a table scan over a large collection costs in proportion to the collection. A search is the exception: it needs a search index, and fails without one, see Pipeline search;
• A SQL statement without an index. A statement that filters on an unindexed field scans the collection, as a pipeline does. Firestore has no setting that rejects a table scan, and the read-only options do not check for one;
• A Core query on Enterprise. On Enterprise a Core query runs as a scan when no index covers it, where Standard fails with FAILED_PRECONDITION, and without order_by the order of its results is unspecified. Its plan cannot be shown, since Core .explain() is rejected there;
• Pipeline find_nearest. The stage does not use a vector index. Its plan is a TableScan followed by a sort by distance, even when a vector index exists on the field, so its cost follows the size of the collection. Google lists this as a known issue of pipelines. A filter before the stage does not narrow what it reads, even with a composite index. A Core vector search is the query that can use a vector index, and it does on Standard;
• Joins and subqueries. A subquery runs once for each row of the query it is in, whether or not it is correlated, that is, whether or not it reads a value of the row it runs for. An unindexed join scans the joined collection once for each left row, see Join cost;
• Null or absent in one filter. genre IS NULL OR is_absent(genre) loses the index range of genre IS NULL, see Query and write semantics.

Choosing Core queries, pipelines or SQL

The library reads Firestore through three APIs: Core queries (db.fluent().select()), pipelines (db.fluent().pipeline()) and SQL (db.fluent().sql(..)), which runs as a pipeline. Which one fits depends on the edition of the database first, and on what the query needs.

On a Standard edition database

The Core API is the only one. Pipelines and SQL fail with FAILED_PRECONDITION, and the library does not retry them. In a cache session, a pipeline that the cache answers on its own succeeds on Standard too, and one it does not answer fails as on Firestore. Be aware not to rely on that: a pipeline that works from the cache fails once it reaches Firestore.

On an Enterprise edition database

The Core query API stays the choice for what it covers. Reasons to stay on Core there:

• listeners take Core targets only, see Listening to document changes;
• the cache answers every supported Core query, and only the pipelines and SQL statements that filter, sort, page and project plain fields, see Caching pipelines. Anything else runs on Firestore, or fails under read_cached_only;
• existing code keeps the Core semantics of null and absent fields, see Query and write semantics;
• Core writes in a transaction are buffered and sent with the commit, see Transactions.

A pipeline adds:

• grouped aggregations, computed fields, distinct, unnest and union;
• joins through subqueries;
• text and geo search;
• writes by query: update, delete, insert and upsert of the documents a pipeline selects, in one request and without reading them first;
• Query Explain on Enterprise, where the Core .explain() is rejected with INVALID_ARGUMENT.

When to use SQL

Application code keeps the fluent API. A field name in path!(Book::rating) is checked when the program compiles, where a misspelled field in SQL text reads as an absent field and matches nothing, without an error.

SQL is for text that the program does not write:

• statements typed by a person or a script, such as an admin command line;
• statements kept in configuration or in files;
• statements an LLM agent writes, which run with the read-only options;
• queries ported from a SQL database, where JOIN and GROUP BY read faster than the builder.

Checking the plan before production

• a pipeline without an index that covers its filter runs as a table scan, and costs in proportion to the collection;
• on Enterprise a Core query runs as a scan when no index covers it, where Standard fails with FAILED_PRECONDITION, and without order_by the order of its results is unspecified;
• Core .explain() is rejected on Enterprise, so the plan of a query there comes from the same query written as a pipeline.

See Query plans and cost.

Vector search

A Core vector search needs a vector index on Standard, and runs without one on Enterprise. The pipeline find_nearest stage never uses the index, so its cost follows the size of the collection.

Moving a Core query to a pipeline

The same filter can return other documents. Check these:

• neq matches the documents that lack the field;
• sort keeps the documents that lack the field, and needs a unique field last for a stable order;
• the stages run in the order they are written, so limit before sort limits first;
• _firestore_id is an Option after a projection, since the result has no document name.

The query below runs both ways. On a book without a genre field, the Core query leaves the book out, and the pipeline returns it:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Book { title: String, genre: Option<String> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Leaves out the books without a genre
let core: Vec<Book> = db
    .fluent()
    .select()
    .from("books")
    .filter(|q| q.field(path!(Book::genre)).neq("Horror"))
    .obj()
    .query()
    .await?;

// Returns the books without a genre too
let pipeline: Vec<Book> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .filter(|q| q.field(path!(Book::genre)).neq("Horror"))
    .obj()
    .query()
    .await?;
let _ = (core, pipeline);
Ok(())
}

To keep the Core result in the pipeline, add q.field(path!(Book::genre)).exists() to a for_all with the neq. Query and write semantics lists every difference.

Fluent API

The Fluent API is the only public API of this library. Everything starts from db.fluent():

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
    one_more_string: String,
    some_num: u64,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use firestore::*;

const TEST_COLLECTION_NAME: FirestoreCollectionId = FirestoreCollectionId::from_static("test");

let my_struct = MyTestStructure {
    some_id: "test-1".to_string(),
    some_string: "Test".to_string(),
    one_more_string: "Test2".to_string(),
    some_num: 42,
};

// Create
let object_returned: MyTestStructure = db
    .fluent()
    .insert()
    .into(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&my_struct)
    .execute()
    .await?;

// Update or Create
// (Firestore supports creating documents with update if you provide the document ID).
let object_updated: MyTestStructure = db
    .fluent()
    .update()
    .fields(paths!(MyTestStructure::{some_num, one_more_string}))
    .in_col(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&MyTestStructure {
        some_num: my_struct.some_num + 1,
        one_more_string: "updated-value".to_string(),
        ..my_struct.clone()
    })
    .execute()
    .await?;

// Get object by id
let find_it_again: Option<MyTestStructure> = db
    .fluent()
    .select()
    .by_id_in(TEST_COLLECTION_NAME)
    .obj()
    .one(&my_struct.some_id)
    .await?;

// Delete data
db.fluent()
    .delete()
    .from(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .execute()
    .await?;
Ok(())
}

Create, update and delete documents covers the write calls above in detail.

The low level “support” traits are crate private. The migration guide lists the fluent replacement of every removed method.

On a database of the Enterprise edition, db.fluent() has two more entry points next to the Core ones:

• db.fluent().pipeline() for pipeline reads and writes;
• db.fluent().sql(..) for SQL statements, behind the sql feature.

Choosing Core queries, pipelines or SQL compares the three.

Alongside it, the library provides core functionality that is public API in its own right:

• Batch writes: db.create_simple_batch_writer(), db.create_streaming_batch_writer()
• Transactions: db.begin_transaction(), db.run_transaction(), and the FirestoreTransactionOps trait implemented by both FirestoreTransaction and FirestoreTransactionData
• Listeners: db.create_listener(), and the FirestoreResumeStateStorage trait for custom resume token storage
• Caching: the FirestoreCacheBackend / FirestoreCacheDocsByPathSupport traits for custom cache backends
• Dynamic documents: FirestoreDb::serialize_map_to_doc(), FirestoreDb::serialize_to_doc() and FirestoreDb::deserialize_doc_to(), used together with the fluent .document(...) builders

Get and batch get

.one(id) reads one document by its ID, and returns None when it does not exist. .batch(ids) reads several documents in one request, and streams each ID with its document or None:

use firestore::*;
use futures::stream::BoxStream;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb, my_struct: MyTestStructure) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let find_it_again: Option<MyTestStructure> = db
    .fluent()
    .select()
    .by_id_in(TEST_COLLECTION_NAME)
    .obj()
    .one(&my_struct.some_id)
    .await?;

let object_stream: BoxStream<(String, Option<MyTestStructure>)> = db
    .fluent()
    .select()
    .by_id_in(TEST_COLLECTION_NAME)
    .obj()
    .batch(vec!["test-0", "test-5"])
    .await?;
Ok(())
}

What a read by ID costs on each edition is in Query plans and cost.

Querying

The library supports rich querying API with filters, ordering, pagination, etc.

use firestore::*;
use futures::stream::BoxStream;
use futures::TryStreamExt;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
    one_more_string: String,
    some_num: u64,
    created_at: FirestoreTimestamp,
}
const TEST_COLLECTION_NAME: &str = "test-query";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Query as a stream our data
let object_stream: BoxStream<FirestoreResult<MyTestStructure>> = db
    .fluent()
    .select()
    .fields(
        paths!(MyTestStructure::{some_id, some_num, some_string, one_more_string, created_at}),
    ) // Optionally select the fields needed
    .from(TEST_COLLECTION_NAME)
    .filter(|q| {
        // Fluent filter API example
        q.for_all([
            q.field(path!(MyTestStructure::some_num)).is_not_null(),
            q.field(path!(MyTestStructure::some_string)).eq("Test"),
            // Sometimes you have optional filters
            Some("Test2")
                .and_then(|value| q.field(path!(MyTestStructure::one_more_string)).eq(value)),
        ])
    })
    .order(|o| o.fields([o.field(path!(MyTestStructure::some_num)).desc()]))
    .obj() // Reading documents as structures using Serde gRPC deserializer
    .stream_query_with_errors()
    .await?;

let as_vec: Vec<MyTestStructure> = object_stream.try_collect().await?;
println!("{:?}", as_vec);
Ok(())
}

Filtering

.filter() takes a closure that receives a filter builder and returns an Option<FirestoreQueryFilter>. Conditions can be optional, since a None entry in the list is dropped, as one_more_string in the example above.

Use:

• q.for_all for AND conditions;
• q.for_any for OR conditions.

You can nest q.for_all/q.for_any.

Ordering

.order() takes a closure that receives an order builder and returns a Vec, the same shape as .filter() and .transforms(). List multiple fields to sort by more than one, in priority order:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_num: u64,
}
const TEST_COLLECTION_NAME: &str = "test-query";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let ordered: Vec<MyTestStructure> = db
    .fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .order(|o| {
        o.fields([
            o.field(path!(MyTestStructure::some_num)).desc(),
            o.field(path!(MyTestStructure::some_id)).asc(),
        ])
    })
    .obj()
    .query()
    .await?;
let _ = ordered;
Ok(())
}

An entry can be conditional, since field(..).asc()/.desc() return Option<FirestoreQueryOrder> and None is dropped:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_num: u64,
}
const TEST_COLLECTION_NAME: &str = "test-query";
async fn example(db: FirestoreDb, sort_by_num: bool) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let ordered: Vec<MyTestStructure> = db
    .fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .order(|o| {
        o.fields([sort_by_num.then(|| o.field(path!(MyTestStructure::some_num)).desc())])
    })
    .obj()
    .query()
    .await?;
let _ = ordered;
Ok(())
}

Each call to .order() replaces any ordering set by a previous call; it does not add to it. An empty result (every entry conditional and every condition false) clears the ordering entirely.

If the query also uses start_at/end_at, the cursor’s value count must match the number of ordered fields - this is not enforced by the library, so a mismatch is a runtime error from Firestore rather than a compile-time one.

.order_by([(path!(..), FirestoreQueryDirection::Descending)]) still works but is deprecated in favour of .order().

On the Enterprise edition

On Enterprise a Core query runs as a scan when no index covers it, where Standard fails with FAILED_PRECONDITION, and without order_by the order of its results is unspecified. An Enterprise edition database builds no single-field index on its own, so a query reads through an index only when one is declared, see TTL and the Enterprise edition. order_by still leaves out the documents that lack the field, on both editions.

A Core .explain() is rejected there with INVALID_ARGUMENT; the plan of a query comes from the same query written as a pipeline, see Query plans and cost. A scan is billed by the data it reads, see Cost per edition.

Nested collections

You can work with nested collections specifying path/location to a parent for documents. parent_path and the builder’s at also validate the collection name they are given, so a FirestoreCollectionId or a bad raw &str is caught in the same call rather than at the server.

use firestore::*;
use futures::stream::BoxStream;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyParentStructure { some_id: String }
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyChildStructure { some_id: String }
const TEST_PARENT_COLLECTION_NAME: &str = "nested-test";
const TEST_CHILD_COLLECTION_NAME: &str = "test-childs";
async fn example(
    db: FirestoreDb,
    parent_struct: MyParentStructure,
    child_struct: MyChildStructure,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Creating a parent doc
db.fluent()
    .insert()
    .into(TEST_PARENT_COLLECTION_NAME)
    .document_id(&parent_struct.some_id)
    .object(&parent_struct)
    .execute::<()>()
    .await?;

// The doc path where the children are stored
let parent_path = db.parent_path(TEST_PARENT_COLLECTION_NAME, parent_struct.some_id)?;

// Create a child doc
db.fluent()
    .insert()
    .into(TEST_CHILD_COLLECTION_NAME)
    .document_id(&child_struct.some_id)
    .parent(&parent_path)
    .object(&child_struct)
    .execute::<()>()
    .await?;

// Listing children
println!("Listing all children");

let objs_stream: BoxStream<MyChildStructure> = db
    .fluent()
    .list()
    .from(TEST_CHILD_COLLECTION_NAME)
    .parent(&parent_path)
    .obj()
    .stream_all()
    .await?;
let _ = objs_stream;
Ok(())
}

Complete example available here.

You can nest multiple levels of collections using at():

use firestore::*;
const TEST_PARENT_COLLECTION_NAME: &str = "nested-test";
const TEST_CHILD_COLLECTION_NAME: &str = "test-childs";
const TEST_GRANDCHILD_COLLECTION_NAME: &str = "test-grandchilds";
fn example(db: &FirestoreDb) -> Result<(), Box<dyn std::error::Error>> {
let parent_path = db
    .parent_path(TEST_PARENT_COLLECTION_NAME, "parent-id")?
    .at(TEST_CHILD_COLLECTION_NAME, "child-id")?
    .at(TEST_GRANDCHILD_COLLECTION_NAME, "grand-child-id")?;
let _ = parent_path;
Ok(())
}

Aggregations

The library supports the Core aggregation functions count, sum and avg over the documents a query matches. Each result field is named by a.field(alias), and its operator is chosen with .count(), .count_up_to(limit), .sum(field) or .avg(field):

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyAggTestStructure { counter: usize }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let objs: Vec<MyAggTestStructure> = db
    .fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .aggregate(|a| a.fields([a.field(path!(MyAggTestStructure::counter)).count()]))
    .obj()
    .query()
    .await?;
let _ = objs;
Ok(())
}

Several aggregations run in one query, each under its own alias:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct Totals { count: usize, total: i64, average: f64 }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let totals: Vec<Totals> = db
    .fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .aggregate(|a| {
        a.fields([
            a.field(path!(Totals::count)).count(),
            a.field(path!(Totals::total)).sum("n"),
            a.field(path!(Totals::average)).avg("n"),
        ])
    })
    .obj()
    .query()
    .await?;
let _ = totals;
Ok(())
}

An aggregation result is not a stored document, so it has no document name, see Query and write semantics. Grouped aggregations, by a field, are a pipeline feature on the Enterprise edition, see Pipeline aggregations and joins.

On the Enterprise edition

Core aggregations and transactions behave the same on both editions. An aggregation whose filter needs a composite index fails on Standard with FAILED_PRECONDITION, and runs as a scan on Enterprise, see Querying.

Vector search

The library supports K-nearest neighbour (KNN) vector search in Core queries: find_nearest keeps the documents whose vector field is nearest to a query vector.

A vector is stored in a field of type FirestoreVector, which serialises to a Firestore vector value:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct Article {
    title: String,
    embedding: FirestoreVector,
}

let article = Article {
    title: "Firestore editions".to_string(),
    embedding: FirestoreVector::new(vec![0.1, 0.9, 0.3]),
};
let _ = article;

vec![..].into() builds a FirestoreVector too, from any iterator of f64.

Reading a vector from a document

A document read without a struct, as in Working on dynamic/document level, holds each vector as a map of __type__ and value. FirestoreVector::from_value reads a vector from such a field:

use firestore::*;
fn example(document: &FirestoreDocument) -> FirestoreResult<()> {
if let Some(value) = document.fields.get("embedding") {
    let embedding = FirestoreVector::from_value(value)?;
    println!("{} components", embedding.as_vec().len());
}
Ok(())
}

Any other value, such as a plain map, fails with a DeserializeError of code NOT_A_VECTOR.

The vector index

On a Standard edition database a vector query needs a vector index on the field, and uses it. The index declares the dimension every indexed vector in the field must have. Without one the query fails with FAILED_PRECONDITION, “Missing vector index configuration”, and the message carries the gcloud command that creates the index. Index management declares one with .vector(dimension), and the console or gcloud create one too.

Finding the nearest documents

find_nearest(field, vector, measure, limit) after the source and filters keeps the limit documents nearest to vector, nearest first:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct Article { title: String, embedding: FirestoreVector }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let nearest: Vec<Article> = db
    .fluent()
    .select()
    .from("articles")
    .find_nearest(
        path!(Article::embedding),
        FirestoreVector::new(vec![0.1, 0.8, 0.2]),
        FirestoreFindNearestDistanceMeasure::Cosine,
        5,
    )
    .obj()
    .query()
    .await?;
let _ = nearest;
Ok(())
}

FirestoreFindNearestDistanceMeasure has three measures:

• Euclidean, the straight-line distance between the two vectors;
• Cosine, which compares the angle between the vectors and ignores their length;
• DotProduct, which is like cosine, and takes the length into account too.

Documents at the same distance come back in no fixed order. A Cosine query fails with FAILED_PRECONDITION, “Cannot compute cosine distance against a vector with a magnitude of zero”, when the collection holds a vector of zeros.

Distance and threshold

find_nearest_with_options(..) takes a FirestoreFindNearestOptions, with two more settings:

• .with_distance_result_field(name) outputs the distance of each document under that field, so the result type can read it. It holds the straight-line distance for Euclidean, the cosine distance for Cosine (0 for the same direction), and the dot product itself for DotProduct;
• .with_distance_threshold(value) returns only the documents within that distance. For Euclidean and Cosine it is the largest distance, and a document at exactly that distance is kept. For DotProduct, where a larger value means a nearer vector, it is the smallest one.

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct Article {
    title: String,
    embedding: FirestoreVector,
    distance: Option<f64>,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let nearest: Vec<Article> = db
    .fluent()
    .select()
    .from("articles")
    .find_nearest_with_options(
        FirestoreFindNearestOptions::new(
            path!(Article::embedding),
            FirestoreVector::new(vec![0.1, 0.8, 0.2]),
            FirestoreFindNearestDistanceMeasure::Euclidean,
            10,
        )
        .with_distance_result_field(path!(Article::distance))
        .with_distance_threshold(0.5),
    )
    .obj()
    .query()
    .await?;
let _ = nearest;
Ok(())
}

Full example available here.

A vector query is never answered from a cache: under read_through_cache it goes to Firestore, and under read_cached_only it fails with CacheIncompleteCollection.

On the Enterprise edition

A Core vector query runs on an Enterprise edition database without a vector index. Query Explain is refused there, so its plan cannot show what the query reads.

The pipeline find_nearest stage takes the same vector and measures, and has no distance threshold. It does not use a vector index: its plan is a table scan, so its cost follows the size of the collection, see Query plans and cost.

Google lists this as a known issue of pipelines: they do not yet support the vector index type, and find_nearest is expected to be slower than its Core equivalent, see Known issues in the pipelines overview. Until pipelines use vector indexes:

• a filter before the find_nearest stage does not narrow what the stage reads. The documents are read first and the filter is applied to them afterwards, even when a composite index of the filter field and the vector field exists;
• a Core find_nearest is the query that can use a vector index, so use it where the index should be used. Enterprise rejects a Core .explain(), so whether it does there cannot be seen.

Listening to document changes

To help to work with asynchronous event listener the library supports high level API for listening the events from Firestore on a separate thread:

The listener implementation needs to be provided with a storage for the last received token for specified targets to be able to resume listening the changes from the last handled token and to avoid receiving all previous changes.

The library provides basic implementations for storing the tokens but you can implement your own more sophisticated storage if needed:

• FirestoreTempFilesListenStateStorage - resume tokens stored as temporary files on local FS;
• FirestoreMemListenStateStorage - in memory storage backed by HashMap (with this implementation if you restart your app, you will receive all notifications again);

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
const TEST_COLLECTION_NAME: &str = "test-listen";
const TEST_TARGET_ID_BY_QUERY: FirestoreListenerTarget = FirestoreListenerTarget::new(42_u32);
const TEST_TARGET_ID_BY_DOC_IDS: FirestoreListenerTarget = FirestoreListenerTarget::new(17_u32);
async fn example(db: FirestoreDb, doc_id1: String, doc_id2: String) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut listener = db
    .create_listener(
        FirestoreTempFilesListenStateStorage::new(), // or FirestoreMemListenStateStorage or your own implementation
    )
    .await?;

// Adding query listener
db.fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .listen()
    .add_target(TEST_TARGET_ID_BY_QUERY, &mut listener)?;

// Adding docs listener by IDs
db.fluent()
    .select()
    .by_id_in(TEST_COLLECTION_NAME)
    .batch_listen([doc_id1, doc_id2])
    .add_target(TEST_TARGET_ID_BY_DOC_IDS, &mut listener)?;

listener
    .start(|event| async move {
        match event {
            FirestoreListenEvent::DocumentChange(ref doc_change) => {
                println!("Doc changed: {:?}", doc_change);

                if let Some(doc) = &doc_change.document {
                    let obj: MyTestStructure =
                        FirestoreDb::deserialize_doc_to::<MyTestStructure>(doc)
                            .expect("Deserialized object");
                    println!("As object: {:?}", obj);
                }
            }
            _ => {
                println!("Received a listen response event to handle: {:?}", event);
            }
        }

        Ok(())
    })
    .await?;

// Wait some events like Ctrl-C, signals, etc
// <put-your-implementation-for-wait-here>

// and then shutdown
listener.shutdown().await?;
Ok(())
}

shutdown() finishes the current callback and resume-state writes before returning. Use shutdown_with_timeout(Duration::from_secs(5)) to allow five seconds for graceful shutdown, then abort and join the task. Aborting can interrupt callbacks and token writes, so changes may be replayed.

Targets can also be added and removed on a running listener with add_target and remove_target. The listener changes them on its open stream, so the other targets keep listening as they were. add_target returns an error for a target that cannot be turned into a listen request, so a bad target never reaches the running listener.

Target changes with the Remove type reach the callback in two forms:

• with no cause, Firestore acknowledges a remove_target call, or ends a target added with add_target_once. Events for a removed target can still arrive until this acknowledgement;
• with a cause, Firestore removed the target on its own. For PERMISSION_DENIED, NOT_FOUND, FAILED_PRECONDITION, UNIMPLEMENTED, and INVALID_ARGUMENT on a target added without a resume token or read time, the listener stops listening on that target only and unregisters it. For any other cause it adds the target back from scratch after the retry delay. An ended target is unregistered before the callback receives the change, so has_target called from the callback already tells whether the target comes back.

When Firestore removes a target on its own while a remove_target call for it is on its way, Firestore does not acknowledge the removal. It closes the whole stream with NOT_FOUND instead. The listener then reconnects, and the other targets are added back at once.

The listener reconnects by itself after network errors, closed streams and every other error, after the retry delay. It stops only on an error a retry cannot fix:

• Firestore rejects the listen request as invalid again after every resume token was discarded;
• the listen request fails with InvalidParametersError;
• the AddTarget or RemoveTarget request for a target cannot be built;
• the callback or the token storage panics.

A stopped listener rejects add_target with a SystemError, and shutdown() returns the error it stopped with.

A listen stream can also stay open and stop delivering anything, which HTTP/2 keep-alive does not detect while the connection itself still answers. To reset such a stream, set idle_timeout in the listener params:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let listener = db
    .create_listener_with_params(
        FirestoreMemListenStateStorage::new(),
        FirestoreListenerParams::new().with_idle_timeout(Duration::from_secs(120)),
    )
    .await?;
Ok(())
}

It is off by default. When set:

• any ListenResponse counts as activity, NO_CHANGE target changes included;
• the time counts from when the stream is open, so a connect that never gets that far is not covered, and only while at least one target on the stream waits for an answer from Firestore, not while every target waits to be added back after the retry delay;
• the time starts again once the callback and the token storage have handled each response;
• on expiry the listener waits the retry delay and reconnects, adding every target back with its stored resume token. Changes after the last stored token may be delivered again, as after any reconnect.

120 seconds is the recommended value, and the one Google’s Node.js client uses. Firestore documents no interval for the updates it sends on a quiet stream, so a much shorter value can reset healthy streams.

Full example available here.

A listener takes Core targets only: a query or documents by ID. Pipelines and SQL statements cannot be listened to, and their writes reach a listener as any other change, see Retries and listeners.

On the Enterprise edition

A listener the application opens receives changes on an Enterprise database that has real-time updates enabled; Google ties listener support (onSnapshot) to that setting, which is fixed when the database is created, see Settings fixed at creation. Real-time updates are billed separately on Enterprise, see Cost per edition.

Working on dynamic/document level

Sometimes having static structure may restrict you from working with dynamic data, so there is a way to use Fluent API to work with documents without introducing structures at all.

use firestore::*;
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let object_returned = db
    .fluent()
    .insert()
    .into(TEST_COLLECTION_NAME)
    .document_id("test-1")
    .document(FirestoreDb::serialize_map_to_doc(
        "",
        [
            ("some_id", "test-id".into()),
            ("some_string", "test-value".into()),
            ("some_num", 42.into()),
            (
                "embedded_obj",
                FirestoreValue::from_map([
                    ("inner_some_id", "inner-id-value".into()),
                    ("inner_some_string", "inner-some-value".into()),
                ]),
            ),
            ("created_at", FirestoreTimestamp::now().into()),
        ],
    )?)
    .execute()
    .await?;
let _ = object_returned;
Ok(())
}

Full example available here.

Create, update and delete documents

The library provides insert(), update() and delete() in the Fluent API for writing a single document. Each of them sends one request when you call .execute(). To write several documents together, see Transactions and Batch writes.

Creating a document

insert() creates a document in a collection, and fails when a document with the same ID already exists:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
    some_num: u64,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let my_struct = MyTestStructure {
    some_id: "test-1".to_string(),
    some_string: "Test".to_string(),
    some_num: 42,
};

let object_returned: MyTestStructure = db
    .fluent()
    .insert()
    .into(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&my_struct)
    .execute()
    .await?;
Ok(())
}

The conflict is reported as FirestoreError::DataConflictError.

.execute() returns the stored document deserialised into the type you ask for. Use .execute::<()>() when you do not need it.

To have an ID chosen for you, call .generate_document_id() instead of .document_id(..). The library generates it as Firestore’s own clients do: 20 random letters and digits. A write that is sent again after a failure carries the same ID, so it cannot create a second document. The generated ID comes back in a field marked with the _firestore_id alias, see Document metadata as struct fields:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyGeneratedStructure {
    #[serde(alias = "_firestore_id")]
    id: Option<String>,
    some_string: String,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let created: MyGeneratedStructure = db
    .fluent()
    .insert()
    .into(TEST_COLLECTION_NAME)
    .generate_document_id()
    .object(&MyGeneratedStructure {
        id: None,
        some_string: "Test".to_string(),
    })
    .execute()
    .await?;

let generated_id: Option<String> = created.id;
Ok(())
}

Updating a document

update() writes the object to the document with the given ID. Without .fields(..) the object replaces the whole document, and the fields it does not have are removed. With .fields(..) only the listed fields are written, and the rest of the stored document is left as it is:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
    some_num: u64,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb, my_struct: MyTestStructure) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Replaces the document
db.fluent()
    .update()
    .in_col(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&my_struct)
    .execute::<()>()
    .await?;

// Writes `some_num` only
let object_updated: MyTestStructure = db
    .fluent()
    .update()
    .fields(paths!(MyTestStructure::{some_num}))
    .in_col(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&MyTestStructure {
        some_num: my_struct.some_num + 1,
        ..my_struct.clone()
    })
    .execute()
    .await?;
Ok(())
}

update() also creates the document when it does not exist yet, and then stores only the listed fields when .fields(..) is given. Add a precondition to make it fail on a missing document, or to make it create only, see Update/delete preconditions. Server side changes such as incrementing a number are in Document transformations.

Deleting a document

delete() removes the document with the given ID. Deleting a document that does not exist is not an error:

use firestore::*;
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .delete()
    .from(TEST_COLLECTION_NAME)
    .document_id("test-1")
    .execute()
    .await?;
Ok(())
}

Documents in the subcollections of a deleted document stay in place. To remove a whole collection see Bulk delete.

When a write fails

The library sends insert(), update() and delete() again after RESOURCE_EXHAUSTED and UNAVAILABLE, up to max_retries times with a growing delay. These are the codes that Google’s Firestore server libraries retry a write on, see Single writes. When the error still comes back after the last retry, the write may or may not be applied.

Other failures to expect from a single write:

• ABORTED: Firestore received too many updates for one document per second, and answers with “Too much contention on these datastore entities. Please try again.” Google’s Java, Go and Python server libraries do not retry it on a single write, and neither does the library. Lower the rate of writes to that document;
• ALREADY_EXISTS, as FirestoreError::DataConflictError: an insert() of an ID that exists, or an update() with Exists(false) on a document that exists. After a retried write it can mean that the first attempt created the document;
• NOT_FOUND, as FirestoreError::DataNotFoundError: a delete() with Exists(true) on a document that does not exist. After a retried delete() it can mean that the first attempt deleted the document;
• FAILED_PRECONDITION: the precondition of the write did not hold, see Update/delete preconditions. After a retried write with an UpdateTime precondition it can mean that the first attempt was applied.

All three builders accept .parent(..) for documents in nested collections. insert() and update() accept .document(..) instead of .object(..) for working on dynamic/document level.

Full example available here.

Transactions

To manage transactions manually you can use db.begin_transaction(), and then the Fluent API to add the operations needed in the transaction.

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String, some_string: String }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut transaction = db.begin_transaction().await?;

db.fluent()
    .update()
    .fields(paths!(MyTestStructure::{
      some_string
    }))
    .in_col(TEST_COLLECTION_NAME)
    .document_id("test-0")
    .object(&MyTestStructure {
        some_id: format!("test-0"),
        some_string: "UpdatedTest".to_string(),
    })
    .add_to_transaction(&mut transaction)?;

db.fluent()
    .delete()
    .from(TEST_COLLECTION_NAME)
    .document_id("test-5")
    .add_to_transaction(&mut transaction)?;

transaction.commit().await?;
Ok(())
}

You may also execute transactions that automatically retry with exponential backoff using run_transaction.

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { test_string: String }
const TEST_COLLECTION_NAME: &str = "test";
const TEST_DOCUMENT_ID: &str = "test_doc_id";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.run_transaction(|db, transaction| {
    Box::pin(async move {
        let mut test_structure: MyTestStructure = db
            .fluent()
            .select()
            .by_id_in(TEST_COLLECTION_NAME)
            .obj()
            .one(TEST_DOCUMENT_ID)
            .await?
            .expect("Missing document");

        // Perform some kind of operation that depends on the state of the document
        test_structure.test_string += "a";

        db.fluent()
            .update()
            .fields(paths!(MyTestStructure::{
             test_string
            }))
            .in_col(TEST_COLLECTION_NAME)
            .document_id(TEST_DOCUMENT_ID)
            .object(&test_structure)
            .add_to_transaction(transaction)?;

        Ok(())
    })
})
.await?;
Ok(())
}

The transaction retries automatically on a transient failure: the callback returning BackoffError::Transient (a plain ? on a FirestoreResult inside the callback already converts to one), or commit() answering ABORTED. Either way the whole transaction reruns from the start as a new attempt, so the callback must be safe to run more than once; a failed attempt is rolled back first, releasing its locks, before the retry begins. A read inside the transaction that comes back ABORTED is not retried on its own; it fails the attempt, which then retries as usual.

Commit is sent only once per attempt. Any failure other than ABORTED, such as UNAVAILABLE or a dropped connection, is returned to the caller without retrying, since the writes may already be applied, and the callback is not run again in that case.

Retries are capped at max_retries attempts after the first, 4 by default (5 attempts in total), the same default as Google’s official clients; retry_max_elapsed_time adds an optional time cap on top, off by default. Each retry waits a backoff delay first, or the delay the callback named through BackoffError::Transient’s retry_after. Return Err(BackoffError::Permanent(err)) instead for an error that must not be retried; it stops immediately and comes back wrapped in FirestoreError::ErrorInTransaction.

Set max_retries and retry_max_elapsed_time through run_transaction_with_options:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { test_string: String }
const TEST_COLLECTION_NAME: &str = "test";
const TEST_DOCUMENT_ID: &str = "test_doc_id";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use std::time::Duration;

let options = FirestoreTransactionOptions::new()
    .with_max_retries(2)
    .with_retry_max_elapsed_time(Duration::from_secs(30));

db.run_transaction_with_options(
    |db, _transaction| {
        Box::pin(async move {
            let _existing: Option<MyTestStructure> = db
                .fluent()
                .select()
                .by_id_in(TEST_COLLECTION_NAME)
                .obj()
                .one(TEST_DOCUMENT_ID)
                .await?;
            Ok(())
        })
    },
    options,
)
.await?;
Ok(())
}

See the complete example available here.

A Core write in a transaction cannot create a document with an ID that Firestore generates. To create a document in a transaction, use update() with an ID of your own, which creates the document when it does not exist. A pipeline insert without .document_id(..) is the exception on the Enterprise edition: it gets an ID from Firestore and runs in a transaction, see Pipeline writes.

Transactions and the cache

A read in a transaction never uses the cache. The same holds for a handle from clone_with_consistency_selector, which reads at a read time:

• under read_through_cache, get, batch get, query and list are answered by Firestore, so the transaction reads and locks the documents. A document read this way is not written to the cache;
• under read_cached_only, the same reads return FirestoreError::CacheError with the code CacheConsistencySelectorUnsupported, because that mode never contacts Firestore. A transaction needs a read_through_cache handle instead.

Inside run_transaction, a ? on that error makes it transient, so the transaction is retried 4 times, with a few seconds of backoff, before the error is returned wrapped in FirestoreError::ErrorInTransaction. Return BackoffError::permanent from the closure to fail on the first attempt.

On the Enterprise edition

Core aggregations and transactions behave the same on both editions; under contention an Enterprise database, which is optimistic by default, aborts and retries transactions more often than a Standard one. The concurrency mode is a setting of the database, shown as concurrencyMode by gcloud firestore databases describe, see Telling which edition a database is.

A transaction on Enterprise can also hold pipeline and SQL writes. They are sent when they run, where Core writes are buffered and sent with the commit. A transaction takes pipeline writes or buffered writes: Firestore rejects a commit that mixes them with INVALID_ARGUMENT.

Batch writes

The library supports batch writes as an alternative to transactions for bulk loads and migrations. A batch write does not commit atomically: each queued write succeeds or fails on its own. Use a transaction instead when the writes need to commit together.

There are two batch writers:

• db.create_simple_batch_writer(): one Firestore BatchWrite request per batch, the right default for occasional or moderate-sized batches;
• db.create_streaming_batch_writer(): one long-lived Write stream for many batches sent back-to-back, throttled to stay under Firestore’s per-stream write rate limit.

The simple writer

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let batch_writer = db.create_simple_batch_writer().await?;

let mut current_batch = batch_writer.new_batch();

for idx in 0..500 {
    let my_struct = MyTestStructure {
        some_id: format!("test-{idx}"),
        some_string: "Test".to_string(),
    };

    db.fluent()
        .update()
        .in_col(TEST_COLLECTION_NAME)
        .document_id(&my_struct.some_id)
        .object(&my_struct)
        .add_to_batch(&mut current_batch)?;

    if idx % 100 == 0 {
        let response = current_batch.write().await?;
        current_batch = batch_writer.new_batch();
        println!("{response:?}");
    }
}
Ok(())
}

write() returns a FirestoreBatchWriteResponse with write_results and statuses for each queued write, in order, plus a commit_time. One failing write does not stop the rest from being reported. The request retries transient failures with exponential backoff; FirestoreSimpleBatchWriteOptions::retry_max_elapsed_time bounds how long, 60 seconds by default; set it through db.create_simple_batch_writer_with_options(...), or to None for no limit.

Full example available here.

The streaming writer

use firestore::*;
use futures::TryStreamExt;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_id: String,
    some_string: String,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let (batch_writer, mut batch_results_reader) = db.create_streaming_batch_writer().await?;

let response_thread = tokio::spawn(async move {
    while let Ok(Some(response)) = batch_results_reader.try_next().await {
        println!("{response:?}");
    }
});

let mut current_batch = batch_writer.new_batch();
for idx in 0..10000 {
    let my_struct = MyTestStructure {
        some_id: format!("test-{idx}"),
        some_string: "Test".to_string(),
    };

    db.fluent()
        .update()
        .in_col(TEST_COLLECTION_NAME)
        .document_id(&my_struct.some_id)
        .object(&my_struct)
        .add_to_batch(&mut current_batch)?;

    if idx % 100 == 0 {
        current_batch.write().await?;
        current_batch = batch_writer.new_batch();
    }
}

batch_writer.finish().await;
let _ = tokio::join!(response_thread);
Ok(())
}

The response stream yields one FirestoreBatchWriteResponse per batch, in send order, with position as the batch’s sequence number in the stream. statuses is always empty on this writer; Firestore’s Write RPC does not report per-write status the way BatchWrite does. A write failure fails the whole stream instead, surfacing as an Err on the response stream and ending it.

Consume the response stream, for example from a spawned task as above, before calling finish(), or it blocks forever waiting for responses nobody reads. Be aware not to just drop the writer instead of calling finish(); that only logs a warning and leaves its background task running.

options.throttle_batch_duration paces how often batches go out over the stream, 500ms by default. See request tags for attaching request tags through options.request_options on either writer.

Full example available here.

Adding writes to a batch

update() and delete() both have add_to_batch(&mut batch), the same as add_to_transaction(&mut transaction) on a transaction. See document transformations for .transforms(...).

insert() has no add_to_batch: a batch write cannot create a document with a server-generated ID. Use update() with an explicit document_id to create one instead.

For writes the fluent builders do not cover, queue them directly on FirestoreBatch: update_object, delete_by_id and transform, plus _at variants for an explicit parent path. Each takes an optional write precondition. On the simple writer a failed precondition fails just that one write; on the streaming writer it fails the whole stream.

On the Enterprise edition

An Enterprise edition database can hold unique indexes, which refuse a write that would give two documents the same entry. Such a write returns DataConflictError, see Unique indexes.

Document transformations

The library supports server side document transformations in transactions, batch writes and single updates:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_num: i32, some_array: Vec<i32> }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb, my_obj: MyTestStructure) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut transaction = db.begin_transaction().await?;
// Only transformation
db.fluent()
    .update()
    .in_col(TEST_COLLECTION_NAME)
    .document_id("test-4")
    .transforms(|t| {
        // Transformations
        t.fields([
            t.field(path!(MyTestStructure::some_num)).increment(10),
            t.field(path!(MyTestStructure::some_array))
                .append_missing_elements([4, 5]),
            t.field(path!(MyTestStructure::some_array))
                .remove_all_from_array([3]),
        ])
    })
    .only_transform()
    .add_to_transaction(&mut transaction)?; // or add_to_batch

// Update and transform (in this order and atomically):
db.fluent()
    .update()
    .in_col(TEST_COLLECTION_NAME)
    .document_id("test-5")
    .object(&my_obj) // Updating the objects with the fields here
    .transforms(|t| {
        // Transformations after the update
        t.fields([t.field(path!(MyTestStructure::some_num)).increment(10)])
    })
    .add_to_transaction(&mut transaction)?; // or add_to_batch
Ok(())
}

A single update with .execute() applies the object and then the transformations, in this order and atomically, and returns the document as stored by that write:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_num: i32, some_array: Vec<i32> }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb, my_obj: MyTestStructure) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let updated: MyTestStructure = db
    .fluent()
    .update()
    .in_col(TEST_COLLECTION_NAME)
    .document_id("test-5")
    .object(&my_obj)
    .transforms(|t| t.fields([t.field(path!(MyTestStructure::some_num)).increment(10)]))
    .execute()
    .await?;
Ok(())
}

The update is sent as a commit, and the document is read back at the time of that commit, with .object(..) as with .document(..). When the read fails, .execute() returns its error although the write was applied. A transformation-only update with .only_transform() has no .execute(): queue it on a transaction or a batch.

Be aware that a sent-again update applies a transformation such as increment twice when the first attempt was applied, see Single writes.

Update/delete preconditions

The library supports the preconditions:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    some_num: u64,
}
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let object_updated: MyTestStructure = db
    .fluent()
    .update()
    .fields(paths!(MyTestStructure::{some_num}))
    .in_col(TEST_COLLECTION_NAME)
    .precondition(FirestoreWritePrecondition::Exists(true))
    .document_id("test-1")
    .object(&MyTestStructure { some_num: 1 })
    .execute()
    .await?;
let _ = object_updated;
Ok(())
}

Concurrent writes to one document

Single writes to the same document are applied one after another, and the last one wins. There is no conflict error between them: an update() overwrites the fields an earlier write stored, whether or not its sender saw that write. A document that gets more writes per second than it can take fails them with ABORTED, see When a write fails.

FirestoreWritePrecondition::UpdateTime makes a write conditional on the document not having changed since it was read. A write that finds a different update time fails with FAILED_PRECONDITION, as a DatabaseError with the code FailedPrecondition. The library does not retry it, and sending the same write again fails the same way. The error also comes back when the library sent the write again after a first attempt that was applied, see Single writes.

To read a document and write a value computed from it, use run_transaction, which runs the whole function again when the document changed, see Transactions.

Pipelines

A pipeline is a query that Firestore runs as an ordered list of stages. The first stage is a source, such as a collection; every later stage takes the documents of the stage before it and passes its own output on: filter, sort, fields, limit, and so on. The library sends it through Firestore’s ExecutePipeline call.

Pipelines need a database of the Firestore Enterprise edition, the edition that runs them, see Standard and Enterprise editions. A pipeline sent to a Standard edition database fails at once with FAILED_PRECONDITION, and the library does not retry it, unless a cache session answers it, see Caching pipelines.

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Book { title: String, author: String, rating: Option<f64>, stock: i64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let books: Vec<Book> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .filter(|q| {
        q.for_all([
            q.field(path!(Book::rating)).greater_than_or_equal(4.2),
            q.field(path!(Book::stock)).greater_than(0),
        ])
    })
    .sort(|e| [e.field(path!(Book::rating)).desc()])
    .limit(10)
    .obj()
    .query()
    .await?;
let _ = books;
Ok(())
}

Full example available here.

When to use a pipeline

The Core query API (db.fluent().select()) stays the choice for a Standard edition database, and for what it covers. A pipeline adds grouped aggregations, computed fields, joins, search, writes by query and Query Explain on Enterprise, see Choosing Core queries, pipelines or SQL.

Indexes are optional for a pipeline, except for a search. A pipeline without an index that covers its filter runs as a table scan, and costs by the data it scans, see Query plans and cost.

A pipeline with the same filter and order as a Core query can return other documents: not_equal matches a document that lacks the field, sort keeps such documents, and sort adds no tie-break on the document name. See Query and write semantics.

Sources

db.fluent().pipeline().select() starts a pipeline that reads documents, and the next call picks the source, in the words of Core’s select().from(..):

• .from("books") reads a root collection;
• .from("reviews").parent(db.parent_path("restaurants", "rest-01")?) reads the collection under a document, as parent does in Core;
• .from("reviews").all_descendants() reads every collection with that ID, anywhere in the database, as a Core collection-group query does;
• .documents(["books/b1", "books/b2"]) reads the documents at those paths, relative to the database’s documents root; a path that names no document is left out;
• .literals(rows) starts from rows given in the request instead of stored documents, and every row serialises to a map;
• .database() reads every document of the database;
• .from_pipeline(pipeline) runs a pipeline built with the model types, such as FirestorePipeline.

parent and all_descendants exclude each other: a pipeline that calls both does not compile, since Firestore ignores a parent on a collection group.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let parent = db.parent_path("restaurants", "rest-01")?;

let reviews = db
    .fluent()
    .pipeline()
    .select()
    .from("reviews")
    .parent(parent)
    .query()
    .await?;

let every_review = db
    .fluent()
    .pipeline()
    .select()
    .from("reviews")
    .all_descendants()
    .query()
    .await?;
let _ = (reviews, every_review);
Ok(())
}

Filters

.filter() takes the same filter builder q as Core. A condition starts with q.field(path!(Book::rating)) and ends in a comparison; eq, neq, is_in and is_not_in keep Core’s names. q.for_all joins conditions with AND and q.for_any with OR; both nest, and both drop a None entry, so an optional condition is written as it is in Core:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Book { title: String, genre: Option<String>, rating: Option<f64>, stock: i64 }
async fn example(db: FirestoreDb, genre: Option<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let books: Vec<Book> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .filter(|q| {
        q.for_all([
            q.for_any([
                q.field(path!(Book::stock)).greater_than(0),
                q.field(path!(Book::title)).starts_with("The "),
            ]),
            genre.and_then(|genre| q.field(path!(Book::genre)).eq(genre)),
        ])
    })
    .obj()
    .query()
    .await?;
let _ = books;
Ok(())
}

A closure that returns None, such as a for_all of conditions that are all None, adds no stage and keeps every document.

Besides the comparisons, the builder has is_null, is_nan, exists, is_absent, is_type, starts_with, ends_with, string_contains, like, regex_contains, regex_match, array_contains, array_contains_all, array_contains_any and equal_any. A condition on a computed value starts with q.expr(..), and q.condition(..) adds a condition built with the value builder below.

The expression builder

Every closure of a pipeline except filter receives the value builder e, whose conditions are plain values rather than Options. It starts expressions and combines them:

• e.field(path!(Book::title)) is a field of the current document;
• e.constant(10) is a constant; a value passed to a comparison becomes one without it;
• e.and([..]), e.or([..]), e.not(..) combine conditions;
• e.conditional(condition, if_true, if_false), e.coalesce([..]), e.array([..]) and e.map([..]) build values;
• e.variable("name") reads a variable set by define_variables.

An operation on one expression is a method on it: e.field("price").multiply(e.field("count")), e.field("title").to_lower(), e.field("tags").array_length(). A function without a method of its own is called with e.raw("name", [args]).

Projection and computed fields

.fields() replaces each document with the fields it returns; an item that is not a plain field needs an alias. .add_fields() computes fields and keeps the others, and .remove_fields() drops fields by path:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Book { title: String, author: String, pages: i64 }
#[derive(Debug, Clone, Deserialize)]
struct Summary {
    #[serde(alias = "_firestore_id")]
    id: Option<String>,
    title: String,
    author_name: String,
    reading_hours: f64,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let summaries: Vec<Summary> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .fields(|e| {
        [
            FirestoreSelection::from(e.field(path!(Book::title))),
            e.field(path!(Book::author)).alias("author_name"),
            e.field(path!(Book::pages)).divide(40.0).alias("reading_hours"),
        ]
    })
    .obj()
    .query()
    .await?;
let _ = summaries;
Ok(())
}

A computed item without an alias is rejected with InvalidParametersError before the request is sent. The items of one list can have different types, so a list that mixes plain fields and aliased ones starts its plain field with FirestoreSelection::from(..), as above.

Sorting and paging

.sort() takes a list of orderings, e.field(..).asc() or .desc(). .limit(n) keeps the first n documents and .offset(n) skips the first n. They are stages, so their position matters: a limit before a sort limits the documents first and sorts what is left, which is not what the same two calls mean in Core.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let page = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .sort(|e| [e.field("rating").desc(), e.field("title").asc()])
    .offset(20)
    .limit(10)
    .query()
    .await?;
let _ = page;
Ok(())
}

Typed results and metadata

.obj::<T>() deserialises the results into T, and .query(), .stream_query(), .stream_query_with_errors() and .stream_query_with_metadata() have the shapes they have in Core. Without .obj() the results are FirestoreDocuments.

A stored document carries its name, so _firestore_id and the timestamp fields are filled as they are in Core. The result of .fields(), .aggregate() and the other stages that build new documents has no name, which leaves _firestore_id out: a field of T for it must be an Option. Selecting e.field("__name__") keeps the document name in the result, without the timestamps.

.stream_query_with_metadata() returns each result as a FirestoreWithPipelineMetadata<T>: the document, and the metadata of the response it arrived in, with its execution_time. A response that carries explain statistics, or no documents at all, adds an item without a document.

.request_tags(..) and .request_options(..) attach request options to one pipeline, as they do for a Core query.

A session created with read_through_cache or read_cached_only answers a pipeline from the cache when it only filters, sorts and pages cached documents, see Caching pipelines.

Query plans and indexes

.explain() and .explain_with_options(..) return the plan Firestore chose for a pipeline, and .force_index(..) on a from(..) source picks the index it reads through. Both are described in Query plans and cost, with how to recognise a table scan in the plan.

Pipeline aggregations and joins

The stages in this chapter build new documents from the ones before them, so their results have no document name. A field of the result type for _firestore_id must be an Option, as described in Pipelines.

Aggregations

.aggregate() computes accumulators over all documents and outputs one document that holds them. Each accumulator needs an alias, which names its field in the result:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Totals { total: i64, in_stock: i64, average_rating: Option<f64> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let totals: Vec<Totals> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .aggregate(|e| {
        [
            e.count_all().alias("total"),
            e.field("stock").greater_than(0).count_if().alias("in_stock"),
            e.field("rating").average().alias("average_rating"),
        ]
    })
    .obj()
    .query()
    .await?;
let _ = totals;
Ok(())
}

The accumulators are count_all on e, which counts every document whether or not it has any field, and these on an expression: count, count_distinct, sum, average, minimum, maximum, first, last, array_agg and array_agg_distinct, plus count_if on a condition. They are separate from the scalar functions of the same names: sum, minimum and maximum inside .aggregate() accumulate over documents, and the array versions are array_sum, array_minimum and array_maximum.

.aggregate_grouped(groups, accumulators) outputs one document per group, with the group’s values and its accumulators. A null value is a group of its own:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct GenreStats { genre: Option<String>, books: i64, longest: i64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let stats: Vec<GenreStats> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .aggregate_grouped(
        |e| [e.field(path!(GenreStats::genre))],
        |e| {
            [
                e.count_all().alias("books"),
                e.field("pages").maximum().alias("longest"),
            ]
        },
    )
    .sort(|e| [e.field(path!(GenreStats::books)).desc()])
    .obj()
    .query()
    .await?;
let _ = stats;
Ok(())
}

Full example available here.

Distinct values

.distinct() outputs one document per distinct combination of the values it selects, and a null is a value of its own:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let genres = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .distinct(|e| [e.field("genre")])
    .query()
    .await?;
let _ = genres;
Ok(())
}

Sampling

.sample() keeps a random sample of the documents: a number of documents, or a fraction from 0.0 to 1.0 of them. Firestore names the second mode percent, but it takes a fraction.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let some_books = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .sample(FirestoreSampleSize::Documents(3))
    .query()
    .await?;

let tenth = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .sample(FirestoreSampleSize::Fraction(0.1))
    .query()
    .await?;
let _ = (some_books, tenth);
Ok(())
}

Union

.union() appends the documents of another pipeline. The closure receives the same source picker as a subquery does, so it starts with p.from(..):

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let names = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .fields(|e| [e.field("title").alias("name")])
    .union(|p| p.from("restaurants").fields(|e| [e.field("name")]))
    .query()
    .await?;
let _ = names;
Ok(())
}

Unnest

.unnest() outputs one document per element of an array field, with the element under the alias of the selection. .unnest_with_index_field() also outputs each element’s position in the array. A selection without an alias replaces the array field with its elements:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let tags = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .unnest_with_index_field(|e| e.field("tags").alias("tag"), "tag_index")
    .fields(|e| [e.field("title"), e.field("tag"), e.field("tag_index")])
    .query()
    .await?;
let _ = tags;
Ok(())
}

Replacing documents

.replace_with(expr, mode) reshapes each document in the pipeline’s output and changes nothing in storage. The stored documents stay as they are; only the documents the pipeline returns have the new shape. Stored documents change only through the write stages, see Pipeline writes.

expr must evaluate to a map. The mode says how the map and the document combine:

• FirestoreReplaceWithMode::FullReplace: the map becomes the document, and the document’s own fields are dropped;
• FirestoreReplaceWithMode::MergeOverwriteExisting: the map’s fields are added to the document, and replace its fields of the same name;
• FirestoreReplaceWithMode::MergeKeepExisting: the map’s fields are added to the document, and its fields of the same name stay.

For books/book-10, which has the fields title (“Dune”), author, genre, awards (a map of hugo and nebula) and others, each mode outputs:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let awards = db
    .fluent()
    .pipeline()
    .select()
    .documents(["books/book-10"])
    .replace_with(|e| e.field("awards"), FirestoreReplaceWithMode::FullReplace)
    .query()
    .await?;
let _ = awards;
Ok(())
}

The query returns the two award fields, and a read of books/book-10 afterwards still returns the whole book. The same holds for the two merge modes.

A write stage stores what the stages before it output, so a write after replace_with stores the replaced documents. This insert stores the awards of books/book-10 as the document book-awards/book-10, and leaves books/book-10 as it is:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let written = db
    .fluent()
    .pipeline()
    .insert()
    .into("book-awards")
    .document_id(|e| e.constant("book-10"))
    .documents(["books/book-10"])
    .replace_with(|e| e.field("awards"), FirestoreReplaceWithMode::FullReplace)
    .atomic()
    .execute()
    .await?;
let _ = written;
Ok(())
}

Nearest neighbours

.find_nearest(field, vector, measure, limit) keeps the limit documents whose vector field is nearest to the query vector. It takes Core’s FirestoreVector and FirestoreFindNearestDistanceMeasure, and has no counterpart to Core’s distance_threshold. .find_nearest_with_options() takes a FirestoreFindNearestStage, which can also name a field to output each document’s distance under.

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Nearest { title: String, distance: f64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let nearest: Vec<Nearest> = db
    .fluent()
    .pipeline()
    .select()
    .from("books")
    .find_nearest_with_options(
        FirestoreFindNearestStage::new(
            "embedding".to_string(),
            FirestoreVector::new(vec![0.9, 0.1, 0.2]),
            FirestoreFindNearestDistanceMeasure::Euclidean,
            3,
        )
        .with_distance_result_field("distance".to_string()),
    )
    .fields(|e| [e.field("title"), e.field("distance")])
    .obj()
    .query()
    .await?;
let _ = nearest;
Ok(())
}

The stage does not use a vector index, so its cost follows the size of the collection, see Known table scans. A Core vector query, select().find_nearest(..), has a distance threshold and uses a vector index on a Standard edition database, see Vector search.

Joins

Joins are in Preview: Firestore announced them without a launch stage, so treat their behaviour as subject to change.

A join adds to each document the result of another pipeline, a subquery. A subquery is correlated when it reads a value of the document it runs for. Pipelines pass that value through a variable: .define_variables() sets variables for the stages after it, and e.variable("name") or q.variable("name") reads one, inside a subquery as well.

e.array_subquery(..) returns the results of its pipeline as an array, which is empty when the pipeline finds nothing:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Review { rating: i64, reviewer: String }
#[derive(Debug, Clone, Deserialize)]
struct RestaurantWithReviews { name: String, reviews: Vec<Review> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let restaurants: Vec<RestaurantWithReviews> = db
    .fluent()
    .pipeline()
    .select()
    .from("restaurants")
    .filter(|q| q.field("cuisine").eq("breakfast"))
    .define_variables(|e| [e.field("__name__").alias("restaurant")])
    .fields(|e| {
        [
            FirestoreSelection::from(e.field("name")),
            e.array_subquery(|p| {
                p.from("reviews")
                    .all_descendants()
                    .filter(|q| q.field("restaurant").eq(q.variable("restaurant")))
                    .sort(|e| [e.field("rating").desc()])
                    .fields(|e| [e.field("rating"), e.field("reviewer")])
            })
            .alias("reviews"),
        ]
    })
    .obj()
    .query()
    .await?;
let _ = restaurants;
Ok(())
}

The subquery closure receives the source picker directly, p.from("reviews"), since a nested pipeline has no verb. The example above matches reviews to restaurants through a restaurant reference field on each review.

e.scalar_subquery(..) returns the one result of its pipeline: null when the pipeline finds nothing, and an INVALID_ARGUMENT failure of the whole request, “Subpipeline returned multiple results”, when it finds several. Use it with an aggregation, which always yields one document:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct RestaurantWithAverage { name: String, average_rating: Option<f64> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let restaurants: Vec<RestaurantWithAverage> = db
    .fluent()
    .pipeline()
    .select()
    .from("restaurants")
    .define_variables(|e| [e.field("__name__").alias("restaurant")])
    .fields(|e| {
        [
            FirestoreSelection::from(e.field("name")),
            e.scalar_subquery(|p| {
                p.from("reviews")
                    .all_descendants()
                    .filter(|q| q.field("restaurant").eq(q.variable("restaurant")))
                    .aggregate(|e| [e.field("rating").average().alias("average")])
            })
            .alias("average_rating"),
        ]
    })
    .obj()
    .query()
    .await?;
let _ = restaurants;
Ok(())
}

A subquery whose last stage outputs a single field returns the bare values of that field, and one that outputs two or more fields returns maps. The average_rating above is a number rather than a map with an average field, and a subquery that selects only rating returns an array of numbers:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct RestaurantWithRatings { name: String, ratings: Vec<i64> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let restaurants: Vec<RestaurantWithRatings> = db
    .fluent()
    .pipeline()
    .select()
    .from("restaurants")
    .fields(|e| {
        [
            FirestoreSelection::from(e.field("name")),
            e.array_subquery(|p| p.subcollection("reviews").fields(|e| [e.field("rating")]))
                .alias("ratings"),
        ]
    })
    .obj()
    .query()
    .await?;
let _ = restaurants;
Ok(())
}

p.subcollection("reviews") reads the collection under the current document of the enclosing pipeline, so it needs no variable. Only a subquery or a union branch can start with it: a top-level pipeline that does does not compile.

Full example available here.

Pipeline search

A pipeline can search a collection for text or for places near a point. A search is part of the source: .search(..) follows .from(..) directly, and every other stage, such as filter, fields or sort, comes after it.

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Hit { name: String, score: f64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let hits: Vec<Hit> = db
    .fluent()
    .pipeline()
    .select()
    .from("restaurants")
    .search(|s| {
        s.text(|t| t.for_all([t.term("waffles"), t.exclude(t.term("bacon"))]))
            .sort_by_score()
            .score_field("score")
            .limit(20)
    })
    .fields(|e| [e.field("name"), e.field("score")])
    .obj()
    .query()
    .await?;
let _ = hits;
Ok(())
}

A search needs a search index that covers it, see Search indexes. Without one the request fails with FAILED_PRECONDITION, and the library does not retry it. A search takes a collection or a collection group as its source, and nothing else; a collection-group search needs an index scoped to the collection group.

Text search

s.text(..) searches every indexed text field of the collection. Its closure builds the query from three kinds of clause, in the style of the filter builder q:

• t.term("waffles") is a word the document must contain; matching ignores case and accents and finds other forms of the word, so waffle finds “Waffles”;
• t.phrase("maple syrup") is a list of words the document must contain in that order; a phrase is matched as written, without other forms of the words, so the phrase "waffle" does not find a document that contains “waffles”, where the term waffle does;
• t.exclude(..) requires the term or phrase to be absent.

t.for_all([..]) lists the clauses that must all hold. It takes plain clauses and Options of them, and drops a None, so an optional clause is written as an optional filter condition is. A query with no clause that must match, such as one that only excludes, or a term without text, matches nothing in Firestore, so the library rejects it with InvalidParametersError before the request is sent.

The closure may also return a string, which is sent as written, for a query the builder does not build.

What the builder renders

The service reads the query text with a grammar of its own, and the builder renders each clause so that the text means what the clause says. FirestoreTextQuery implements Display, which shows the text that is sent:

use firestore::pipeline_search_builder::FirestoreTextQueryBuilder;

let t = FirestoreTextQueryBuilder;

let query = t
    .for_all([t.term("waffles"), t.exclude(t.phrase("maple syrup"))])
    .expect("the query has clauses");
assert_eq!(query.to_string(), r#"waffles -"maple syrup""#);

// A word the service would read as an operator, an exclusion or a field scope is quoted.
let query = t.for_all([t.term("menu:waffles"), t.term("-bacon")]).unwrap();
assert_eq!(query.to_string(), r#""menu:waffles" "-bacon""#);

A quoted word is matched as written, without other forms of the word. A quote character inside a phrase is replaced by a space.

What the grammar supports

The query grammar is Firestore’s, and it is narrow. These are its limits:

• terms separated by spaces must all match; there is no OR, and pancakes OR crepes matches only the documents that contain both words;
• there is no field scoping: menu:waffles is read as the two terms menu and waffles, so the search always covers every indexed text field;
• parentheses do not group;
• there is no prefix or wildcard syntax: waff* and w?ffles match nothing.

FirestoreTextQuery leaves OR and field scoping out for that reason.

Geo search

s.geo_within(field, center, max_distance_meters) keeps the documents whose geo point field lies at most that distance, in metres, from the centre:

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Name { name: String }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let center = FirestoreLatLng(FirestoreGeoPoint {
    latitude: 38.989177,
    longitude: -107.065076,
});

let nearby: Vec<Name> = db
    .fluent()
    .pipeline()
    .select()
    .from("restaurants")
    .search(|s| s.geo_within("location", center, 1000.0).sort_by_distance())
    .fields(|e| [e.field("name")])
    .obj()
    .query()
    .await?;
let _ = nearby;
Ok(())
}

Firestore accepts only the “at most” comparison, so the search has no “farther than” form. A single search is a text search or a geo search; combining the two in one search is not supported by Firestore.

Full example available here.

Options

Both kinds of search take these options:

• .limit(n) keeps the first n results, and .offset(n) skips the first n; an offset needs a limit;
• .retrieval_depth(n) reads at most n index entries before the results are scored and sorted; it must be at least 1;
• .language_code("en") sets the language of the query as a BCP 47 language tag, the standard short code for a language, such as en.

A text search also has .sort_by_score() and .score_field("score"); a geo search has .sort_by_distance(). Each ordering has a single direction, because Firestore accepts no other: the most relevant result first for a score, and the nearest first for a distance. .score_field(..) outputs the relevance score of each document under a field, where a later stage can read it. The score and the distance are not general expressions: Firestore rejects them outside the search.

A .filter(..) after the search narrows its matches.

Search indexes

A search runs only against search indexes:

• a text search needs a text index on the fields it searches, and Firestore searches all of the indexed text fields of the collection, never one by name;
• a geo search needs a geo index on the geo point field;
• one index can hold text fields and geo fields together, and mixing search fields with ordinary ones is rejected;
• a collection-group search needs indexes scoped to the collection group.

Index management declares them in Rust, with i.field(..).text() and i.field(..).geo(), and creates them with .sync().

Pipeline writes

A pipeline can end in a write: it changes the documents that its source and stages select. A write starts with its verb, then names its source, so the direction reads left to right, as in Core: delete().from(..), update(..).in_col(..), insert().into(..), upsert(..).into(..). Every stage of a read, such as filter, sort and limit, works between the source and the run.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let deleted = db
    .fluent()
    .pipeline()
    .delete()
    .from("books")
    .filter(|q| q.field("stock").eq(0))
    .execute()
    .await?;
println!("Deleted {} documents", deleted.documents_modified);
Ok(())
}

A write returns a count, documents_modified, and not the documents. It has no explain, query or obj, and a write pipeline that calls them does not compile.

Full example available here.

When to use a pipeline write

Core has no write by query. To change the documents a query matches, the program reads them, computes the new values and writes each one back, with update or a batch writer. A pipeline write selects and writes in one request, and Firestore runs both on the server:

• no document is sent to the client, and the response is the count documents_modified;
• a value can be an expression over the fields of the document it is set on, so the program does not read a field to compute from it;
• insert and upsert take their documents from another collection, or from the output of a join, aggregation or search;
• .atomic() makes the write apply in full or not at all, see Atomic writes;
• many literal rows go in one request, each named by a field with .document_id(..).

use firestore::*;
use serde::Serialize;
#[derive(Debug, Clone, Serialize)]
struct Item { sku: String, price: f64, stock: i64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Raises the price of every item in stock by a tenth, on the server
let repriced = db
    .fluent()
    .pipeline()
    .update(|e| [e.field(path!(Item::price)).multiply(1.1).alias(path!(Item::price))])
    .in_col("items")
    .filter(|q| q.field(path!(Item::stock)).greater_than(0))
    .execute()
    .await?;
let _ = repriced;
Ok(())
}

A write to one document whose ID the program knows stays Core: it works on a Standard edition database too, and it can be buffered in a transaction. What a pipeline write costs:

• it needs an Enterprise edition database, and fails on Standard with FAILED_PRECONDITION, see Standard and Enterprise editions;
• it returns a count and not the documents;
• an atomic write has size limits, see Write limits;
• the library never retries it, see Retries and listeners;
• a transaction takes pipeline writes or buffered writes, not both, see Writes in a transaction.

The four writes

• delete() deletes each selected document;
• update(|e| [..]) sets the fields it returns on each selected document, and leaves its other fields. Each item needs an alias unless it is a field. Its source is named with .in_col(..), and .documents(..), .literals(..) and .database() are available too;
• insert() creates each document, and fails with ALREADY_EXISTS when one exists;
• upsert(|e| [..]) creates each document or replaces a stored one, with the fields it returns set on it. The stored document is replaced, not merged into.

insert and upsert write to the collection that .into("archive") names, relative to the database’s documents root, so it can be a path such as "restaurants/rest-01/reviews". .document_id(|e| e.field("key")) after .into(..) names each document by the ID the expression evaluates to; the field it reads stays in the stored document. Without it, a document keeps the ID of its __name__, and Firestore assigns one to a row that has none. Without .into(..), each document is written to its own __name__.

use firestore::*;
use serde::Serialize;
#[derive(Debug, Clone, Serialize)]
struct Item { sku: String, stock: i64 }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let inserted = db
    .fluent()
    .pipeline()
    .insert()
    .into("items")
    .document_id(|e| e.field(path!(Item::sku)))
    .literals([
        Item { sku: "a-1".to_string(), stock: 3 },
        Item { sku: "a-2".to_string(), stock: 0 },
    ])
    .execute()
    .await?;

let updated = db
    .fluent()
    .pipeline()
    .update(|e| [e.constant("sold_out").alias("status")])
    .in_col("items")
    .filter(|q| q.field(path!(Item::stock)).eq(0))
    .execute()
    .await?;
let _ = (inserted, updated);
Ok(())
}

A literal row that names its document by __name__ holds the full document path in a FirestoreReference, such as projects/p/databases/d/documents/items/a-1; a relative path fails with INVALID_ARGUMENT.

Atomic writes

.execute() writes each document on its own. A failure stops the run and can leave part of the documents written. An update of a document that does not exist writes nothing and counts 0.

.atomic() runs the write in a transaction of its own that commits as soon as the write is done, so the write applies in full or not at all. An update of a missing document then fails with NOT_FOUND:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let archived = db
    .fluent()
    .pipeline()
    .insert()
    .into("archive")
    .document_id(|e| e.field("key"))
    .from("books")
    .filter(|q| q.field("stock").eq(0))
    .atomic()
    .execute()
    .await?;
let _ = archived;
Ok(())
}

Firestore requires an atomic write, or a transaction, for these:

• every upsert;
• an insert from any source but literals;
• a write after aggregate, distinct, unnest, find_nearest, union, a search or a subquery.

The library does not set .atomic() itself and does not check these rules: Firestore rejects a write that breaks one with INVALID_ARGUMENT.

Write limits

Firestore rejects an atomic write that is too big with InvalidArgument, “Transaction too big. Decrease transaction size.” The limit is not one number of documents: it depends on the statement and on the size of the documents. For SQL statements, which run as atomic pipeline writes, on documents of two small fields (an integer and a short string), the limits are about:

The DELETE limit is about a tenth of the UPDATE limit. Larger documents lower both. A large write commits at roughly 1,000 to 3,000 documents per second.

A large write can also fail with ResourceExhausted (“This database has exceeded their maximum bandwidth for writes”) or with Aborted. Wait, and send the write again in smaller parts.

Split a large write by a filter on a range, such as the key or a timestamp, and send each part after the one before.

A TTL (time-to-live) policy suits retention better than a scheduled delete: Firestore deletes the documents itself once the timestamp in the policy’s field has passed, see TTL and the Enterprise edition. The bulk delete of the admin feature empties a whole collection.

Writes in a transaction

execute_in_transaction(&mut transaction) runs the write in an open transaction, whether it comes from db.begin_transaction() or from run_transaction. Firestore holds the write until the transaction commits and drops it when the transaction rolls back. .atomic() has no counterpart there, since an atomic write commits on its own, and .atomic().execute_in_transaction(..) does not compile.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.run_transaction(|db, transaction| {
    Box::pin(async move {
        db.fluent()
            .pipeline()
            .update(|e| [e.constant("shipped").alias("status")])
            .in_col("orders")
            .filter(|q| q.field("status").eq("packed"))
            .execute_in_transaction(transaction)
            .await?;
        Ok(())
    })
})
.await?;
Ok(())
}

execute_in_transaction differs from Core’s add_to_transaction in when the write happens. add_to_transaction buffers a write in the library and sends it with the commit. A pipeline write is sent when execute_in_transaction runs, with the transaction’s ID, because the pipeline has to run to find the documents it writes. Its count comes back at once, and the documents change at the commit.

Firestore sets these rules for a transaction that holds a pipeline write:

• every read comes before the first pipeline write; Firestore fails a read after it with FAILED_PRECONDITION;
• a transaction takes pipeline writes or buffered writes, not both: Firestore rejects a commit that mixes them;
• a write cannot run at a read time, and execute() or .atomic().execute() on a db that is bound to a transaction or a read time fails with InvalidParametersError; use execute_in_transaction inside the transaction.

Retries and listeners

The library sends a pipeline write once and never retries it: a run without a transaction can have written part of its documents before it failed, and an insert sent again fails with ALREADY_EXISTS. A run_transaction retries the whole transaction on ABORTED, as it does for any other transaction, and a retried attempt starts again from the callback. A failed .atomic() write reports a retry as a failed commit does: only ABORTED, which means nothing was applied, is retry_possible.

The changes of a pipeline write reach listeners. A Core listener on the collection receives a change for each inserted, updated or upserted document, and a delete for each deleted one. See Listening to document changes.

SQL

The library translates a SQL statement into a pipeline and runs it. It is meant for SQL text that comes from outside the program: command line tools, admin scripts, reports defined in configuration, LLM agents, and queries ported from a SQL database. Reads return the documents of a pipeline, and INSERT, UPDATE and DELETE run as pipeline writes. With the admin feature it also runs index statements: CREATE INDEX, DROP INDEX, ALTER COLLECTION and SHOW INDEXES. The first three run only when the options say .allow_index_statements().

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Book { title: String, rating: Option<f64> }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let books: Vec<Book> = db
    .fluent()
    .sql("SELECT title, rating FROM books WHERE genre = @genre ORDER BY rating DESC LIMIT 10")
    .param("genre", "Dystopian")
    .obj()
    .query()
    .await?;
let _ = books;
Ok(())
}

Full example available here.

SQL needs a database of the Firestore Enterprise edition, as pipelines do: a statement sent to a Standard edition database fails with FAILED_PRECONDITION, unless a cache session answers it. The index statements and SHOW COLLECTIONS use the Admin and Core APIs, and run on both editions. A statement means what the same pipeline means in the fluent API: Firestore’s semantics, listed in Semantics, where a few results differ from those of a SQL database.

A read through a cache session is answered from the cache when the pipeline it translates to only filters, sorts and pages cached documents, see Caching pipelines for the cached shapes.

When to use SQL

Application code keeps the fluent API, whose field paths are checked when the program compiles. SQL is for text that the program does not write: an admin command line, statements kept in configuration, statements an LLM agent writes, and queries ported from a SQL database. See Choosing Core queries, pipelines or SQL. With the codegen feature, Fluent code for a statement returns the fluent API code that a SELECT translates to, to move a ported statement into application code.

Setup

SQL is behind the sql cargo feature, off by default:

[dependencies]
firestore = { version = "...", features = ["sql"] }

db.fluent().sql(text) starts a statement. The rest of the builder is:

• .param(name, value) binds @name, and .params(values) binds the ? parameters in order;
• .options(FirestoreSqlOptions) limits what the statement may do, see Read-only options;
• .obj::<T>() deserialises the rows into T, as in a pipeline;
• query() returns every row, and stream_query(), stream_query_with_errors() and stream_query_with_metadata() stream them;
• run() runs any statement, a SELECT, a write, SHOW or an index statement, and returns a FirestoreSqlOutcome with one variant for each kind of statement, see Index statements;
• .wait_until_ready(timeout) and .wait_until_ready_with_options(..), with the admin feature, make run() wait for the operation of a CREATE INDEX or an ALTER COLLECTION;
• execute() and execute_in_transaction(&mut transaction) run an INSERT, UPDATE or DELETE, see Writes;
• explain() and explain_with_options(..) return the plan of a SELECT in the metadata of the last item of stream_query_with_metadata(), and so does an EXPLAIN [ANALYZE] prefix in the text, see Explaining a pipeline;
• .request_tags(..) and .request_options(..) attach request options to one statement, as they do for a pipeline.

A row holds the columns the statement selects, so it has no document name unless the statement selects __name__. An _firestore_id field of the result type must be an Option, as described in Pipelines. A field that a document lacks is absent from its row, and the result type needs an Option for it too.

A statement that is a SELECT fails from execute(), and a write fails from query(), with InvalidParametersError and before any request. So does a SHOW or an index statement from either of them: they run with run().

FirestoreSqlStatement::translate(&db, text, &parameters, &options) translates a statement without running it. The Query variant holds a FirestorePipeline, which from_pipeline(..) runs:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let statement = FirestoreSqlStatement::translate(
    &db,
    "SELECT title FROM books WHERE stock > @min",
    &FirestoreSqlParameters::new().param("min", 0),
    &FirestoreSqlOptions::new(),
)?;

if let FirestoreSqlStatement::Query(pipeline) = statement {
    let rows = db.fluent().pipeline().select().from_pipeline(pipeline).query().await?;
    let _ = rows;
}
Ok(())
}

Dialect and parameters

The dialect is BigQuery’s (GoogleSQL), and there is no switch. It reads a collection ID with hyphens or slashes between backticks, "text" as a string literal, @name as a parameter and UNNEST(..) WITH OFFSET. A statement holds exactly one SQL statement: text that parses to none or to several is refused.

Parameters are the way to put a value into a statement:

• @name is bound with .param("name", value);
• ? is bound with .params([..]), in the order the ?s appear;
• a statement uses one form, and mixing them is refused, and so is $1 or :name;
• the value is any Serialize type, bound as a typed constant of the pipeline, as a value in the fluent API is. A string parameter stays a string, and a number stays a number;
• binding is strict: a parameter that the text uses and no value is bound to is an error, and so is a bound value that the text does not use;
• a backticked `@name` is a field name.

A parameter is a value. It can stand where a value, a LIMIT or OFFSET count, or a search query can stand, and it cannot name a collection, a field or a source: collection_group(..), documents(..) and subcollection(..) take string literals.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let rows = db
    .fluent()
    .sql("SELECT title FROM books WHERE genre = ? AND stock > ? LIMIT ?")
    .params([serde_json::json!("Dystopian"), serde_json::json!(0), serde_json::json!(5)])
    .query()
    .await?;
let _ = rows;
Ok(())
}

Injection

A parameter value never enters the SQL text, so it cannot change what the statement does. This holds for every value that is bound.

Be aware of what a parameter does not cover:

• Collection and field names must never come from input. Building them into the text with format! gives whoever supplies the name control of the statement. Map the input to a name from a fixed list, and refuse anything else. For text that arrives from outside the program, the collection allowlist of the read-only options is the second line of defence;
• % and _ in a LIKE parameter are still wildcards. title LIKE @pattern with the value % matches every title. A backslash before % makes it literal. To match user text as it is, use starts_with(title, @text), ends_with(..) or string_contains(..), which have no wildcards;
• A text search parameter keeps the search operators. The value of document_matches(@query) goes to the search as written, so -bacon excludes a term and a quoted phrase matches as a phrase. See Pipeline search for the grammar.

use firestore::*;
async fn example(db: FirestoreDb, requested: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let column = match requested {
    "title" => "title",
    "rating" => "rating",
    "pages" => "pages",
    _ => return Err(format!("{requested} is not a column to sort by").into()),
};

let rows = db
    .fluent()
    .sql(format!("SELECT title FROM books ORDER BY {column}"))
    .query()
    .await?;
let _ = rows;
Ok(())
}

Read-only options

FirestoreSqlOptions limits a statement before it runs. A statement that breaks a limit fails at translation with InvalidParametersError, and no request is sent. FirestoreSqlOptions::new() allows every statement the translator supports, except joins and subqueries and the statements that change an index or a field, and FirestoreSqlOptions::read_only() starts the set for text from outside the program:

• read_only() refuses INSERT, UPDATE and DELETE, the database() source, and CREATE, DROP and ALTER COLLECTION. It allows SELECT, SHOW and EXPLAIN [ANALYZE] SELECT, which only read;
• .allow_index_statements() allows CREATE INDEX, DROP INDEX and ALTER COLLECTION, which new() refuses. read_only() still refuses them after it: read-only wins;
• .refuse_collection_groups() refuses collection_group(..), which reads every collection with that ID under any document;
• .with_collections([BOOKS, REVIEWS]) allows only these FirestoreCollectionIds, wherever the statement names one: a collection, each collection of a path, a collection group and the paths of documents(..). It refuses database() too, and SHOW COLLECTIONS and SHOW INDEXES without FROM, which list names outside the set;
• .with_max_rows(1_000) returns at most that many rows: the LIMIT is lowered to it, or one is added. The cap counts the rows returned, not the rows read: an OFFSET still reads the rows it skips, and ARRAY_AGG(..) or ARRAY(SELECT ..) puts a whole collection into one row;
• .allow_subqueries() allows joins and subqueries, see Joins and subqueries.

The translator also refuses a statement longer than 16 KiB, the parser one nested deeper than 16 levels, and the translator one nested deeper than 64. These limits are fixed.

use firestore::*;
async fn example(db: FirestoreDb, text: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
const BOOKS: FirestoreCollectionId = FirestoreCollectionId::from_static("books");
const REVIEWS: FirestoreCollectionId = FirestoreCollectionId::from_static("reviews");

let rows = db
    .fluent()
    .sql(text)
    .options(
        FirestoreSqlOptions::read_only()
            .refuse_collection_groups()
            .with_collections([BOOKS, REVIEWS])
            .with_max_rows(1_000),
    )
    .query()
    .await?;
let _ = rows;
Ok(())
}

The options guard against what a statement names. What it can reach stays bounded by the IAM roles of the credentials the client runs with: server client libraries are not subject to Firestore Security Rules. A client that runs text from outside the program needs credentials whose role can read and cannot write.

Firestore has no setting that rejects a table scan, and the options do not check for one. A statement that filters on an unindexed field scans the collection, as a pipeline does, see Known table scans.

Index statements

CREATE INDEX, DROP INDEX, ALTER COLLECTION, SHOW INDEXES and SHOW COLLECTIONS run with run(), and the first three need .allow_index_statements() on the options. Index statements lists their forms, outcomes, waits and refusals.

Reads, joins and subqueries

Reads, joins and subqueries lists the sources a FROM clause names, reads by ID, the supported and refused constructs, search, and joins and subqueries with their cost.

Semantics

A SQL statement keeps Firestore’s semantics, because it runs as a pipeline. Where that differs from what a SQL database does, and from a Core query, is listed in one table for pipelines and SQL: Query and write semantics. The main cases:

• v IS NULL matches null only, and a document that lacks v does not match;
• v != 1 matches null and absent too;
• a selected field that a document lacks is missing from its row, where a SQL database holds NULL;
• a comparison matches values of its own type, and a misspelled field is an absent field, with no error.

Writes

INSERT, UPDATE and DELETE run with execute(), which returns the count of documents written in documents_modified, or with execute_in_transaction(&mut transaction) in a transaction, which behaves as in Pipeline writes.

Every SQL write is atomic: it applies in full or not at all, as a SQL statement does. It is sent once and never retried, and a failed write has retry_possible set only for ABORTED, which means nothing was applied. To have a write retried on ABORTED, run it inside run_transaction, see Concurrent writes. A write has no explain, and .explain() on one fails with InvalidParametersError.

The rules of each statement, ON CONFLICT in detail, concurrent writes and the size limit of a write are in Writes and ON CONFLICT.

Errors

A text that does not parse, a construct that is refused, a parameter bound on one side only, and a statement that the options refuse all fail with InvalidParametersError, before any request. Its field names the construct: the keyword or function, the @name of a parameter, or sql for text that does not parse. error says why, and location holds the line and column where the construct starts, counting from 1, when the position is known.

use firestore::*;
use firestore::errors::FirestoreError;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let result = db
    .fluent()
    .sql("SELECT * FROM books TABLESAMPLE SYSTEM (10 PERCENT)")
    .query()
    .await;

if let Err(FirestoreError::InvalidParametersError(error)) = result {
    println!("{} at {:?}: {}", error.public.field, error.public.location, error.public.error);
}
Ok(())
}

Errors from Firestore itself arrive as the other FirestoreError variants, as they do for any pipeline. An unknown function, a type mismatch and a write that is too big are among them.

Reads, joins and subqueries

This page lists what a SQL SELECT can read and how, from the FROM clause to joins and their cost. Setup, parameters and the read-only options are in SQL, and the results that differ from a SQL database in Query and write semantics.

Sources

The FROM clause names where the documents come from:

A table alias is written FROM books AS b. A name that is defined by WITH matches without regard to case, as a table alias does, and hides a collection of the same name.

Reads by ID

A WHERE on __name__, __name__ = 'book-01', __name__ IN ('book-01', 'book-02') or __name__ IN UNNEST(@ids), reads each document by its name: the plan is a lookup of the named documents. A bare ID is a document of the FROM collection, and a path such as books/book-01 is accepted too.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let rows = db
    .fluent()
    .sql("SELECT title FROM books WHERE __name__ IN UNNEST(@ids)")
    .param("ids", ["book-01", "book-02"])
    .query()
    .await?;
let _ = rows;
Ok(())
}

A read by ID costs 1 read unit for each document of up to 4 KiB, as a Core get does, see Query plans and cost.

A name compared with a column, such as the __name__ of a joined table, is described in Join cost.

Supported constructs

SQL names that differ from the pipeline function are renamed: LOWER to to_lower, UPPER to to_upper, POWER to pow, IFNULL to if_null, REGEXP_CONTAINS to regex_contains, AVG to average, MIN to minimum, MAX to maximum, COUNTIF to count_if. SUBSTR, SUBSTRING and STRPOS count from 1 as in SQL, and ILIKE compares both sides in lower case. A SUBSTR position of 0 or below passes to Firestore as written: 0 is the whole string, and a negative position counts from the end with -1 the last character, as in SQL. ORDINAL(0) reads the first element, and a negative ORDINAL or OFFSET counts from the end, where SQL fails or gives NULL. NULLS FIRST and NULLS LAST add a leading sort key that tests for an absent or null value, since Firestore sorts both first in an ascending order.

Search

The search is part of the source, so the translator lifts it out of the WHERE: one document_matches('query') or one geo_distance(..) <= meters that is a top-level AND condition, on a collection or collection group source and before any join. score() AS relevance selects the relevance, and ORDER BY score() DESC or ORDER BY geo_distance(..) as the only sort key becomes the search’s own sort. A search needs a search index, see Search indexes.

Sampling

TABLESAMPLE BERNOULLI (p PERCENT) keeps each document with probability p / 100, independently of the others, so the count varies from run to run, and the input order is kept. Bernoulli sampling decides for each item separately, as a coin toss does. It becomes the sample stage, see Sampling in the pipeline chapter.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let about_a_tenth = db
    .fluent()
    .sql("SELECT title FROM books AS b TABLESAMPLE BERNOULLI (10 PERCENT) WHERE b.rating > 4")
    .query()
    .await?;
let _ = about_a_tenth;
Ok(())
}

As in SQL, the sample is drawn from the whole table before WHERE, so the WHERE filters the documents that were drawn: in the example, each document that matches is in the result with probability 0.1. The clause follows the table and its alias. p is a literal or a parameter, @name or ?. Firestore refuses a value outside 0 to 100.

TABLESAMPLE is refused with InvalidParametersError in these places:

• a derived table, a joined table and a subquery, and a derived table or a CTE that holds a TABLESAMPLE and is read by a join or a subquery. Firestore runs a join and a subquery once for each row of the outer query, which would draw the sample again every time;
• UPDATE and DELETE;
• a statement with a search, document_matches(..) or geo_distance(..), which reads the collection directly and cannot follow a sample.

A CTE that holds a TABLESAMPLE draws a new sample at each reference, since a CTE referenced twice runs twice.

These forms are refused too, since no sampling of the stage matches them:

• SYSTEM and BLOCK, since Firestore has no block sampling;
• REPEATABLE and SEED, since the stage takes no seed and every run draws a new sample;
• BUCKET and the SAMPLE keyword;
• a sample with ROWS, or with no unit or method.

RESERVOIR (n ROWS) does not parse. A fixed number of documents is .sample(FirestoreSampleSize::Documents(n)) in the pipeline API.

Refused constructs

A construct with no pipeline form fails with InvalidParametersError that names it, before any request. Anything in a parsed statement that the translator does not map is refused and never dropped, so a statement does not run with a clause missing.

Joins and subqueries

Joins and subqueries are in Preview in Firestore, see Pipeline aggregations and joins. Firestore plans a join as a nested loop: for each row of the left side it runs the right side as a subquery. A subquery in an expression, [NOT] EXISTS, [NOT] IN (SELECT ..), (SELECT ..) or ARRAY(SELECT ..), runs once per row of the query it is in too, whether or not it reads a column of that row, so the rows scanned multiply. Rows scanned and read units on an Enterprise edition database, for an outer collection of 11 documents and an inner one of 300, as Query Explain with analyze reports them:

The plan of each subquery statement is an Apply of the outer scan to the subquery, so even an uncorrelated COUNT(*) runs once for each outer row.

The cost depends on the index. Without one on the join key, each run reads the whole joined collection. A join or subquery is therefore refused unless the options say .allow_subqueries(), and the error says it runs one subquery per row, see Read-only options. These need no opt-in:

• a join or subquery into subcollection(..), which reads only the children of each parent;
• a derived table or a CTE in the FROM of a query, which is the query’s own source and does not multiply with the rows; a JOIN into one is still a join and needs the opt-in;
• UNNEST of an array of the row.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let rows = db
    .fluent()
    .sql(
        "SELECT r.name, v.rating, v.reviewer \
         FROM restaurants AS r LEFT JOIN collection_group('reviews') AS v ON v.restaurant = r.__name__ \
         WHERE r.cuisine = @cuisine",
    )
    .param("cuisine", "breakfast")
    .options(FirestoreSqlOptions::new().allow_subqueries())
    .query()
    .await?;
let _ = rows;
Ok(())
}

Rules for the joined names:

• a qualified name, v.rating, is a column of the table or alias v, searched from the nearest query outwards, and a table or alias matches without regard to case; an alias used by two tables is an error;
• a joined table lives in a field named after its alias, and a document can hold a field of any name, so the translation cannot tell whether the alias is the name of a field of the left side. It refuses every way to read such a field instead. In FROM orders AS o JOIN customers AS customer, o.customer is refused, and so is a bare customer in any place but customer IS NULL, since the field customer of the orders is replaced by the joined table. Give the table another alias. A bare alias of an UNNEST names its element, and is allowed;
• in a query with a join or in a subquery, an unqualified name in an ON clause is an error, since either side could hold it, and so is a dotted name that starts with no table;
• SELECT * and b.* are refused in a query with a join, for the base table as well as for a joined one. The joined tables would come with the base document’s fields, and replace any of them that has the name of an alias. Name the columns;
• the anti-join, the left rows that have no match, is WHERE v IS NULL on the alias. WHERE v.rating IS NULL matches nothing, since the field of an unmatched row is absent and not null;
• a correlated subquery, one that reads a column of the row it runs for, cannot sit in a query that groups, or in a clause with no stage before it to carry the column, such as the SET of an UPDATE;
• EXISTS (SELECT 1 ..) ignores the select list, so an aggregate, GROUP BY or HAVING inside EXISTS is refused: it would change which rows exist, and the select list that holds it is not evaluated.

A CTE, a named subquery defined with WITH, runs again for each place that uses it: a CTE referenced twice runs twice. Results are not kept or reused between references. A statement that repeats CTE bodies more than 1,024 times in all is refused.

Join cost

Rows scanned and read units for some joins on an Enterprise edition database, as Query Explain with analyze reports them, over small collections: 9 restaurants with 6 reviews, 11 books and 1,500 items.

The unindexed join scans the joined collection once for each left row: parents × child collection. The indexed join reads an index range for each left row, and a join into subcollection(..) reads each parent’s children only.

Rules that follow from it:

• index the join key, and run .explain() before the join reaches production data. A plan with a TableScan under the join is the unindexed case, see Query plans and cost;
• a join that compares the joined table’s __name__ with a column of the left row becomes a lookup by name for each left row. A plain = of __name__ with a variable in a per-row subquery scans the collection instead: over a collection of 60 documents, run for 3 rows, it scans up to 180 rows, where the lookup by name reads 2;
• a join over a large collection that is run for analysis belongs in BigQuery. Firestore joins fit a left side that a filter has narrowed to a few documents.

Writes and ON CONFLICT

INSERT, UPDATE and DELETE run as pipeline writes, with execute() or execute_in_transaction(..). Writes in the SQL overview lists the statements and what each runs as. This page holds the rules of each statement.

Rules per statement

• INSERT .. VALUES needs a column list. Each value is a literal or a bound parameter, and each column a top-level field name. An INSERT fails with ALREADY_EXISTS when a document of that name exists;
• a __name__ column gives the document’s name. In VALUES a bare ID resolves against the target collection and a path outside it is refused; the rows of a SELECT keep the name of the document they were read from. A row without __name__ gets an ID from Firestore. A column list over a SELECT renames its columns by position, and is refused over *;
• UPDATE and DELETE require a WHERE, as in BigQuery, and WHERE TRUE selects every document. The target is one collection. UPDATE .. SET __name__ is refused;
• the same ID twice in one statement fails with INVALID_ARGUMENT, “Multiple mutations per key are not allowed”, and nothing is written.

ON CONFLICT

• ON CONFLICT takes only the target (__name__), since Firestore has no other unique constraint, and DO NOTHING also accepts no target;
• in DO UPDATE, excluded.col is a field of the new row, and a bare column is a field of the stored document. A WHERE on DO UPDATE leaves the stored document unchanged, and does not count it, when it is false for it;
• the lookup of the stored document by its name is bounded by the rows written. In ON CONFLICT, rows of a SELECT from another collection are matched by their ID, which scans the target collection once per row, and so need .allow_subqueries();
• the __name__ of such a row must be the name of a document that the translation can see: SELECT __name__, .., or * of a collection. A string or another field renamed to __name__, * of a derived table, and * EXCEPT (__name__) are refused before any request, since a row without a document reference cannot be matched;
• ON CONFLICT keeps the stored document of each row in a field named _conflict_stored. A column of that name is refused, and a field of that name in the rows of a SELECT * is dropped from them.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let written = db
    .fluent()
    .sql(
        "INSERT INTO items (__name__, stock) VALUES (@id, @stock) \
         ON CONFLICT (__name__) DO UPDATE SET stock = stock + excluded.stock",
    )
    .param("id", "a-1")
    .param("stock", 5)
    .execute()
    .await?;
println!("Wrote {} documents", written.documents_modified);
Ok(())
}

Concurrent writes

Concurrent writes to one document conflict: some commit, and the others fail with ABORTED, “Optimistic transaction was aborted due to a conflict”, with nothing of them applied. To have such writes retried, run them with execute_in_transaction(..) inside run_transaction, which runs the whole transaction again on ABORTED, see Writes in a transaction. execute() on its own sends a write once.

Write limits

Firestore refuses an atomic write that is too big. Write limits lists the limits.

Index statements

With the admin cargo feature next to sql, SQL text can list, create and drop indexes, and change the TTL and the single-field indexes of a field. CREATE INDEX, DROP INDEX and ALTER COLLECTION run only when the caller opts in with .allow_index_statements() on the options, so they suit command line tools and admin scripts that the program’s own operators write or run. SHOW INDEXES, SHOW COLLECTIONS and EXPLAIN only read and need no opt-in. Text from users or LLM agents runs with read_only(), which refuses the three statements. The same operations have a fluent API, see One index at a time.

[dependencies]
firestore = { version = "...", features = ["sql", "admin"] }

SHOW COLLECTIONS and EXPLAIN need sql alone. An index statement in a build without admin fails with InvalidParametersError, before any request; its field is the statement, such as CREATE INDEX.

Running a statement

run() of the SQL builder runs any statement and returns a FirestoreSqlOutcome, one variant for each kind of statement:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let outcome = db.fluent().sql("SHOW INDEXES FROM orders").run().await?;

if let FirestoreSqlOutcome::Indexes(listing) = outcome {
    for listed in listing.indexes {
        println!("{} {}", listed.id, listed);
    }
}
Ok(())
}

query(), its streaming variants and obj() run SELECT and EXPLAIN [ANALYZE] SELECT, and fail with InvalidParametersError, before any request, for every other new statement, as they do for a write. execute() fails the same way.

Names, scope and options

• a collection or a collection group is named by its ID. An ID with hyphens or dots goes between backticks, as in a FROM clause, and so does an index ID, which can hold -, such as `CICAgPj-z4oK`. A path such as users/alice/orders is refused: an index belongs to every collection with the ID, wherever it is;
• a column of an index is a field path, written as in a WHERE;
• an index has a scope. ON orders is an index for queries on one collection, which a query reaches with .from("orders").parent(..). ON collection_group('orders') is an index for queries across every collection with that ID, which a query reaches with .all_descendants(). In both the index belongs to the collection group orders, and SHOW INDEXES FROM and DROP INDEX .. ON name that group;
• keywords and option values are matched without regard to case;
• an option value is a literal. A @name parameter is refused there, and in every name.

Listing

SHOW INDEXES;
SHOW INDEXES FROM orders;
SHOW COLLECTIONS;
SHOW COLLECTIONS IN `users/alice`;

SHOW INDEXES returns a FirestoreIndexListing for one collection group, or for every group of the database except the ones Firestore reserves, such as __default__:

• indexes holds the composite indexes, each with its collection_group, id, state and the index itself: query_scope, fields, unique, density and the search options. A composite index is an index over several fields, which a query that filters or sorts on them needs. On a Standard edition database fields ends with the __name__ field that Firestore appends;
• fields holds the fields that carry a single-field override or a TTL policy, see ALTER COLLECTION;
• unrecognised holds the items the library cannot represent, each with its name and reason.

id is a FirestoreIndexId, the type that force_index(..) of a pipeline takes.

SHOW COLLECTIONS returns the collection IDs under the documents root, or under the document that IN names, in the order Firestore lists them. A document that does not exist has none, and the list is empty.

CREATE INDEX

CREATE INDEX ON orders (customer_id, placed_at DESC);
CREATE INDEX ON collection_group('orders') (tags CONTAINS, placed_at DESC);
CREATE INDEX ON `order-lines` (`unit-price` DESC, sku);
CREATE UNIQUE INDEX ON users (email) OPTIONS (density = 'SPARSE_ANY');
CREATE INDEX ON events (kind, at DESC) OPTIONS (density = 'DENSE');
CREATE SEARCH INDEX ON restaurants (menu, description, location GEO)
    OPTIONS (text_language = 'en', text_language_override_field_path = 'lang');
CREATE VECTOR INDEX ON books (genre, embedding) OPTIONS (dimension = 768);
CREATE INDEX IF NOT EXISTS ON orders (customer_id, placed_at DESC);

The statement starts the build and returns the index as Firestore lists it: in state CREATING, or READY after a wait. With IF NOT EXISTS, an index that already exists is returned instead, as the list below says. Firestore assigns the ID of an index, so an index cannot be named: CREATE INDEX by_customer ON .. is refused.

The kind of index sets what a column can be:

• a column takes a direction or a mode, never both;
• the options of a search index are those of Search indexes. The density of an index says which documents it holds an entry for, see Density, and a unique index refuses a second document with the same values;
• IF NOT EXISTS makes an index that already exists the outcome instead of an error: the statement returns the listed index that the definition matches. Without a wait it comes back in the state Firestore lists; with a wait, an index another caller is still building is returned once it is READY, and one that ends in NEEDS_REPAIR fails the statement. CREATE of an index that exists, without IF NOT EXISTS, fails with DataConflictError. Firestore keeps answering ALREADY_EXISTS for a while after an index is deleted, when the listing no longer shows it. IF NOT EXISTS returns an index only when the listing shows it, so the statement fails with DataConflictError in that window; run it again later.

A fluent .create() takes the same definition, see One index at a time.

DROP INDEX

DROP INDEX `CICAgPj-z4oK`;
DROP INDEX CICAgJim14AK ON orders;
DROP INDEX ON orders (customer_id, placed_at DESC);
DROP UNIQUE INDEX ON users (email) OPTIONS (density = 'SPARSE_ANY');
DROP INDEX IF EXISTS `CICAgPj-z4oK`;

A drop deletes one index and returns it as it was listed just before. Firestore deletes it at once, so there is nothing to wait for.

• By ID. The index is read first, and ON orders makes the drop fail with DataNotFoundError, deleting nothing, when the index belongs to another collection group. Firestore itself ignores the group of a request, and answers 200 for the second delete of an ID, so the check is made by the library. An ID that no index has fails with DataNotFoundError;
• By definition. DROP [UNIQUE | SEARCH | VECTOR] INDEX ON target (..) [OPTIONS (..)] takes the text that CREATE took, and deletes the one listed index that it matches, as .sync() matches a declaration. On an Enterprise edition database two indexes over the same fields differ by unique and density, and the options pick between them. No match fails with DataNotFoundError, and several matches fail with DataConflictError; nothing is deleted in either case;
• IF EXISTS makes an ID that no index has, or a definition that nothing matches, delete nothing and return None in IndexDropped. An index in another group than ON names, and a definition with several matches, stay errors;
• UNIQUE, SEARCH and VECTOR belong to the definition form. A drop takes one index.

ALTER COLLECTION

ALTER COLLECTION sessions ALTER COLUMN expires_at SET OPTIONS (ttl = TRUE, indexes = []);
ALTER COLLECTION sessions ALTER COLUMN expires_at SET OPTIONS (ttl = FALSE, indexes = NULL);
ALTER COLLECTION orders ALTER COLUMN tags SET OPTIONS (indexes = ['CONTAINS', 'COLLECTION_GROUP CONTAINS']);
ALTER COLLECTION orders ALTER address.city SET OPTIONS (indexes = NULL);
ALTER COLLECTION logs SET OPTIONS (indexes = []);

ALTER COLLECTION changes one field of a collection group, or every field that has no setting of its own. It sends one write, with the parts that the options name, and returns the field as Firestore reads it after the write.

• ttl = TRUE enables TTL, time to live: Firestore deletes a document once the timestamp in the field has passed. ttl = FALSE disables it. TTL applies to a named field, so ALTER COLLECTION c SET OPTIONS (ttl = ..) is refused, and the value is TRUE or FALSE;
• indexes sets the single-field override of the field. Firestore indexes every field of a Standard edition database on its own, and an override replaces those indexes, see Single-field overrides. indexes = [..] replaces the whole set with the entries listed, [] exempts the field from single-field indexing, and NULL removes the override, so that the field inherits the configuration of its group, or of the database;
• an entry of indexes is 'ASC', 'DESC' or 'CONTAINS' for queries on one collection, or the same word after COLLECTION_GROUP for queries across the collection group;
• ttl and indexes in one SET OPTIONS are one write;
• without ALTER COLUMN, indexes sets the configuration of every field that has no setting of its own.

Firestore refuses a write that names indexes on an Enterprise edition database, unless the list is empty or NULL, and then the write changes nothing; see Editions. A ttl in the same refused statement is not applied either.

Writes to one field are not sequenced: Firestore accepts a second write while the first is still running. The statement reads the field after its own write, so a second write that overlaps it can show in the result.

Waiting for an operation

CREATE .. INDEX and ALTER COLLECTION start an operation that Firestore runs in the background. .wait_until_ready(timeout) and .wait_until_ready_with_options(..) of the SQL builder make run() return after it has finished, and the outcome is the index or field read after that:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let outcome = db
    .fluent()
    .sql("CREATE INDEX ON orders (customer_id, placed_at DESC)")
    .options(FirestoreSqlOptions::new().allow_index_statements())
    .wait_until_ready(Duration::from_secs(900))
    .run()
    .await?;

if let FirestoreSqlOutcome::IndexCreated(created) = outcome {
    assert_eq!(created.state, FirestoreIndexState::Ready);
}
Ok(())
}

A build takes minutes even for an empty collection, see Waiting for changes to finish; set the timeout above that. With IF NOT EXISTS, an index that another caller is still building is polled until it is no longer CREATING, under the same timeout and with the same retries as the poll of an operation. A deadline that passes fails with SystemError, code OPERATION_WAIT_TIMEOUT, and a failed operation fails with the operation’s status, as in .sync(). An index that ends in NEEDS_REPAIR, the state of an index whose last build failed and left no operation running, fails the same way, with SystemError, code INDEX_NEEDS_REPAIR.

A wait on any other statement fails with InvalidParametersError, field wait, before any request.

Editions and the emulator

The library runs no edition rule of its own on these statements. Firestore answers a statement its edition does not take with INVALID_ARGUMENT, which arrives as DatabaseError with Firestore’s message.

The emulator serves no Firestore Admin API, whatever its edition, and answers each index call with UNIMPLEMENTED; it arrives as DatabaseError, with retry_possible unset. .plan() and .sync() fail the same way, see The Firestore emulator. SHOW COLLECTIONS works on the emulator.

Options

FirestoreSqlOptions apply to the new statements as to a SELECT, see Read-only options:

• FirestoreSqlOptions::new() refuses CREATE, DROP and ALTER COLLECTION. They run after .allow_index_statements(); without it they fail with InvalidParametersError, field allow_index_statements, before any request. SHOW INDEXES, SHOW COLLECTIONS and EXPLAIN [ANALYZE] SELECT only read, and run without it;
• read_only() refuses CREATE, DROP and ALTER COLLECTION even when .allow_index_statements() follows it: read-only wins. The error field is read_only. read_only() allows the statements that only read;
• .with_collections([..]) applies to every collection ID a statement names. It refuses SHOW INDEXES without FROM, and SHOW COLLECTIONS, which list names outside the set, and DROP INDEX id without ON, which reaches an index in any group. DROP INDEX id ON collection needs a collection in the set;
• .refuse_collection_groups() refuses collection_group('..') in an index definition;
• .with_max_rows(..) caps the rows of the SELECT in EXPLAIN ANALYZE; a SHOW returns every entry.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
const ORDERS: FirestoreCollectionId = FirestoreCollectionId::from_static("orders");

let outcome = db
    .fluent()
    .sql("DROP INDEX `CICAgPj-z4oK` ON orders")
    .options(
        FirestoreSqlOptions::new()
            .allow_index_statements()
            .with_collections([ORDERS]),
    )
    .run()
    .await?;
let _ = outcome;
Ok(())
}

Refused forms

A statement of these forms fails with InvalidParametersError, before any request. field names the construct, and location says where it starts:

-- an index cannot be named
CREATE INDEX by_customer ON orders (customer_id, placed_at DESC);
-- a path is not a collection group ID
CREATE INDEX ON `users/alice/orders` (placed_at DESC, total);
-- a column is a field path, with a direction or a mode, not both
CREATE INDEX ON orders (tags CONTAINS DESC);
CREATE INDEX ON orders (lower(name));
-- CREATE VECTOR INDEX takes its dimension in OPTIONS
CREATE VECTOR INDEX ON books (genre, embedding);
-- an option the statement does not take
CREATE INDEX ON orders (customer_id, placed_at DESC) OPTIONS (dimension = 3);
-- an option is a literal
CREATE INDEX ON orders (customer_id, placed_at DESC) OPTIONS (density = @density);
-- an index has no WHERE
CREATE INDEX ON orders (customer_id, placed_at DESC) WHERE total > 0;
-- one index for each DROP
DROP INDEX CICAgJim14AK, CICAgOi36pgK;
-- ttl is TRUE or FALSE, and it applies to a named field
ALTER COLLECTION sessions ALTER COLUMN expires_at SET OPTIONS (ttl = 'yes');
ALTER COLLECTION logs SET OPTIONS (ttl = TRUE);
-- an entry is ASC, DESC or CONTAINS, optionally after COLLECTION_GROUP
ALTER COLLECTION orders ALTER COLUMN tags SET OPTIONS (indexes = ['ASCENDING']);
-- FROM names a collection group ID
SHOW INDEXES FROM `users/alice/orders`;
-- only SHOW INDEXES and SHOW COLLECTIONS
SHOW TABLES;

Explaining a SELECT

EXPLAIN SELECT .. returns the plan of a SELECT without reading any document, and EXPLAIN ANALYZE SELECT .. also returns the rows and the execution profile. The prefix is the SQL spelling of .explain() and .explain_with_options(..), see Explaining a pipeline:

EXPLAIN SELECT title FROM books WHERE genre = 'Dystopian';
EXPLAIN ANALYZE SELECT title FROM books WHERE genre = @genre ORDER BY rating DESC LIMIT 10;

The plan is Firestore’s own text, in the metadata of the last item of the rows. EXPLAIN takes a SELECT. It fails for a write, which Firestore rejects with Query Explain or, with ANALYZE, runs, and for an option other than ANALYZE. EXPLAIN in the text together with .explain() on the builder fails with InvalidParametersError, field explain: the text and the builder would both set the options. .explain() on SHOW or on an index statement fails the same way, since they have no plan, and so does .request_options(..), or .request_tags(..), on SHOW INDEXES and the index statements, field request_options: they go to the Admin API, which takes no request options.

Errors

Every error is an existing FirestoreError variant. A code is a constant of firestore::errors::codes:

Permissions

SHOW INDEXES and SHOW COLLECTIONS need roles/datastore.viewer. CREATE, DROP and ALTER COLLECTION need roles/datastore.indexAdmin. Grant the viewer role alone to credentials that run text from users or LLM agents, with read_only(), so that the role refuses a statement the options let through.

Full example available here.

1. [] and NULL are accepted and write nothing: an Enterprise edition database has no automatic single-field indexes to override. ↩

2. Like every SQL statement on a Standard edition database, FAILED_PRECONDITION. ↩

3. The Enterprise emulator returns the rows and no explain output. The Standard emulator fails every pipeline and SQL statement with INVALID_ARGUMENT. ↩

Fluent code for a statement

With the codegen cargo feature, FirestoreSqlStatement::fluent_rust(..) turns a SQL SELECT into the Rust code of the equivalent pipeline in the fluent API. It is meant for people who port a statement to application code: the snippet shows the builder calls, the row struct and the function arguments, and the person who pastes it keeps and edits the result. It sends nothing, so it needs no client and no credentials.

[dependencies]
firestore = { version = "...", features = ["codegen"] }
serde_json = "1"
serde_bytes = "0.11"

The snippet reads values that have no fixed type into serde_json::Value, and bytes through serde_bytes, so the project that pastes it needs serde_json, and serde_bytes when the statement reads or binds bytes.

Calling it

FirestoreSqlStatement::fluent_rust(sql, &parameters, &sample) -> FirestoreResult<String>

• sql is a SELECT;
• parameters binds every @name and ? of the text to a sample value, as .param(..) and .params(..) of the SQL builder do. A parameter without a value fails as it does when the statement runs;
• sample holds rows that the statement returned, as Vec<FirestoreDocument>. A sample is enough to type the fields of the row struct, and an empty one is accepted.

The options of a statement are not an argument: nothing runs, so there is nothing for them to guard, and a with_max_rows(..) used to fetch the sample would add a LIMIT that the snippet must not show.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let sql = "SELECT title, author, rating, publishedAt, tags, edition FROM books
WHERE genre = @genre AND rating >= @min_rating ORDER BY rating DESC LIMIT 20";
let parameters = FirestoreSqlParameters::new()
    .param("genre", "Dystopian")
    .param("min_rating", 4.0);

let statement =
    FirestoreSqlStatement::translate(&db, sql, &parameters, &FirestoreSqlOptions::read_only())?;
let FirestoreSqlStatement::Query(pipeline) = statement else {
    return Ok(());
};
let sample = db.fluent().pipeline().select().from_pipeline(pipeline).query().await?;

println!("{}", FirestoreSqlStatement::fluent_rust(sql, &parameters, &sample)?);
Ok(())
}

Without a database, &[] is the sample and every field is typed as unknown, see Row struct. For a sample where author is missing from one row, rating is an integer in some rows and a double in others, and edition is an integer in some rows and a string in others, the text it returns is:

// SELECT title, author, rating, publishedAt, tags, edition FROM books
// WHERE genre = @genre AND rating >= @min_rating ORDER BY rating DESC LIMIT 20
use firestore::*;
use serde::Deserialize;

#[derive(Debug, Clone, Deserialize)]
struct BooksRow {
    title: String,
    author: Option<String>,
    rating: f64,
    #[serde(rename = "publishedAt")]
    published_at: FirestoreTimestamp,
    tags: serde_json::Value,
    /// Observed as integer and string in the sample.
    edition: serde_json::Value,
}

async fn query_books(
    db: &FirestoreDb,
    genre: &str,
    min_rating: f64,
) -> FirestoreResult<Vec<BooksRow>> {
    db.fluent()
        .pipeline()
        .select()
        .from("books")
        .filter(|q| {
            q
                .for_all([
                    q.field("genre").equal(genre),
                    q.field("rating").greater_than_or_equal(min_rating),
                ])
        })
        .sort(|e| [e.field("rating").desc()])
        .fields(|e| [
            e.field("title"),
            e.field("author"),
            e.field("rating"),
            e.field("publishedAt"),
            e.field("tags"),
            e.field("edition"),
        ])
        .limit(20)
        .obj::<BooksRow>()
        .query()
        .await
}

The snippet follows the pipeline that the statement translates to, stage by stage, so its stages are in the order of the translation: sort before fields, as above. The text is formatted by the crate, not by rustfmt, so run cargo fmt after pasting it.¹

Row struct

The snippet’s struct holds the columns of the select list, in the order of the list. Without a select list, as in SELECT *, it holds the field names of the sample, in alphabetical order, since the fields of a document have no order. Each field is typed from the values of its column across the whole sample:

serde_json::Value cannot hold bytes, so a column that mixes bytes with another kind, or holds them inside a map or an array, fails to read.

A column that is always present in the sample can still be absent from a document that the statement returns later, and the type of a field is the type of the sample’s values. Check the types against the data before relying on them.

With an empty sample every field is an Option<serde_json::Value>, since nothing says which fields are always present. With an empty sample and no select list there is nothing to type: the function returns Vec<FirestoreDocument> and the snippet has no struct.

A statement that selects __name__, and a sample whose rows carry a document name, give the struct an id: Option<String> field that reads _firestore_id, as in Pipelines.

Naming and arguments

The struct and the function are named after the collection, BooksRow and query_books for books, and a field after its column in snake case, with a serde rename when the two differ. A name that Rust cannot use, such as a keyword or one that starts with a digit, is made valid, and a name that another one took gets a suffix, as db_2 does for a parameter named db.

The function takes db: &FirestoreDb, then one argument for each parameter, in the alphabetical order of the names, and param_1, param_2 for ? in the order of the text. An argument has the type of the value bound to it, as in the table of the row struct, borrowed unless it is Copy. A null, a map or an array is a &serde_json::Value.

What it shows

fluent_rust(..) covers:

• reads of one collection, a path to a nested collection, or a collection group, with WHERE, ORDER BY, LIMIT, OFFSET, TABLESAMPLE and a select list;
• aggregates, with and without GROUP BY, and DISTINCT;
• text and geo search, document_matches(..) and geo_distance(..), see Search. Vector distance is a function like any other and is shown as one.

A statement outside this list fails with InvalidParametersError, before anything is generated, and its message names the construct:

• UNION, UNNEST, subqueries, variables, and so a join, which the translator writes as a subquery and an UNNEST, see Reads, joins and subqueries;
• * EXCEPT, and a column written beside *;
• the sources documents(..), database() and subcollection(..);
• INSERT, UPDATE, DELETE, index statements, EXPLAIN and SHOW: a statement that is not a SELECT runs with the SQL builder, and has no fluent code to show;
• a parameter whose value the translation reads when it builds the pipeline, since the snippet has no value to put in its place. Such a parameter is LIMIT @n and OFFSET @n, the percentage of TABLESAMPLE, __name__ = @id and the other comparisons on __name__, the key of a subscript as in x[@key], and the arguments of search such as document_matches(@query). Write the value in the text instead. A parameter anywhere else becomes an argument of the function.

1. The crate prints the snippet with a library formatter, since rustfmt is a separate program that a library cannot count on. It breaks some lines differently from rustfmt, such as q.for_all above. ↩

Timestamps support

By default, date/time values serialise as a string to Firestore (while deserialisation works from Timestamps and Strings). To store them as native Firestore timestamps there are three options.

• Using std::time::SystemTime directly, with no attribute and no wrapping type, since it is recognised automatically:

use serde::{Deserialize, Serialize};
use std::time::SystemTime;
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    created_at: SystemTime,
    updated_at: Option<SystemTime>,
}

It works in the queries as well:

use firestore::*;
use std::time::SystemTime;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct MyTestStructure { created_at: SystemTime }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent().select().from(TEST_COLLECTION_NAME).filter(|q| { q.for_all([
q.field(path!(MyTestStructure::created_at))
    .less_than_or_equal(SystemTime::now())
]) }).obj::<MyTestStructure>().query().await?;
Ok(())
}

Note that SystemTime cannot carry the instants before the Unix epoch, since serde itself refuses them.

• Using the type FirestoreTimestamp, which needs no attributes:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    created_at: FirestoreTimestamp,
    updated_at: Option<FirestoreTimestamp>,
}

It can be created with FirestoreTimestamp::now(), parsed from a string, and converted from/to std::time::SystemTime:

use firestore::*;
use std::time::SystemTime;
fn example() -> Result<(), Box<dyn std::error::Error>> {
let now = FirestoreTimestamp::now();
let from_system_time: FirestoreTimestamp = SystemTime::now().try_into()?;
let back_to_system_time: SystemTime = now.into();
let _ = (from_system_time, back_to_system_time);
Ok(())
}

Use it in your queries as well, for example:

use firestore::*;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct MyTestStructure { created_at: FirestoreTimestamp }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent().select().from(TEST_COLLECTION_NAME).filter(|q| { q.for_all([
q.field(path!(MyTestStructure::created_at))
    .less_than_or_equal(FirestoreTimestamp::now())
]) }).obj::<MyTestStructure>().query().await?;
Ok(())
}

• Or, if you prefer to keep a plain instant in your model, use FirestoreInstant (an alias for jiff::Timestamp) with #[serde(with)] attributes:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    #[serde(with = "firestore::serialize_as_timestamp")]
    created_at: FirestoreInstant,

    #[serde(default)]
    #[serde(with = "firestore::serialize_as_optional_timestamp")]
    updated_at: Option<FirestoreInstant>,
}

Firestore stores the timestamps with microsecond precision and discards the nanoseconds on write. FirestoreTimestamp truncates to that precision in all of its constructors and conversions, so its values survive a round trip unchanged. A FirestoreInstant or a SystemTime carrying nanoseconds does not, since those are the standard types this library cannot change.

All of them change the representation only for Firestore serialisation. FirestoreTimestamp and FirestoreInstant still serialise as a string to JSON, so the same model can be reused for JSON and Firestore, while a plain SystemTime keeps the default serde representation.

Document metadata as struct fields

Firestore provides additional fields for each document:

• _firestore_id: the document ID, whether the client chose it or Firestore generated it;
• _firestore_full_id: the full document name, such as projects/my-project/databases/(default)/documents/my-collection/my-id;
• _firestore_created: the time at which the document was created;
• _firestore_updated: the time at which the document was last changed.

A stored document carries its name in Core and pipeline results. A Core aggregation result, and the result of a pipeline stage that builds new documents, such as .fields() or .aggregate(), has no document name, and so neither _firestore_id nor _firestore_full_id. A SQL row has no name unless the statement selects __name__. Make a field that reads either of them an Option, see Query and write semantics.

To be able to read them the library makes them available as system fields for the Serde deserializer with reserved names, so you can specify them in your structures as:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
    #[serde(alias = "_firestore_id")]
    id: Option<String>,
    #[serde(alias = "_firestore_created")]
    created_at: Option<FirestoreTimestamp>,
    #[serde(alias = "_firestore_updated")]
    updated_at: Option<FirestoreTimestamp>,
    some_string: String,
    one_more_string: String,
    some_num: u64,
}

Complete example available here.

Explicit null value serialisation

By default, every Option serialises as an absent field, which is convenient for many cases. Some cases need explicit nulls instead.

Null and absent are not the same to a query. A Core filter or order_by never returns a document that lacks the field, and is_null() matches only a stored null. A pipeline or SQL statement tells them apart too: not_equal and sort keep a document that lacks the field, and SQL IS NULL matches a stored null only. Query and write semantics lists every case.

To help with that there are additional attributes implemented for serde(with):

• For any type:

use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
#[serde(default)]
#[serde(with = "firestore::serialize_as_null")]
test_null: Option<String>,
}

• For Firestore timestamps attribute:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure {
#[serde(default)]
#[serde(with = "firestore::serialize_as_null_timestamp")]
test_null: Option<FirestoreInstant>,
}

Document and collection IDs

FirestoreDocumentId and FirestoreCollectionId are validated newtypes for IDs that arrive from outside your process - a request path segment, a JSON body, a runtime-chosen collection name. Construct one where the ID enters your system; a /, an empty string, or ./.. is rejected there instead of silently reaching Firestore or, in the case of a collection name, retargeting the operation at a different collection.

use firestore::*;
fn example() -> Result<(), Box<dyn std::error::Error>> {
let id = FirestoreDocumentId::new("user-42")?;
let collection = FirestoreCollectionId::new("users")?;
let _ = (id, collection);
Ok(())
}

A name known up front - the sort of thing you would today write as const NAME: &str = "..." - can be declared as a validated const or static with from_static. An invalid literal there is a compile error, not a runtime one:

use firestore::*;
const USERS: FirestoreCollectionId = FirestoreCollectionId::from_static("users");

Use from_static for a literal you control, checked at compile time with no Result to handle; use new for a value arriving at runtime, which returns a FirestoreResult.

Both implement AsRef<str>, so a reference drops straight into any call that already takes a document or collection ID or name, with no conversion:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
async fn example(
    db: FirestoreDb,
    collection: FirestoreCollectionId,
    id: FirestoreDocumentId,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .select()
    .by_id_in(&collection)
    .obj::<MyTestStructure>()
    .one(&id)
    .await?;
Ok(())
}

Store the validated type in your own structs instead of a bare String to carry the proof along with the value; it deserialises through the same validation, so a CreateSession built from an untrusted request body rejects a bad ID before it reaches Firestore:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct CreateSession {
    user_id: FirestoreDocumentId,
}

See examples/crud.rs and examples/nested_collections.rs for a document ID and a parent_path/at pair built from validated IDs, and examples/dynamic_doc_level_crud.rs / examples/caching_dynamic_collections.rs for a collection name chosen at runtime.

JSON conversion

With the json-conversion cargo feature, associated functions of FirestoreDb convert Firestore values and documents to JSON and back. It is meant for tools that move documents out of Firestore and into it again, such as an export and an import, and for tools that show documents to a person. It does not send anything.

Like FirestoreDb::serialize_to_doc, the functions take their context explicitly, so they need no client. The first argument is the documents root of the database, projects/{project}/databases/{database}/documents, which db.get_documents_path() returns.

[dependencies]
firestore = { version = "...", features = ["json-conversion"] }
serde_json = "1"

The functions write two encodings, chosen with FirestoreJsonEncoding:

• Tagged, for export and import. It reads back to the same Firestore value, without loss. A type that JSON lacks becomes a one-key object, a wrapper, whose key starts with $;
• Plain, for display. It is the most readable JSON for each type, and it has no way back: a timestamp, a reference and a string look alike.

A value is a gcloud_sdk::google::firestore::v1::Value, the type that FirestoreDocument::fields holds, so a field converts without a copy. FirestoreValue::value is one.

FirestoreJsonOptions holds the encoding of a document and whether it carries its metadata. Both are set through its builder, and the defaults are the tagged encoding and no metadata:

use firestore::*;
let options = FirestoreJsonOptions::new()
    .with_encoding(FirestoreJsonEncoding::Plain)
    .with_metadata(true);

The plain encoding of a value never fails; the tagged one refuses the values that exist only in pipelines, see below. The metadata option means nothing for a single value, so serialize_value_to_json takes the encoding alone.

Write a line of an export with serde_json::to_string(&json) and read one with serde_json::from_str::<serde_json::Value>(..). The deserialising functions read the tagged encoding. The plain encoding parses too, but a timestamp and a reference come back as strings, since the plain encoding does not tell them from a string; nothing prevents reading one.

Do not turn the JSON into a FirestoreValue with .into(). The conversion serialises the JSON as data, so {"$timestamp": "2023-11-14T22:13:20Z"} is stored as a map holding a string. A tagged value goes back to a value through FirestoreDb::deserialize_value_from_json(..).

Export and import

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let documents_path = db.get_documents_path();
let options = FirestoreJsonOptions::new();

let Some(document) = db.fluent().select().by_id_in("books").one("dune").await? else {
    return Ok(());
};
let json = FirestoreDb::serialize_doc_to_json(documents_path, &document, &options)?;
let line = serde_json::to_string(&json)?;

let json = serde_json::from_str(&line)?;
let document = FirestoreDb::deserialize_doc_from_json(documents_path, json)?;
db.fluent()
    .update()
    .in_col("books")
    .document(document)
    .execute()
    .await?;
Ok(())
}

An imported document carries its name, so write it with update(), or with a batch writer. insert().document(..) refuses a document whose name is set.

Encodings of each type

• Bytes are base64 with the standard alphabet and padding;
• a map is escaped as {"$map": {..}} only in the tagged encoding, and only when its single key starts with $, so that it is not read as a wrapper. A map with no key or several is never escaped;
• map keys are written in byte order. Whether a JSON reader keeps that order depends on the preserve_order feature of serde_json; the format does not depend on it;
• a Value with no type set is written as null in both encodings.

Documents

A document is one flat object holding __name__ and the fields.

{"__name__": "books/dune", "published": {"$timestamp": "2023-11-14T22:13:20Z"}, "title": "Dune"}

The keys follow the order of serde_json’s map. By default it is sorted, so a field named Title is written before __name__. With the preserve_order feature of serde_json, which the json-conversion feature does not enable, the order is __name__, the metadata, then the fields in byte order.

FirestoreJsonOptions::new().with_metadata(true) adds __create_time__ and __update_time__ when the document carries them: wrapped as $timestamp in the tagged encoding, as strings in the plain one. deserialize_doc_from_json ignores both, so the document it returns has neither.

__name__ is left out of a document without a name, such as an aggregation or a pipeline select result. A top-level field whose name starts with $ needs no escape, since the top level of a document is always read as fields.

Firestore refuses __name__, __create_time__, __update_time__ and every other name between double underscores as a stored field, so these keys cannot collide with a stored document’s fields. The functions do not check for them. A pipeline result with a field of such a name overwrites the key of the name in the object.

References and portability

A reference and __name__ are written relative to the database of documents_path when they point into it, and as a full name otherwise: projects/latestbit/databases/other/documents/books/dune. relative_document_name gives the same form for any name. A relative name resolves against the database of the documents_path that reads it, so an export from one database imports into another, and a full name keeps pointing where it pointed.

A name is read as a full name when it has the form projects/{project}/databases/{database}/documents/{path} with an odd number of segments. A relative document path has an even number of segments, so a document in a collection called projects is not mistaken for another project’s.

Integers and doubles

Firestore integers are 64-bit. A JSON number that a JavaScript reader parses as an IEEE double holds integers exactly only up to 2^53 − 1, so the tagged encoding writes anything beyond as $int with the decimal text. 2^53 itself goes to $int, since a JavaScript reader cannot tell it from 2^53 + 1. The plain encoding writes every integer as a number.

A double is written with a . or an exponent, 3.0 and 1e+300, so it reads back as a double and not as an integer. A coordinate of a geo point follows the same rule.

Exact doubles and float_roundtrip

serde_json parses a float one ULP (unit in the last place, the gap between a double and its neighbour) off for about one in six values with many digits, such as a vector component widened from f32. The text then reads back as the neighbouring double. A float round trip, a double written to text and parsed again to the same double, needs the float_roundtrip feature of serde_json.

The json-conversion feature turns it on. Cargo builds one serde_json per program with the features of every crate that uses it, so a program that enables json-conversion parses every float exactly, at the cost of a slower float parse.

Timestamps

Timestamps are written as RFC 3339 in UTC with a Z, with 0, 3, 6 or 9 fraction digits: 2023-11-14T22:13:20Z, 2023-11-14T22:13:20.500Z, 2023-11-14T22:13:20.123456Z. The range is the whole range Firestore accepts, from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999Z. FirestoreInstant ends 26 hours earlier, so the functions do not go through it.

Reading accepts an offset (+01:00), a lowercase t and z, and a space for T, and keeps nanoseconds. Firestore stores microseconds and cuts the digits beyond; the functions leave that to Firestore. FirestoreTimestamp::truncated_to_firestore_precision cuts them before a write.

Values that exist only in pipelines

Field references, variable references, functions and pipelines are parts of a pipeline request, and a stored document never holds one.

• The tagged encoding refuses them with JSON_EXPRESSION_VALUE, since they cannot be read back;
• the plain encoding shows what the caller holds: the text of a field or variable reference, the name of a function, and "pipeline".

Reading the tagged encoding

A wrapper takes exactly one form of content:

A component of $geo and $vector may be an integer literal, so {"$vector": [1, 2]} reads as two doubles.

Only the form of a wrapper is checked. An empty $vector, a latitude of 91 and a timestamp in the year 10000 read without an error, and Firestore refuses them on write. A timestamp in the year 10000 reads only in the signed form the writer produces, +10000-01-01T00:00:00Z; without the sign it is a JSON_INVALID_WRAPPER error.

A number above u64 in hand-written input reads as a double.

Errors

A failed conversion is a DeserializeError when it reads JSON and a SerializeError when it writes it. The code is in public.code, a constant of firestore::errors::codes, and the message names the JSON path of the value, such as `reviews[2].at`. A document error carries the full document name in document_path.

Caching

The library supports caching for collections and documents. A Firestore listener keeps the cache up to date when documents change, so updates are propagated across distributed instances automatically.

This avoids reading, and paying for, the same documents repeatedly. It is particularly useful for dictionaries, configuration and other data that changes rarely, and can reduce both cost and latency noticeably.

Caching is opt-in through cargo features:

• caching-memory for an in-memory cache, implemented with the moka cache library;
• caching-persistent for a persistent, disk backed cache, implemented with redb and protobuf.

The cache serves Core reads, and pipeline and SQL reads, on both editions. Which reads it answers is listed in Choosing a cache mode, and the Core query rules in What the cache answers.

Pipelines

Pipelines and SQL reads use the cache in the same sessions, read_through_cache and read_cached_only. The cache answers a pipeline on its own when every part of it is in the lists below, with the semantics Firestore gives a pipeline. Caching is meant for data such as static dictionaries, or sessions read by ID, so the cache filters, sorts and pages documents, and leaves computing to Firestore. Anything else runs on Firestore under read_through_cache, and fails with a CacheError under read_cached_only.

use firestore::*;
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize)]
struct Country { name: String, region: String }
async fn example(db: FirestoreDb, cache: FirestoreMemoryCache) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let european: Vec<Country> = db
    .read_through_cache(&cache)
    .fluent()
    .pipeline()
    .select()
    .from("countries")
    .filter(|q| q.field(path!(Country::region)).eq("Europe"))
    .sort(|e| [e.field(path!(Country::name)).asc()])
    .obj()
    .query()
    .await?;
let _ = european;
Ok(())
}

Sources:

• a cached collection, .from(name), with .parent(..) for a sub-collection. The collection must be preloaded, as for a Core query;
• .documents([..]), and a collection whose first filter pins __name__ with eq or is_in, read the named documents by ID, from any cached collection.

Stages: filter, sort, limit, offset, fields, add_fields and remove_fields. A limit or offset needs a sort before it, since Firestore returns unsorted documents in an order of its own. fields, add_fields and remove_fields take plain fields, with or without an alias, as top-level fields. A computed expression in them runs on Firestore.

Functions:

• comparisons: equal, not_equal, less_than, less_than_or_equal, greater_than, greater_than_or_equal, and is_null, is_not_null, is_nan, is_not_nan, which are built from them;
• and, or, not, exists, is_absent;
• equal_any, not_equal_any, array_contains, array_contains_all, array_contains_any.

Everything else runs on Firestore: arithmetic and string functions, aggregates, distinct, search, find_nearest, union, unnest, subqueries and variables, collection groups, database(), literals, force_index, raw stages, writes, like and every other function. So do Query Explain and a pipeline in a transaction or at a read time. The cache also sends a pipeline to Firestore when it meets a value Firestore fails the request on, such as not of a string, when a field name is not a simple identifier (letters, digits and underscores, not starting with a digit), and when it would have to order arrays, maps, geo points or vectors, in a sort or a less_than.

What differs from a pipeline that Firestore runs

• Results are eventually consistent, as for Core reads, see How the cache is updated.
• The cache does not check the edition of the database. On a Standard edition database, where Firestore fails every pipeline with FAILED_PRECONDITION, a pipeline the cache answers succeeds, and one it does not answer fails as on Firestore.
• A result from the cache has no execution_time in its metadata, and an empty result has no item without a document.
• A read by ID that names a document the cache does not hold runs on Firestore under read_through_cache. Under read_cached_only the missing documents are left out, as a Core batch read leaves them. In a preloaded collection, a __name__ filter that names a missing document is answered from the collection, where the document is absent.
• Pipelines never write to the cache: a pipeline that runs on Firestore leaves the cache as it was. Reads by ID through get fill a lazily loaded collection.
• Documents that tie on every sort key come back in document-name order from the cache, where Firestore’s order of them is unspecified, so a limit that cuts through them can keep other documents than Firestore would.

SQL

A SQL read runs as a pipeline, so the same lists apply. These shapes are cached, over one cached collection or documents(..):

• SELECT of columns, aliases, * and * EXCEPT (..);
• WHERE with comparisons, AND, OR, NOT, IS [NOT] NULL, IS [NOT] DISTINCT FROM, [NOT] IN (list), IN UNNEST(array) and BETWEEN;
• WHERE __name__ = 'id' and WHERE __name__ IN ('id', ..), read by ID;
• ORDER BY, and LIMIT and OFFSET with an ORDER BY.

GROUP BY, aggregates, DISTINCT, UNION, joins and subqueries, collection_group(..), database(), + - * / %, ||, LOWER, UPPER, CHAR_LENGTH, LIKE, ILIKE, CASE, COALESCE, IFNULL, SUBSTR and search run on Firestore.

On the Enterprise edition

The listener that keeps a cache current needs real-time updates on a database of the Enterprise edition. They are enabled when the database is created:

gcloud firestore databases create --database=my-database --edition=enterprise \
  --location=nam5 --enable-firestore-data-access --enable-realtime-updates

The setting is immutable, and Google’s documentation does not agree on its default for an Enterprise edition database, so set the flag explicitly, see Settings fixed at creation.

A cached Core query follows the order of the Standard edition, and documents with equal values can come back in another order from an Enterprise edition database, see What the cache answers.

Usage

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
fn config_env_var(name: &str) -> Result<String, String> {
    std::env::var(name).map_err(|e| format!("{name}: {e}"))
}
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Create an instance
let db = FirestoreDb::new(&config_env_var("PROJECT_ID")?).await?;

// Build the cache. This creates an internal Firestore listener, preloads the configured
// collections and starts listening for changes.
let cache = FirestoreCache::memory(&db)
    .preloaded_collection("test-caching")
    .build()
    .await?;

// Read through the cache: served from the cache when possible, from Firestore otherwise.
let my_struct: Option<MyTestStructure> = db
    .read_through_cache(&cache)
    .fluent()
    .select()
    .by_id_in("test-caching")
    .obj()
    .one("test-1")
    .await?;

// Read only from the cache, never contacting Firestore.
let my_struct: Option<MyTestStructure> = db
    .read_cached_only(&cache)
    .fluent()
    .select()
    .by_id_in("test-caching")
    .obj()
    .one("test-1")
    .await?;

cache.shutdown().await?;
let _ = my_struct;
Ok(())
}

For a persistent cache, use FirestoreCache::persistent(&db) and give it a directory with .data_dir("/var/cache/my-app"), which keeps the cache database and the listener resume tokens together.

Listener target IDs are assigned automatically starting at 1000. If your application runs its own listeners, move the cache’s range with .listener_target_base(...) or pin individual collections with .collection_with(name, |c| c.listener_target(...)).

Because load and shutdown take &self, a built cache can be shared directly as Arc<FirestoreMemoryCache> in your application state. FirestoreMemoryCache and FirestorePersistentCache are aliases that save you from spelling out the generic parameters.

shutdown stops the listener and releases the backend’s resources: the in-memory cache drops its documents, and the persistent cache closes its database file, so another cache can be opened over the same directory.

Caching named documents instead of a whole collection

.collection(name) subscribes the listener to the entire collection, even though it does not preload it - “lazy” only means the initial download is skipped. When you know which documents you care about, say so:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let cache = FirestoreCache::memory(&db)
    .collection_with("configs", |c| {
        c.documents(["site", "billing"]).preload_all()
    })
    .build()
    .await?;
let _ = cache;
Ok(())
}

The listener then watches exactly those documents, so unrelated changes in the collection are never streamed to your process or written into the cache, and preloading reads just those IDs. This is the shape to reach for with configuration, feature flags and reference data.

Such a collection is never listable, whatever its load mode: it holds a chosen subset, so list and query would return a partial answer that looks complete.

Changing the cached collections at runtime

The set of cached collections does not have to be fixed when the cache is built:

use firestore::*;
async fn example(cache: FirestoreMemoryCache) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
cache
    .add_collection(FirestoreCacheCollection::new("currencies").preload_all())
    .await?;

cache.remove_collection("currencies").await?;
Ok(())
}

add_collection downloads the collection first if it is preloaded, publishes it only once it is complete, and then extends the listener - so a listing never observes it half filled, and nothing written during the download is missed. A download that fails, even partway, fails add_collection with that error and publishes nothing. On a persistent cache, add_collection first discards any rows an earlier run left for the collection, whatever its load mode, so PreloadAllIfEmpty downloads it again. remove_collection does the reverse, and also forgets the collection’s resume token so its listener target ID cannot be reused against a different query. Use remove_collection_at for a sub-collection, whose absolute path a bare name cannot address. A collection whose target Firestore ends, for example on PERMISSION_DENIED, stays configured and suspended, so add_collection reports it as already cached until remove_collection is called. On an Enterprise edition database the listener needs real-time updates, see Caching.

FirestoreDb handles created earlier with read_through_cache or read_cached_only pick both up immediately: they share the cache’s backend rather than a copy of it. A cache can also be built with no collections at all and populated entirely at runtime.

Choosing a cache mode

• db.read_through_cache(&cache) serves what it can from the cache and goes to Firestore for the rest. This is the mode to reach for by default.
• db.read_cached_only(&cache) never contacts Firestore. Reads by ID return None on a miss, and requests the cache cannot answer completely return an error.

Which operations use the cache:

What the cache answers

A Core query on a preloaded collection runs in the library’s cached query engine. The engine returns the documents Firestore returns, in the order Firestore returns them:

• timestamps and numbers. Timestamps compare by seconds and then by the fraction of a second, at the precision Firestore uses:

  • ==, the range operators and array-contains compare a timestamp on its own at the nanosecond;
  • in, not-in and array-contains-any compare at the microsecond that Firestore stores;
  • a timestamp inside an array or a map compares at the microsecond, whichever operator is used.

  An integer and a double compare by value, so a filter on 1 matches the double 1.0, and order_by sorts them together;

• order and range filters on every type. <, <=, >, >=, order_by and cursors follow Firestore’s order of types (null, booleans, numbers, timestamps, strings, bytes, references, geo points, arrays, vectors, maps) and its order within each type. A range filter matches values of its own type only, and never NaN;

• several order_by fields. The first field that differs decides. Ties end on the document name, in the direction of the last field. A document that lacks an order_by field is not returned, as in Firestore;

• no order_by. Results come in the order of the fields compared with <, <=, >, >=, != or not-in, and then by document name;

• cursors, offset and limit. The cursors apply first, then offset, then limit, so limit(10).offset(5) skips 5 documents and returns up to 10. A cursor is one position in the order of its fields;

• in, not-in and array-contains. in matches a scalar field equal to any listed value, not-in one equal to none of them, leaving out documents where the field is null or absent, and array-contains an array field holding the value;

• equality on arrays, maps, bytes, references and geo points. ==, !=, in, not-in, array-contains and array-contains-any match these values. Arrays are equal when their elements are equal in order, and maps when their fields are equal by key; inside both, an integer equals the double of the same value, and null and NaN equal themselves. != also matches a field of another type;

• null and NaN in a filter. A null in a filter value matches nothing: == null, != null and array-contains null return no documents, a null in an in or array-contains-any list is ignored, and a not-in list that holds a null returns no documents. NaN equals nothing, itself included, and is in no range; != and not-in match it. Use is_null to find null fields: is_null does not match a document without the field, and is_not_nan does not match a document where the field is null or absent.

The engine applies none of find_nearest, return_only_fields and explain_options. A query that sets one is answered by Firestore under read_through_cache, and fails with CacheIncompleteCollection under read_cached_only. A get or batch get with .fields(..) follows the same rule, and fails with CacheFieldMaskUnsupported under read_cached_only.

A query answered from the cache is eventually consistent, see How the cache is updated.

On the Enterprise edition

The cached order is that of the Standard edition. Documents with equal values can come back in another order from an Enterprise edition database than from the cache, see Query and write semantics.

Load modes, and why listings need preloading

• PreloadNone (.collection(name)): don’t preload anything, just fill the cache while working;
• PreloadAllDocs (.collection_with(name, |c| c.preload_all())): preload all documents in the collection;
• PreloadAllIfEmpty (.collection_with(name, |c| c.preload_all_if_empty())): preload all documents unless an earlier run finished downloading the collection. This is useful for the persistent cache, where the saving applies when the cache loads; for the memory cache it is the same as PreloadAllDocs, since an in-memory cache always starts empty. The persistent cache keeps a marker for each collection whose download has finished, and skips the download only for a collection with a marker. A collection whose download was interrupted, for example because the process was killed, has no marker: its rows are discarded and it is downloaded again on the next start. invalidate_all and a resync after Firestore resets the listener target remove the marker too, and each costs one more download on the next start.

.preloaded_collection(name) picks the appropriate preloading mode for the backend.

A lazily filled collection holds only the documents that happened to be read through it. Answering a list or query from it would return a subset that looks like a complete answer, so the library refuses to do so: read_through_cache quietly falls back to Firestore, and read_cached_only returns an error naming the collection. If partial results are genuinely acceptable, opt in with .incomplete_collection_policy(FirestoreCacheIncompleteCollectionPolicy::PartialResults).

How the cache is updated

• When you read a document by ID through the cache and it is not there, it is fetched from Firestore and cached;
• The Firestore listener updates the cache when a document changes, whether the change came from your application or from elsewhere;
• Preloading at startup.

Cached results are eventually consistent: they reflect the last state the listener delivered, so a write may take a moment to show up. Do not cache data that must be read at strong consistency. This holds for pipeline and SQL reads that the cache answers too.

Firestore also reports how many documents a target matches. When that disagrees with what a preloaded collection holds - which means changes were missed, typically deletes that happened while the listener was disconnected - the cache drops the collection and has Firestore replay it. The count is only compared when it can be trusted: a collection the cache expires entries from, or fills lazily, is never checked this way.

When Firestore resets or removes a listener target - after a reconnect, or when a stored resume token has expired - the cache drops what it holds for that collection and Firestore replays it. While that replay is in progress the collection stops answering list and query from the cache: read_through_cache falls back to Firestore, and read_cached_only returns a CacheError rather than a listing that looks complete but is not.

On the Enterprise edition

The listener that updates the cache needs real-time updates on an Enterprise edition database, a setting chosen when the database is created, see Caching.

Full examples are available here and here.

Index management

Composite indexes, vector indexes, single-field overrides and TTL policy are usually managed outside your Rust code: through the console, gcloud or the Firebase CLI’s firestore.indexes.json. The library supports declaring them in Rust instead, next to the structs and queries that need them, and reconciling Firestore with one explicit call.

This page covers the statement, its runs and its reports. The kinds of index have pages of their own:

• Composite and vector indexes, with the single-field overrides of the Standard edition;
• Search indexes, on the Enterprise edition;
• TTL and the Enterprise edition: the TTL policy, and the index rules of an Enterprise edition database, unique indexes and density.

Enabling the admin feature

Index management, and the other administrative operations under db.fluent() that need the Firestore Admin API, are behind the admin cargo feature, off by default:

[dependencies]
firestore = { version = "...", features = ["admin"] }

Nothing runs at FirestoreDb construction. An administrative operation only runs when your code calls one of its terminals: .plan()/.sync() for a declaration, or .get()/.execute() for one index at a time.

Declaring a collection group’s indexes

Everything starts from db.fluent().indexes().collection_group(...), then a chain of declarations ending in .plan() or .sync():

use firestore::*;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct Order {
    customer_id: String,
    placed_at: FirestoreTimestamp,
    tags: Vec<String>,
    embedding: Vec<f64>,
    bio: String,
    expires_at: FirestoreTimestamp,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let report = db
    .fluent()
    .indexes()
    .collection_group("orders")
    .composite(|i| {
        i.indexes([
            i.index([
                i.field(path!(Order::customer_id)).asc(),
                i.field(path!(Order::placed_at)).desc(),
            ]),
            i.index([
                i.field(path!(Order::tags)).array_contains(),
                i.field(path!(Order::placed_at)).desc(),
            ])
            .all_descendants(),
            i.index([
                i.field(path!(Order::customer_id)).asc(),
                i.field(path!(Order::embedding)).vector(768),
            ]),
        ])
    })
    .field_overrides(|f| {
        f.fields([f.field(path!(Order::bio)).exempt()])
    })
    .ttl([path!(Order::expires_at)])
    .sync()
    .await?;
println!("{report}");
Ok(())
}

• .composite() builds the group’s composite and vector indexes, see Composite and vector indexes;
• .field_overrides() builds the single-field overrides of a Standard edition database, see Single-field overrides;
• .ttl() names the one field per collection group whose timestamp value enables document TTL, see TTL policy.

Each of .composite(), .field_overrides() and .ttl() replaces whatever an earlier call in the same chain set, so call each at most once. A field override or a TTL path may not repeat. These rules, and the structural rules of each index in Composite and vector indexes, are checked at .plan()/.sync() before anything is written, and return InvalidParametersError naming what is wrong. Every other rule, including the rules of the database’s edition, is Firestore’s: .plan() and .sync() send what you declare, and Firestore answers INVALID_ARGUMENT with its own message when it refuses it.

plan() versus sync()

.plan() is read-only: it lists what Firestore already has for the owned group and reports what .sync() would change, without writing anything.

.sync() reconciles Firestore with the declaration: it creates missing composite indexes, writes field overrides that differ from what is declared, and enables TTL fields that are not yet configured.

Both return a value with a Display impl, so either can be printed or logged directly, with or without a tracing subscriber:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let plan = db.fluent().indexes().collection_group("orders").plan().await?;
println!("{plan}");
Ok(())
}

Ownership: one statement owns one collection group

A fluent chain names exactly one collection group, the same way .from() names one collection for a query, and that group is the unit of ownership: .prune_undeclared() never reaches an index, override or TTL field on any other group. Several groups need several statements; a startup function typically runs one per group in turn.

There is no name prefix for a statement’s own indexes. Firestore’s Index and Field resources are server-assigned and carry no label or description, so ownership is expressed entirely by which collection group a statement names, never by anything stored on the index itself.

Subcollections

.collection_group(...) takes a subcollection’s ID the same way it takes a top-level collection’s: .collection_group("posts") names every collection anywhere in the database whose ID is posts, at any depth. A slash-delimited path is rejected the same way FirestoreCollectionId rejects one everywhere else in the library, with a message pointing at .parent(db.parent_path(...)).

Collection scope, the default, declares an index for a query scoped to one subcollection through .from("posts").parent(parent_path). Add .all_descendants() to the same index for a query across every posts subcollection at once, through .from("posts").all_descendants(). The two scopes are different query shapes, so a group can declare both at the same time, one index for each:

use firestore::*;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct Post {
    published: bool,
    created_at: FirestoreTimestamp,
    tags: Vec<String>,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("posts")
    .composite(|i| {
        i.indexes([
            i.index([
                i.field(path!(Post::published)).asc(),
                i.field(path!(Post::created_at)).desc(),
            ]),
            i.index([
                i.field(path!(Post::tags)).array_contains(),
                i.field(path!(Post::created_at)).desc(),
            ])
            .all_descendants(),
        ])
    })
    .sync()
    .await?;

let user_path = db.parent_path("users", "alice")?;
let own_posts: Vec<Post> = db
    .fluent()
    .select()
    .from("posts")
    .parent(&user_path)
    .filter(|q| q.for_all([q.field(path!(Post::published)).eq(true)]))
    .order(|o| o.fields([o.field(path!(Post::created_at)).desc()]))
    .obj()
    .query()
    .await?;

let tagged_rust: Vec<Post> = db
    .fluent()
    .select()
    .from("posts")
    .all_descendants()
    .filter(|q| q.for_all([q.field(path!(Post::tags)).array_contains("rust")]))
    .order(|o| o.fields([o.field(path!(Post::created_at)).desc()]))
    .obj()
    .query()
    .await?;
let _ = (own_posts, tagged_rust);
Ok(())
}

The ID is shared across the whole database, including subcollections your own code never writes to. A statement owns the collection group, so .prune_undeclared() on collection_group("posts") reaches the indexes of every posts collection in the database.

Removing what you no longer declare

By default, an index, override or TTL field Firestore lists for the owned group but this statement does not declare is only reported, never touched. Add .prune_undeclared() to also delete undeclared composite indexes, revert undeclared field overrides to automatic indexing, and disable undeclared TTL fields:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("orders")
    .prune_undeclared()
    .sync()
    .await?;
Ok(())
}

An undeclared index in state NEEDS_REPAIR is deleted like any other undeclared index once pruning is on; Firestore does not exempt it. A declared index matched to a listed NEEDS_REPAIR index is different: .sync() only ever reports that one, never deletes or recreates it automatically.

A pruning sync sends its deletes last, and only after every index the same sync created has finished building, so a replaced index keeps serving queries until the new one can. The sync waits for that even without .wait_until_ready(...), and a composite index build can take minutes. The wait ends by the sync’s deadline: the .wait_until_ready(timeout) timeout when you set one, 30 minutes from the start of the sync otherwise. If a new index fails to build, or is still building at the deadline, the sync deletes nothing and returns the error.

The sync also deletes nothing when Firestore answers a CreateIndex with ALREADY_EXISTS and a fresh listing shows a matching index: another deployment created it after the sync planned its changes. The sync reports the index as unchanged, logs a warning naming it and carries on with everything else.

When the fresh listing shows no matching index, the sync returns the ALREADY_EXISTS error instead of reporting the index as unchanged. Firestore keeps answering ALREADY_EXISTS for a while after an index with the same fields is deleted, while the listing no longer shows it, for up to about a minute on an Enterprise edition database. Run the sync again once the answer changes.

In both cases withheld_deletes on the report lists the indexes left in place and the reason. Reverts and TTL disables do not wait for new indexes.

.prune_undeclared().plan() previews a pruning sync without writing anything: delete_indexes, revert_fields and disable_ttl list exactly what a pruning .sync() would remove, and kept_undeclared_indexes, kept_undeclared_fields and kept_undeclared_ttl list what it would leave alone without .prune_undeclared().

Running from several instances

Several replicas of the same application, such as a Kubernetes deployment’s pods, should not each call .sync() on their own. Two replicas racing on the same collection group create, delete and revert the same resources at the same time, and during a rolling update old and new replicas declare different indexes, so .prune_undeclared() on both flips indexes back and forth for as long as the rollout takes.

Run .sync() once per release, in two steps. A sync that prunes before the new replicas are up deletes an index the still-running old ones query, and their queries fail with FAILED_PRECONDITION for as long as the rollout takes. So create and update what the release declares before the rollout, without .prune_undeclared(), then prune what it no longer declares once the rollout has finished:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let report = db
    .fluent()
    .indexes()
    .collection_group("orders")
    .wait_until_ready(Duration::from_secs(30 * 60))
    .sync()
    .await?;
println!("{report}");
Ok(())
}

Add .prune_undeclared() for the second run. .wait_until_ready(...) makes either run fail when a build does not finish in time, so neither reports success while indexes are still building.

From a Kubernetes Job, run the first as an early step and the second once the rollout is serving; from a CI/CD step, the same split before and after the deploy step. From Helm, the helm.sh annotations turn a Job into a chart hook; a pre-install,pre-upgrade hook runs the create-only sync, and a post-upgrade hook the pruning one:

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-index-sync-create
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 2100
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: orders-index-sync
      containers:
        - name: sync-indexes
          image: your-app:1.4.2
          args: ["--sync-indexes"]

The pruning hook is the same manifest under a different name, "helm.sh/hook": post-upgrade, and --sync-indexes --prune (or whatever flag your binary uses to turn on .prune_undeclared()) in args. Run helm upgrade with --wait or --atomic (which implies --wait): Helm’s documented install lifecycle waits for the chart’s resources to reach a ready state before running post-install, and post-upgrade is documented the same way, “after all resources have been upgraded”; without --wait, nothing holds post-upgrade back until the new pods are actually serving, and pruning at that point can still delete an index one of them still queries. See Helm’s chart hooks and the --wait and --atomic flags on helm upgrade.

Set the Job’s backoffLimit to 0 and activeDeadlineSeconds above the sync’s timeout, so Kubernetes does not restart or kill it mid build. A Helm hook is only waited for up to helm upgrade’s own --timeout, 5 minutes by default, independent of activeDeadlineSeconds: pass a longer one (helm upgrade --timeout 35m, or more) alongside an activeDeadlineSeconds in the same range, or Helm gives up on the hook long before Kubernetes does.

A pre-install hook runs before Helm creates any of the chart’s ordinary resources, so the serviceAccountName the Job names must already exist by then: give the ServiceAccount its own pre-install,pre-upgrade hook annotation, at helm.sh/hook-weight: "-1", a lower weight than the create Job’s 0, so Helm creates it first.

Helm’s documented hook ordering also names pre-rollback and post-rollback: pre-rollback runs “after templates are rendered, but before any resources are rolled back”, the same shape as pre-install and pre-upgrade. Put the create-only Job on pre-rollback too, so the indexes the restored release needs exist before its pods come back; Helm’s docs do not say whether a rollback’s hooks are rendered from the revision being restored or the one currently installed, so confirm which one runs in your own cluster before relying on it. Leave the pruning Job off post-rollback: a rollback has no record of what the release you are rolling back from declared, so pruning there can delete an index the restored release still needs.

A plain Job outside Helm has no hook-delete-policy to remove the previous run, and a completed Job under the same fixed name fails a later kubectl apply with an immutable-field error. Set ttlSecondsAfterFinished on it, or delete the previous Job before applying the next one.

The application’s own replicas do not need to sync at all. If you want one to check its own declaration against Firestore, .plan() with .prune_undeclared() at startup is read-only and safe from any number of replicas at once: log a warning, or emit a metric, when create_indexes, update_fields, enable_ttl, delete_indexes, revert_fields or disable_ttl comes back non-empty. unchanged, unchanged_fields and unchanged_ttl are non-empty in the steady state and say nothing about drift on their own. Do not wire this into the replica’s readiness probe: after a Helm rollback, or when an old pod restarts once the release job has already run, every replica would see the same drift and fail its probe at once.

The job’s service account needs roles/datastore.indexAdmin, the role .sync() always needs. Replicas that only call .plan() need roles/datastore.viewer.

When two syncs do overlap regardless, index creates and deletes converge on their own: a CreateIndex answered ALREADY_EXISTS for an index a fresh listing shows, and a DeleteIndex answered NOT_FOUND, both count as done. A field revert Firestore refuses because another writer already reverted it counts as done too. One refused while another writer’s revert is still in progress is polled until that revert finishes, within this sync’s deadline, and then counts as done. A TTL disable does not get that wait: Firestore reports no in-progress state for a TTL field, so a disable refused while another writer’s disable is still running fails. It converges only when the other writer’s disable had already finished. A field override write or a TTL enable that races another sync’s is not checked against the field’s current state either, so it can still fail.

The order .sync() writes changes in

.sync() applies one plan in a fixed order, stopping at the first write that fails:

• creates missing composite indexes;
• writes declared field overrides;
• disables undeclared TTL fields;
• enables declared TTL fields;
• reverts undeclared field overrides;
• deletes undeclared composite indexes, once the new ones have finished building.

Every TTL disable finishes before any TTL enable is sent, since Firestore allows only one TTL field per collection group. Declaring a .ttl() field while the group lists a different TTL field fails .plan() and .sync() with InvalidParametersError before anything is written, unless .prune_undeclared() disables the listed field.

Waiting for changes to finish

By default .sync() returns as soon as the last change is requested; a created index or a newly enabled TTL field can still be building. .wait_until_ready(timeout) polls until every started change reaches a terminal state, or returns an error naming what is still pending once timeout is up:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("orders")
    .wait_until_ready(Duration::from_secs(900))
    .sync()
    .await?;
Ok(())
}

.wait_until_ready_with_options(...) also sets the poll interval, 5 seconds by default.

An index build takes time even on an empty collection: about a minute for one index of any kind (ordered, array, vector, text, geo, or text with geo) on an Enterprise edition database. On a Standard edition database a lone vector index takes about six minutes, and a sync of three composite indexes, two field overrides and a TTL field about seven. Set the timeout above these, and larger for a collection that holds data.

The timeout covers the whole sync, from the start of the call, and every wait inside it ends by the same deadline. The last round of polls goes out at the deadline itself. One poll may take up to 10 seconds to answer (or the whole timeout, if that is shorter), so an error can arrive that long after the deadline. A poll that takes longer is sent again next round, and so is one that Firestore answers with UNAVAILABLE, DEADLINE_EXCEEDED or NOT_FOUND.

A sync that fails after it started writing leaves the writes already applied in place, since .sync() never rolls those back. It logs the report built so far at warn before returning the error, so what went through survives in the log even though the returned Result is an error. This holds for any failure: a refused write, a failed operation, or a deadline passed in the final wait or in a wait between two writes.

A write to a field waits for its own earlier write in the same sync

Two writes to the same field resource are never sent back to back: overlapping writes to one field are not known to be safe, so the later one waits for the earlier one to reach a terminal state first. This applies whenever one sync writes a field override and also enables or disables TTL on that same field, and to a TTL move in particular: disabling the old TTL field, then enabling the new one, waits for the disable to settle before the enable is sent. Writes to different fields are sent back to back, since Firestore accepts that.

These waits happen even without .wait_until_ready(...). They end by the sync’s deadline: the .wait_until_ready(timeout) timeout when you set one, 30 minutes from the start of the sync otherwise, which is long enough for a field override’s own single-field index build. There is no separate way to configure that default today.

A revert can wait the same way for a different sync’s write, not this one’s own: see Running from several instances.

One index at a time

.plan() and .sync() reconcile a whole declaration. Four calls under db.fluent().indexes() do one thing each, with no declaration, diff or pruning: list(), create(), delete() and update_field(). The SQL index statements run through the same calls.

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let created = db
    .fluent()
    .indexes()
    .create()
    .collection_group("orders")
    .index(|i| i.index([i.field("customer_id").asc(), i.field("placed_at").desc()]))
    .wait_until_ready(Duration::from_secs(900))
    .execute()
    .await?;

let listing = db
    .fluent()
    .indexes()
    .list()
    .collection_group("orders")
    .get()
    .await?;
println!("{} indexes", listing.indexes.len());

db.fluent()
    .indexes()
    .update_field()
    .collection_group("sessions")
    .field("expires_at")
    .enable_ttl()
    .exempt()
    .wait_until_ready(Duration::from_secs(900))
    .execute()
    .await?;

db.fluent().indexes().delete().id(created.id).execute().await?;
Ok(())
}

• list() returns a FirestoreIndexListing of the composite indexes and field overrides of every collection group, or of the one that .collection_group(..) names. The listing leaves out the groups Firestore reserves, such as __default__;
• create() starts the build of one index and returns it as Firestore lists it: CREATING, or READY after .wait_until_ready(timeout). The id of the result is the one Firestore assigned. Creating an index that exists fails with DataConflictError, as .sync() does not: it counts that answer as done when the listing shows the index. .if_not_exists() makes an existing index the result instead, in whatever state it has. With a wait, an index that another caller is still building is polled until it is no longer CREATING, under the same deadline and retries as a build of the call’s own, and one that ends in NEEDS_REPAIR fails with SystemError, code INDEX_NEEDS_REPAIR. NEEDS_REPAIR is the state of an index whose last build failed and left no operation running;
• delete() takes an index by .id(id), by .collection_group(group).id(id), or by .collection_group(group).index(..), the definition that create() took, which deletes the one listed index it matches, as .sync() matches a declaration. It returns the index as it was listed just before. .if_exists() makes an index that is not there, and a definition that matches none, return None instead of DataNotFoundError;
• update_field() writes one field, or every field without a setting of its own with .all_fields(), in one request: .indexes(..) or .exempt() set its single-field indexes, .revert_override() removes the override, and .enable_ttl() and .disable_ttl() set its TTL. It returns the field as Firestore reads it after the write, or after the wait. An update that changes nothing, and TTL on .all_fields(), fail with InvalidParametersError before any request.

A delete by ID reads the index before it deletes anything. Firestore ignores the collection group in a DeleteIndex request, so an ID that belongs to another group would be deleted all the same. The library deletes the index only when it belongs to the group the caller named, and fails with DataNotFoundError otherwise, deleting nothing. Without a group the ID alone names the index: an ID is unique within one database.

A delete by definition fails with DataNotFoundError when nothing matches, and with DataConflictError, naming the IDs, when several indexes match. On an Enterprise edition database two indexes over the same fields differ by unique and density, and the definition tells them apart.

The four calls run no edition rule of their own, and neither do .plan() and .sync(): Firestore’s INVALID_ARGUMENT answers arrive as DatabaseError, and so does the emulator’s UNIMPLEMENTED, see The Firestore emulator. A reserved group such as __default__ fails with InvalidParametersError.

Failures the library finds itself carry a code in the error. The codes are the constants of firestore::errors::codes:

Field paths

Field paths compare in one spelling, so `expires_at` and expires_at, or a.`b` and `a`.b, are the same field. A segment needs backticks only when it is not a plain identifier (letters, digits and _, not starting with a digit).

Unrecognised items

Firestore can list index or field shapes this crate’s domain model has no representation for: a search field configured other than as one text or geo spec, a MongoDB-compat or Datastore-mode API scope, a COLLECTION_RECURSIVE query scope, an unspecified field order, or a multikey or sharded index. These are reported in unrecognised, with the reason, and never proposed for deletion or reversion, even with .prune_undeclared() set, since the plan cannot know that removing them is what the declaration intends.

Logs and spans

Index management logs through tracing. One span covers a whole .plan() or .sync() call (Firestore Index Plan or Firestore Index Sync), and nests child spans for listing, diffing, applying and, when .wait_until_ready() is set, waiting. Every span records its own elapsed time.

An excerpt of the log of a .plan() call for an empty collection group (no composite indexes, field overrides or TTL fields declared, and none listed for the group):

DEBUG Firestore Index Plan{/firestore/collection_group="firestore-rs-index-sync-test" /firestore/prune=false}:Firestore Index List{...}: firestore::db::admin::indexes: Listed the collection group's indexes and fields. indexes=0 fields=0 skipped_indexes=7 skipped_fields=0
 INFO Firestore Index Plan{...}: firestore::db::admin::indexes: Existing state: 0 indexes, 0 field overrides, 0 TTL fields collection_group="firestore-rs-index-sync-test" composite_indexes=0 field_overrides=0 ttl_fields=0
 INFO Firestore Index Plan{...}: firestore::db::admin::indexes: Firestore index plan: no changes (0 unchanged, 0 pending, 0 field overrides unchanged, 0 ttl unchanged)
 collection_group="firestore-rs-index-sync-test" create_indexes=0 update_fields=0 enable_ttl=0 unchanged=0 pending=0 delete_indexes=0 revert_fields=0 disable_ttl=0 kept_undeclared_indexes=0 kept_undeclared_fields=0 kept_undeclared_ttl=0 unrecognised=0
 INFO Firestore Index Plan{...}: firestore::db::admin::indexes: plan() reports what sync() would change; nothing was applied. collection_group="firestore-rs-index-sync-test"

skipped_indexes counts composite indexes ListIndexes returned for other collection groups sharing the same database; the library filters listed items down to the owned group before computing anything, since a raw ListIndexes on one group’s parent can return every group’s indexes.

The plan event’s fields split undeclared items the same way pruning does: delete_indexes, revert_fields and disable_ttl are what a pruning sync would remove, and kept_undeclared_indexes, kept_undeclared_fields and kept_undeclared_ttl are what it would leave alone. Both are zero here, since the group carries nothing undeclared.

Each call reads the database’s edition first, in a Firestore Database Edition span, ahead of the listing. Its time counts in the report’s total and in no phase.

.sync() also logs a Firestore Index Apply span with one child span per applied action, and a Firestore Index Wait span when waiting is requested, with one child span per operation polled.

Timings and unchanged TTL on the report

FirestoreIndexSyncReport carries a timings value alongside its lists: the whole call’s duration, plus how long listing, applying and, when you waited, waiting each took. Display prints it as timings: total <n> ms, followed by , list <n> ms, , apply <n> ms and , wait <n> ms for whichever phases ran.

unchanged_ttl reports declared TTL fields that were already active before this sync ran, separately from enabled_ttl, which is what this sync itself enabled.

The Firestore emulator

The emulator serves no Firestore Admin API, whatever its edition. .plan(), .sync(), list(), create(), delete() and update_field(), and the SQL index statements, fail with the emulator’s own UNIMPLEMENTED, which arrives as DatabaseError and whose message names the admin method. Startup code that runs .sync() against the emulator handles that error, or runs it only where FIRESTORE_EMULATOR_HOST is not set.

IAM roles

.plan() needs only roles/datastore.viewer. .sync() needs roles/datastore.indexAdmin too, to create, update and delete indexes and fields. Both read the database’s edition, through the datastore.databases.getMetadata permission each of the two roles carries.

list() needs roles/datastore.viewer, and create(), delete() and update_field() need roles/datastore.indexAdmin. A delete by definition also lists the group’s indexes and reads the database’s edition.

Running the tests against a real project

tests/admin-indexes-tests.rs needs the admin feature, a real Firestore project through GCP_PROJECT, and local application default credentials. The fast, read-only test runs by default:

GCP_PROJECT=your-project cargo test --test admin-indexes-tests --features admin -- --nocapture

Every other test in that file creates, updates or deletes real indexes, overrides or TTL policy, and waits for builds that take minutes even on an empty collection, see Waiting for changes to finish, so those carry #[ignore] and only run explicitly:

GCP_PROJECT=your-project cargo test --test admin-indexes-tests --features admin -- --ignored --nocapture

With the sql feature next to admin, the file also runs the index statements against the database of GCP_PROJECT and against the Enterprise edition database of GCP_ENTERPRISE_DATABASE_ID. Each of those tests carries #[ignore] too, and runs on a collection group of its own that it empties again:

GCP_PROJECT=your-project GCP_ENTERPRISE_DATABASE_ID=your-enterprise-database \
  cargo test --test admin-indexes-tests --features admin,sql -- --ignored sql_ --nocapture

Full examples available here, here and here.

Composite and vector indexes

.composite() builds the composite indexes of the collection group a statement owns, see Index management for the statement itself. Order a field with .asc()/.desc(), or mark it .array_contains(), and chain .all_descendants() onto the whole index for a collection-group index rather than a single collection.

use firestore::*;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct Order {
    customer_id: String,
    placed_at: FirestoreTimestamp,
    tags: Vec<String>,
}
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("orders")
    .composite(|i| {
        i.indexes([
            i.index([
                i.field(path!(Order::customer_id)).asc(),
                i.field(path!(Order::placed_at)).desc(),
            ]),
            i.index([
                i.field(path!(Order::tags)).array_contains(),
                i.field(path!(Order::placed_at)).desc(),
            ])
            .all_descendants(),
        ])
    })
    .sync()
    .await?;
Ok(())
}

On a Standard edition database Firestore takes a composite index of at least two fields, unless it is a single vector field on its own; an Enterprise edition database takes an index of one field, see TTL and the Enterprise edition. Firestore answers INVALID_ARGUMENT to an index it refuses, and .plan() and .sync() send it as declared. An index holds at most one .array_contains() field, which .plan() and .sync() check before anything is written, see Declaring a collection group’s indexes.

Vector indexes

A .vector(dimension) field makes a vector index, which a Core vector search needs on a Standard edition database. The dimension is the one every indexed vector in the field must have. A vector field must be the last field of the index it belongs to, and ordered fields can come before it:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("articles")
    .composite(|i| {
        i.indexes([
            i.index([i.field("embedding").vector(768)]),
            i.index([i.field("category").asc(), i.field("embedding").vector(768)]),
        ])
    })
    .sync()
    .await?;
Ok(())
}

A vector index cannot be forced on a pipeline, and the pipeline find_nearest stage does not use it, see Query plans and cost.

__name__ handling

On a Standard edition database, a composite index declared without __name__ matches a listed index that has it where Firestore puts it: last in an ordinary index, or directly before the vector field in a vector index. The direction of that __name__ does not matter for the match. A __name__ anywhere else makes the listed index a different one. Declare __name__ yourself only when a query needs a specific direction; an explicit declaration then matches only a listed index with exactly the same fields, __name__ included.

An Enterprise edition database appends no __name__, so there a declaration matches only a listed index with exactly its fields.

Single-field overrides (Standard edition)

A Standard edition database indexes every field on its own automatically. .field_overrides() builds single-field overrides of that: .exempt() removes a field from automatic single-field indexing entirely, and .indexes([...]) replaces it with exactly the indexes listed. An Enterprise edition database has no automatic indexes to override, see TTL and the Enterprise edition.

Indexing only chosen fields

f.all_fields() targets Firestore’s special * field path, scoped to the owned collection group. Combined with .exempt(), it excludes every field in the group from automatic single-field indexing; list the fields you do want indexed by name alongside it:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("orders")
    .field_overrides(|f| {
        f.fields([
            f.all_fields().exempt(),
            f.field("customer_id").indexes([f.ascending()]),
            f.field("status").indexes([f.ascending(), f.descending()]),
        ])
    })
    .sync()
    .await?;
Ok(())
}

A named field declared alongside all_fields() wins for that field, since it is more specific: this is how “everything except these fields” is expressed. Composite indexes are unaffected by all_fields().

There is no database-wide equivalent. A collection-group statement never reaches the __default__ group’s own * field, so a default applied across every collection group at once is deliberately out of reach from this API.

A field override replaces the whole automatic set

Declaring only .array_contains() on a field Firestore would otherwise also index for equality and ordering removes that automatic support: Firestore does not merge an override with its defaults, and neither does firestore.indexes.json. .indexes([...]) always takes the whole set to maintain for a field, never one index to add to what is already there. An empty list, or .exempt() (the same thing), excludes the field from single-field indexing entirely.

Full examples available here and here.

Search indexes (Enterprise edition)

The pipeline search stage runs against search indexes, which only a database of the Firestore Enterprise edition has. A text index covers one or more string fields, and a geo index covers a geo point field. Declare them as composite indexes with .text() and .geo() on a field:

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("restaurants")
    .composite(|i| {
        i.indexes([
            i.index([i.field("menu").text(), i.field("description").text()])
                .search_options(
                    FirestoreSearchIndexOptions::new().with_text_language("en".to_string()),
                ),
            i.index([i.field("location").geo()]),
        ])
    })
    .sync()
    .await?;
Ok(())
}

• A search index holds text and geo fields only. An index that mixes them with ordered, array or vector fields is rejected with InvalidParametersError at .plan()/.sync();
• A text search covers all the text fields of the index it uses, and Firestore offers no index that matches one field by name;
• .search_options(..) sets the options of a text index. text_language is the BCP 47 language tag the fields are indexed in, such as en; when it is left unset, Firestore detects each document’s language and lists the option as und, the code for an undetermined language. text_language_override_field_path names the field that holds a document’s own language, and Firestore reads language when it is unset. An index declared without options matches the index Firestore lists with those defaults;
• .geo_with(config) takes a FirestoreGeoIndexConfig that can leave GeoJSON points out of a geo index, which Firestore indexes by default;
• .all_descendants() on the index scopes it to a collection group, which a collection-group search needs;
• a search index is always built as FirestoreIndexDensity::SparseAny, see Density;
• a Standard edition database answers a search index with INVALID_ARGUMENT.

TTL and the Enterprise edition

TTL policy

.ttl() names the one field per collection group whose timestamp value enables document TTL (time-to-live): Firestore deletes a document itself once the timestamp in that field has passed. It works the same on both editions:

use firestore::*;
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
struct Session { expires_at: FirestoreTimestamp }
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("sessions")
    .ttl([path!(Session::expires_at)])
    .sync()
    .await?;
Ok(())
}

.sync() enables a declared TTL field that is not yet configured, and .prune_undeclared() disables an undeclared one. Firestore allows only one TTL field per collection group. To move the TTL to another field, declare the new field and set .prune_undeclared(); The order .sync() writes changes in describes how the move is applied, and when a second TTL field is refused.

The Enterprise edition

An Enterprise edition database builds no index on its own: there are no automatic single-field indexes, so every index a query needs is one the statement declares. .plan() and .sync() read the database’s edition once per call to match a declaration with a listed index, and send what the statement declares. Firestore answers what it refuses with INVALID_ARGUMENT. On Enterprise:

• a composite index of one field is how a single field is indexed. Firestore accepts one there, ordered, .array_contains() or __name__, while a Standard edition database refuses it;
• .field_overrides() has nothing to override. An .exempt() is never planned, since Firestore stores nothing for it. Firestore refuses an override that lists indexes;
• .ttl() enables a TTL field as on Standard, and .prune_undeclared() disables an undeclared one the same way;
• Firestore appends no __name__ field to an index, so [a, b] and [a, b, __name__] are two different indexes, and a statement can declare both;
• an index can be unique, refusing a write that would give two documents the same entry, and dense or sparse, holding an entry for every document or only for the ones that have the indexed fields. .unique() and .density(..) set these, as the sections after the example describe.

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .indexes()
    .collection_group("users")
    .composite(|i| {
        i.indexes([
            i.index([i.field("email").asc()]).unique(),
            i.index([i.field("country").asc()]),
            i.index([i.field("country").asc(), i.field("nickname").asc()])
                .density(FirestoreIndexDensity::SparseAny),
        ])
    })
    .ttl(["expires_at"])
    .sync()
    .await?;
Ok(())
}

Without an index that covers it, a Core query on Enterprise runs as a table scan, where Standard fails with FAILED_PRECONDITION, see Query plans and cost.

Unique indexes

A unique index refuses a write that would give two documents the same entry in it. Firestore answers a create, an update and a commit alike with ALREADY_EXISTS (“Duplicate values are found for unique index.”), and the write returns DataConflictError. A unique index of one field is how a value such as an email address is kept unique across a collection.

Density

The density of an index says which documents it holds an entry for. A dense index holds one for every document, with a missing field stored as unset; a sparse index leaves out the documents that lack the indexed fields. Enterprise builds an index dense by default, and a text or geo search index as FirestoreIndexDensity::SparseAny, an entry for each document that has at least one of the fields. Standard builds every index as SparseAll, an entry only for a document that has all of them. Firestore answers a density its edition does not build with INVALID_ARGUMENT.

A declaration that sets a density matches only a listed index of that density. A declaration without one matches a listed index over the same fields, scope and unique flag whatever its density, and when several listed indexes match, the library treats it as it does any declaration that matches several. Two indexes over the same fields are different indexes when one is unique and the other is not.

Bulk delete

Firestore’s BulkDeleteDocuments operation removes every document in one or more collection groups in the background, without you reading or writing each document yourself. The library exposes it through db.fluent().delete().bulk(), behind the same admin feature index management uses; see Enabling the admin feature for turning it on. Google’s own documentation for the underlying operation is at https://cloud.google.com/firestore/docs/manage-data/bulk-delete.

Why the builder takes collection groups

.bulk() takes .collection_groups(...), and there is no .parent(): Google’s BulkDeleteDocuments deletes every document in every collection with the given ID, at any depth, across the whole database, and Firestore offers no parent-scoped bulk delete. Naming a collection group called orders deletes every orders collection under every document in the database, including ones your code never queries.

What survives a bulk delete

• documents written after the operation starts processing;
• any collection whose ID does not match a named group, even nested under a deleted one.

Background and non-transactional

Firestore picks the documents to delete as of one snapshot time, which the result reports as snapshot_time. The deletes themselves are not one transaction, so other writes can interleave with them while the operation runs. execute() returns once the operation is requested unless you ask it to wait; either way the deletion itself continues on Firestore’s side, independent of your process.

Declaring a bulk delete

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let result = db
    .fluent()
    .delete()
    .bulk()
    .collection_groups(["orders"])
    .execute()
    .await?;
println!("{result}");
Ok(())
}

.collection_groups(...) replaces whatever an earlier call in the same chain set, so call it once with every group you want removed.

The empty-list and duplicate-group rules

An empty list is rejected before anything is sent: Firestore reads an empty collection_ids as “the whole database”, and the library refuses to send that on your behalf. Naming the same group twice is rejected too. Both checks run at .execute(), alongside FirestoreCollectionId’s own validation of each name, and return InvalidParametersError naming what is wrong, without making a request.

Waiting for the operation and reading the result

By default execute() returns as soon as Firestore accepts the operation. Add .wait_until_done(timeout) to poll until it reaches a terminal state instead, or .wait_until_done_with_options(...) for control over the poll interval too:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let result = db
    .fluent()
    .delete()
    .bulk()
    .collection_groups(["orders"])
    .wait_until_done(Duration::from_secs(300))
    .execute()
    .await?;
println!("{result}");
Ok(())
}

The returned FirestoreBulkDeleteResult carries the operation name, the collection groups you asked for, and the elapsed time, plus, once you wait for it, the snapshot time Firestore used to decide what to delete and document and byte progress; without .wait_until_done(...), the snapshot time and progress stay unset and print as “progress not reported”, never as 0/0. It implements Display, so printing or logging it directly works with or without a tracing subscriber.

The timeout works the same way as for index management’s .wait_until_ready(...): the last poll goes out at the deadline and may take up to 10 seconds to answer, or the whole timeout if that is shorter.

A failed wait leaves the operation running on Firestore’s side; the library logs the result built from what was polled so far at warn before returning the error, the same as index management’s .sync() does for its own wait.

Logs and spans

A bulk delete logs through tracing, one span covering the whole call (Firestore Bulk Delete), nesting a Firestore Bulk Delete Wait child span when you wait. An excerpt of the log:

Started a bulk delete. operation=".../operations/CyA0..." action="bulk_delete" collection_groups="firestore-rs-bulk-delete-test"
Firestore reported the bulk delete's snapshot time. snapshot_time=2026-09-26T19:50:00Z
Operation reached a terminal state. action="bulk_delete" polls=3

The third line comes from the same wait every long-running admin operation polls through, index sync included, which is why it names the operation generically rather than as a bulk delete.

A bulk delete is an operation that Firestore runs in the background, and it takes time even for a few documents, from about ten seconds to about a minute. The time grows with the data, so set the wait timeout well above that.

Bulk delete works on both editions, and removes the named collection group at every depth.

The Firestore emulator

The emulator answers BulkDeleteDocuments with UNIMPLEMENTED, which a bulk delete returns as DatabaseError.

IAM roles

roles/datastore.bulkAdmin carries datastore.databases.bulkDelete, the permission a bulk delete needs. roles/datastore.owner is a superset and also works.

Running the tests against a real project

tests/admin-bulk-delete-tests.rs writes and bulk-deletes real documents in GCP_PROJECT, so it carries #[ignore] and only runs explicitly:

GCP_PROJECT=your-project cargo test --test admin-bulk-delete-tests --features admin -- --ignored --nocapture

Full example available here.

Command line tool

fsq is a command line client for Firestore built on this crate. It is for working with a database from a terminal: looking at data, trying a SQL statement before it goes into code, and managing indexes.

• a SQL REPL (an interactive prompt) with highlighting, completion of collections and fields, and results as tables, with \x for one block per document and \x full for the path, type and whole value of every field;
• EXPLAIN, which draws the query plan as a tree and warns about a table scan, a read of every document of a collection;
• \l lists collections, with \l --recursive for subcollections, and \d describes the fields of a collection;
• export and import of documents as JSON lines, in the tagged encoding of JSON conversion;
• index plan, index sync and the other index commands, on the firestore.indexes.json file of the Firebase CLI;
• codegen and \gen, which print the fluent API code of a SELECT, as Fluent code for a statement describes;
• --format json and --format jsonl output for scripts.

SQL needs a database of the Enterprise edition, see Standard and Enterprise editions. \l, \d, export, import and the index commands work on both editions.

Installation

fsq has binaries for Linux, macOS, Windows and FreeBSD on its GitHub releases. The install script picks the one for your system.

On macOS, Linux and FreeBSD:

curl -fsSL https://raw.githubusercontent.com/abdolence/firestore-cli-rs/master/install.sh | sh

On Windows (PowerShell):

irm https://raw.githubusercontent.com/abdolence/firestore-cli-rs/master/install.ps1 | iex

The scripts install the latest release, check its SHA-256 digest and put fsq in ~/.local/bin on macOS, Linux and FreeBSD, and in %LOCALAPPDATA%\Programs\fsq on Windows. Running a script again upgrades fsq. FSQ_VERSION and FSQ_INSTALL_DIR choose another release and directory.

With Rust installed, Cargo builds it from source:

cargo install firestore-cli --locked

The crate is firestore-cli and the binary it installs is fsq.

Examples

Connect to a project and a database:

fsq -p my-project -d my-database

Run a statement and print the code of the fluent API for it:

my-project/my-database (enterprise)> SELECT title, rating FROM books WHERE rating >= 4 ORDER BY rating DESC LIMIT 10;
my-project/my-database (enterprise)> \gen

Export a collection and import it again:

fsq -p my-project -d my-database export books -o books.jsonl
fsq -p my-project -d my-database import books.jsonl

Show what changes a firestore.indexes.json file would make, without applying it:

fsq -p my-project index plan firestore.indexes.json

The options and the rest of the commands are in the fsq README.

Google authentication

Looks for credentials in the following places, preferring the first location found:

• A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable.
• A JSON file in a location known to the gcloud command-line tool using gcloud auth application-default login.
• On Google Compute Engine, it fetches credentials from the metadata server.

Local development

Don’t confuse gcloud auth login with gcloud auth application-default login for local development, since the first authorize only gcloud tool to access the Cloud Platform.

The latter obtains user access credentials via a web flow and puts them in the well-known location for Application Default Credentials (ADC). This command is useful when you are developing code that would normally use a service account but need to run the code in a local development environment where it’s easier to provide user credentials. So to work for local development you need to use gcloud auth application-default login.

Crypto provider

Service account keys sign their tokens with a rustls crypto provider. The auth-default-crypto feature, on by default, brings the aws-lc-rs provider of google-cloud-auth. Be aware that aws-lc-rs is built from C sources.

With default-features = false, either enable the feature next to your TLS feature:

[dependencies]
firestore = { version = "0.63", default-features = false, features = ["tls-webpki-roots", "auth-default-crypto"] }

or install a rustls CryptoProvider before creating a client. Without a provider FirestoreDb::new, FirestoreDb::with_options and FirestoreDb::with_options_service_account_key_file return an error with the code CryptoProviderMissing for a service account key. Credentials you build yourself with the google-cloud-auth builders panic on their first token request instead, so build them with GoogleAuthHeaders::from_service_account_key to get the same error.

Firestore emulator

To work with the Google Firestore emulator you can use the environment variable:

export FIRESTORE_EMULATOR_HOST="localhost:8080"

or specify it as an option using FirestoreDb::with_options().

When FIRESTORE_EMULATOR_HOST is set, the library does not look up the Google credentials and uses a stub token instead, since the emulator does not authenticate the requests. This means you do not need any credentials configured to develop against it. Specifying credentials explicitly, with FirestoreDb::with_options_auth() for example, still takes precedence.

The emulator answers bulk delete and index management with UNIMPLEMENTED: .plan(), .sync(), the index calls list(), create(), delete() and update_field(), and the SQL index statements fail with it. SHOW COLLECTIONS works on the emulator.

On the Enterprise edition

The emulator acts as one edition, Standard by default. gcloud starts it, and needs a Java runtime (a JDK) for it:

gcloud emulators firestore start --host-port=127.0.0.1:8080 --edition=enterprise

The emulator prints Database Edition: STANDARD or Database Edition: ENTERPRISE as it starts.

Start the emulator with --edition=enterprise to run pipelines and SQL against it; the default Standard emulator answers them with INVALID_ARGUMENT, and neither edition of the emulator supports bulk delete. Core inserts and queries run on an emulator of either edition, and Core aggregations, transactions and listeners run on the Enterprise one as on a live database.

The Enterprise emulator runs SQL and pipelines, and ignores Query Explain: an EXPLAIN or EXPLAIN ANALYZE statement, and .explain(), return the rows and no plan. Check the plan of a query against a live database, see Explaining a pipeline.

Docker images and TLS

When you design your Dockerfile make sure you either installed Root CA certificates or use base images that already include them. If you don’t have certs installed you usually observe the errors such as:

SystemError(FirestoreSystemError { public: FirestoreErrorPublicGenericDetails { code: "GrpcStatus(tonic::transport::Error(Transport, hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificateData(\"invalid peer certificate: UnknownIssuer\") })))" }, message: "GCloud system error: Tonic/gRPC error: transport error" })

For example for Debian based images, this usually can be fixed using this package:

RUN apt-get install -y ca-certificates

Google Distroless images are a good choice of base image, since they are secure, already include Root CA certs, and are optimised for size.

TLS features

Cargo provides support for different TLS features for dependencies:

• tls-roots: default feature to support native TLS roots
• tls-webpki-roots: feature to switch to webpki crate roots

Request tags

Firestore supports attaching request tags to requests. They are reported by Firestore in its monitoring and billing breakdowns, which makes them useful to attribute reads and writes to a specific feature, tenant or background job.

Tags can be set per operation for queries, aggregations, listings, listeners, and pipelines and SQL statements on the Enterprise edition:

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.fluent()
    .select()
    .from(TEST_COLLECTION_NAME)
    .request_tags(["nightly-report"])
    // or use request_options if you want to provide the options structure directly
    // .request_options(FirestoreRequestOptions::from_tags(["nightly-report"]))
    .obj::<MyTestStructure>()
    .query()
    .await?;
Ok(())
}

Or session wide, for every request issued through a client instance. This is also how you attach tags to the CRUD operations (insert/update/delete/get):

use firestore::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb, my_struct: MyTestStructure) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let tagged_db = db.clone_with_request_tags(["nightly-report"]);

tagged_db
    .fluent()
    .insert()
    .into(TEST_COLLECTION_NAME)
    .document_id(&my_struct.some_id)
    .object(&my_struct)
    .execute::<MyTestStructure>()
    .await?;
Ok(())
}

Transactions and batch writers accept them through their options:

use firestore::*;
use futures::FutureExt;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, Serialize)]
struct MyTestStructure { some_id: String }
const TEST_COLLECTION_NAME: &str = "test";
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
db.run_transaction_with_options(
    |db, _transaction| {
        async move {
            db.fluent()
                .select()
                .by_id_in(TEST_COLLECTION_NAME)
                .obj::<MyTestStructure>()
                .one("test-1")
                .await?;

            Ok(())
        }
        .boxed()
    },
    FirestoreTransactionOptions::new()
        .with_request_options(FirestoreRequestOptions::from_tags(["checkout"])),
)
.await?;
Ok(())
}

A per operation value replaces the session wide default rather than merging with it.

Request timeouts

The library sets no timeout on requests by default. A request waits for Firestore for as long as its connection lives, and a connection to a server that stopped answering is closed only by HTTP/2 keep-alive, after about two minutes.

To limit how long one request waits for Firestore, set request_timeout in the options:

use firestore::*;
use std::time::Duration;
fn config_env_var(name: &str) -> Result<String, String> {
    std::env::var(name).map_err(|e| format!("{name}: {e}"))
}
async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let db = FirestoreDb::with_options(
    FirestoreDbOptions::new(config_env_var("PROJECT_ID")?)
        .with_request_timeout(Duration::from_secs(30)),
)
.await?;
Ok(())
}

What it limits depends on the request:

• unary requests (get, insert, update, delete, list pages, batch writes, transaction commits and the admin API calls): the whole call. The limit is also sent to Firestore as the request deadline;
• streamed reads (queries, aggregations and batch gets): the wait for the stream to start and for each next document, counted only while you are waiting for one, so a slow consumer of a long query is not cut;
• pipeline and SQL reads, on the Enterprise edition: the wait for each response, as for a streamed read;
• pipeline and SQL writes: the wait for each response, for a write that is sent once;
• listeners and the streaming batch writer are not limited, since their streams stay quiet for as long as nothing changes. A listener resets a stream that sent nothing for its own idle_timeout, see Listening to document changes.

To use another limit for a part of your code, or none at all, clone the client with clone_with_request_timeout. Batch writers and transactions created from the clone keep it:

use firestore::*;
use std::time::Duration;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let report_db = db.clone_with_request_timeout(Some(Duration::from_secs(300)));
let unlimited_db = db.clone_with_request_timeout(None);
Ok(())
}

Retries

The limit applies to each attempt. A request that runs out of it fails with a DatabaseError with the code DeadlineExceeded and retry_possible set. Reads and simple batch writes retry it:

• gets, lists, list collection IDs and the start of queries, aggregations, pipelines and SQL reads, up to max_retries times;
• simple batch writes, until retry_max_elapsed_time. A retried batch may apply twice, for example an increment, as it can after a dropped connection.

insert(), update() and delete() return a request timeout that ran out without a retry, see Single writes.

A read makes max_retries + 1 attempts, and the error that comes back after the last one still has retry_possible set. A pipeline or SQL write makes one attempt and clears the flag, see Retries and listeners. A pipeline that a cache session answers sends no request, so no timeout or retry applies to it.

Batch gets, partition queries, a query or aggregation stream that has already started and transaction commits return the error without a retry. A transaction commit is never retried, because its writes may already be applied.

The reads in the first item also retry a DEADLINE_EXCEEDED that Firestore itself answers with, up to max_retries times, with or without a request timeout set. Inserts, updates, deletes, batch gets, partition queries and streams that have already started return it with retry_possible unset.

Single writes

insert(), update() and delete() run with .execute() are sent again up to max_retries times after RESOURCE_EXHAUSTED and UNAVAILABLE. A connection lost while a request is in flight is retried like UNAVAILABLE.

These are the codes that Google’s Firestore server libraries for Java, Go and Python retry a write on. They send every single write, a delete included, as a commit, so one list covers all three. The Node.js library also retries ABORTED on a commit. The library does not, see When a write fails.

Any other code is returned at once, whatever retry_possible says: INTERNAL, ABORTED and DEADLINE_EXCEEDED included, the request timeout running out among them.

A write that is sent again can find that the first attempt was applied, since UNAVAILABLE and a lost connection do not say whether it was. Then:

• an insert() fails with ALREADY_EXISTS when the first attempt created the document;
• an update() or delete() with an UpdateTime precondition fails with FAILED_PRECONDITION, since the document no longer has that update time. The write went through, and the error reports that it did not;
• an update() with FirestoreWritePrecondition::Exists(false) fails with ALREADY_EXISTS when the first attempt created the document;
• a delete() with FirestoreWritePrecondition::Exists(true) fails with NOT_FOUND when the first attempt deleted the document;
• a delete() without a precondition finds the document already gone, which is not an error;
• an update() with transformations applies them again: an increment adds its value twice. Do not use such an update where a repeated increment is wrong; run it in a transaction or set max_retries to 0.

Writes queued with add_to_transaction or add_to_batch, transaction commits, and pipeline and SQL writes are not retried this way.

Be aware not to set retry_max_elapsed_time to None on a simple batch writer when you set a request timeout: against a Firestore that stopped answering, it then retries forever, one timeout per attempt. The 60 second default is checked between attempts, so the last attempt can still run for a whole timeout past it.

Retry limits

Reads, single writes, simple batch writes and transactions wait the same delays between retries: 500 ms, growing by 1.5 times for each next retry, each delay randomised by up to a third either way. They differ in when they stop:

• reads and single writes stop after max_retries of FirestoreDbOptions, 3 by default, with no time limit;
• simple batch writes stop after retry_max_elapsed_time of FirestoreSimpleBatchWriteOptions, 60 seconds by default, and ignore max_retries;
• transactions stop after max_retries of FirestoreTransactionOptions, 4 by default, and, when set, after retry_max_elapsed_time, which is off by default.

Testing your own code

The Fluent API is built on FirestoreDb directly, so there is no built-in way to substitute a fake database into db.fluent(). There are two approaches that work well.

Abstract at your own boundary

Define a trait for what your code needs and implement it over a type holding a FirestoreDb. Your business logic then depends on your trait, and tests provide their own implementation. This keeps Firestore concerns in one place and needs nothing special from this library:

use firestore::*;
use firestore::errors::FirestoreError;
#[derive(Debug, serde::Deserialize)]
struct User;
#[derive(Debug)]
struct MyError;
impl From<FirestoreError> for MyError {
    fn from(_: FirestoreError) -> Self {
        MyError
    }
}
#[async_trait::async_trait]
trait UserRepository {
    async fn find_user(&self, id: &str) -> Result<Option<User>, MyError>;
}

struct FirestoreUserRepository {
    db: FirestoreDb,
}

#[async_trait::async_trait]
impl UserRepository for FirestoreUserRepository {
    async fn find_user(&self, id: &str) -> Result<Option<User>, MyError> {
        Ok(self
            .db
            .fluent()
            .select()
            .by_id_in("users")
            .obj()
            .one(id)
            .await?)
    }
}

// In tests, implement `UserRepository` with an in-memory HashMap.

Run against the Firestore emulator

For tests that should exercise real query, listener and transaction behaviour, point the library at the Firestore emulator:

export FIRESTORE_EMULATOR_HOST="localhost:8080"

No credentials are needed in that mode. This is how the caching and transaction behaviour of this library itself is verified, and it catches things a hand-written fake cannot.

Code that runs pipelines or SQL needs an emulator started in the Enterprise edition, see Firestore emulator.

The low level *Support traits are not public, and the builders’ constructors are crate private, so code cannot be generic over them. The migration guide lists the fluent replacement of every removed method.