JSON conversion

With the json-conversion cargo feature, associated functions of FirestoreDb convert Firestore values and documents to JSON and back. It is meant for tools that move documents out of Firestore and into it again, such as an export and an import, and for tools that show documents to a person. It does not send anything.

Like FirestoreDb::serialize_to_doc, the functions take their context explicitly, so they need no client. The first argument is the documents root of the database, projects/{project}/databases/{database}/documents, which db.get_documents_path() returns.

[dependencies]
firestore = { version = "...", features = ["json-conversion"] }
serde_json = "1"

The functions write two encodings, chosen with FirestoreJsonEncoding:

• Tagged, for export and import. It reads back to the same Firestore value, without loss. A type that JSON lacks becomes a one-key object, a wrapper, whose key starts with $;
• Plain, for display. It is the most readable JSON for each type, and it has no way back: a timestamp, a reference and a string look alike.

A value is a gcloud_sdk::google::firestore::v1::Value, the type that FirestoreDocument::fields holds, so a field converts without a copy. FirestoreValue::value is one.

FirestoreJsonOptions holds the encoding of a document and whether it carries its metadata. Both are set through its builder, and the defaults are the tagged encoding and no metadata:

use firestore::*;
let options = FirestoreJsonOptions::new()
    .with_encoding(FirestoreJsonEncoding::Plain)
    .with_metadata(true);

The plain encoding of a value never fails; the tagged one refuses the values that exist only in pipelines, see below. The metadata option means nothing for a single value, so serialize_value_to_json takes the encoding alone.

Write a line of an export with serde_json::to_string(&json) and read one with serde_json::from_str::<serde_json::Value>(..). The deserialising functions read the tagged encoding. The plain encoding parses too, but a timestamp and a reference come back as strings, since the plain encoding does not tell them from a string; nothing prevents reading one.

Do not turn the JSON into a FirestoreValue with .into(). The conversion serialises the JSON as data, so {"$timestamp": "2023-11-14T22:13:20Z"} is stored as a map holding a string. A tagged value goes back to a value through FirestoreDb::deserialize_value_from_json(..).

Export and import

use firestore::*;
async fn example(db: FirestoreDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let documents_path = db.get_documents_path();
let options = FirestoreJsonOptions::new();

let Some(document) = db.fluent().select().by_id_in("books").one("dune").await? else {
    return Ok(());
};
let json = FirestoreDb::serialize_doc_to_json(documents_path, &document, &options)?;
let line = serde_json::to_string(&json)?;

let json = serde_json::from_str(&line)?;
let document = FirestoreDb::deserialize_doc_from_json(documents_path, json)?;
db.fluent()
    .update()
    .in_col("books")
    .document(document)
    .execute()
    .await?;
Ok(())
}

An imported document carries its name, so write it with update(), or with a batch writer. insert().document(..) refuses a document whose name is set.

Encodings of each type

• Bytes are base64 with the standard alphabet and padding;
• a map is escaped as {"$map": {..}} only in the tagged encoding, and only when its single key starts with $, so that it is not read as a wrapper. A map with no key or several is never escaped;
• map keys are written in byte order. Whether a JSON reader keeps that order depends on the preserve_order feature of serde_json; the format does not depend on it;
• a Value with no type set is written as null in both encodings.

Documents

A document is one flat object holding __name__ and the fields.

{"__name__": "books/dune", "published": {"$timestamp": "2023-11-14T22:13:20Z"}, "title": "Dune"}

The keys follow the order of serde_json’s map. By default it is sorted, so a field named Title is written before __name__. With the preserve_order feature of serde_json, which the json-conversion feature does not enable, the order is __name__, the metadata, then the fields in byte order.

FirestoreJsonOptions::new().with_metadata(true) adds __create_time__ and __update_time__ when the document carries them: wrapped as $timestamp in the tagged encoding, as strings in the plain one. deserialize_doc_from_json ignores both, so the document it returns has neither.

__name__ is left out of a document without a name, such as an aggregation or a pipeline select result. A top-level field whose name starts with $ needs no escape, since the top level of a document is always read as fields.

Firestore refuses __name__, __create_time__, __update_time__ and every other name between double underscores as a stored field, so these keys cannot collide with a stored document’s fields. The functions do not check for them. A pipeline result with a field of such a name overwrites the key of the name in the object.

References and portability

A reference and __name__ are written relative to the database of documents_path when they point into it, and as a full name otherwise: projects/latestbit/databases/other/documents/books/dune. relative_document_name gives the same form for any name. A relative name resolves against the database of the documents_path that reads it, so an export from one database imports into another, and a full name keeps pointing where it pointed.

A name is read as a full name when it has the form projects/{project}/databases/{database}/documents/{path} with an odd number of segments. A relative document path has an even number of segments, so a document in a collection called projects is not mistaken for another project’s.

Integers and doubles

Firestore integers are 64-bit. A JSON number that a JavaScript reader parses as an IEEE double holds integers exactly only up to 2^53 − 1, so the tagged encoding writes anything beyond as $int with the decimal text. 2^53 itself goes to $int, since a JavaScript reader cannot tell it from 2^53 + 1. The plain encoding writes every integer as a number.

A double is written with a . or an exponent, 3.0 and 1e+300, so it reads back as a double and not as an integer. A coordinate of a geo point follows the same rule.

Exact doubles and float_roundtrip

serde_json parses a float one ULP (unit in the last place, the gap between a double and its neighbour) off for about one in six values with many digits, such as a vector component widened from f32. The text then reads back as the neighbouring double. A float round trip, a double written to text and parsed again to the same double, needs the float_roundtrip feature of serde_json.

The json-conversion feature turns it on. Cargo builds one serde_json per program with the features of every crate that uses it, so a program that enables json-conversion parses every float exactly, at the cost of a slower float parse.

Timestamps

Timestamps are written as RFC 3339 in UTC with a Z, with 0, 3, 6 or 9 fraction digits: 2023-11-14T22:13:20Z, 2023-11-14T22:13:20.500Z, 2023-11-14T22:13:20.123456Z. The range is the whole range Firestore accepts, from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999Z. FirestoreInstant ends 26 hours earlier, so the functions do not go through it.

Reading accepts an offset (+01:00), a lowercase t and z, and a space for T, and keeps nanoseconds. Firestore stores microseconds and cuts the digits beyond; the functions leave that to Firestore. FirestoreTimestamp::truncated_to_firestore_precision cuts them before a write.

Values that exist only in pipelines

Field references, variable references, functions and pipelines are parts of a pipeline request, and a stored document never holds one.

• The tagged encoding refuses them with JSON_EXPRESSION_VALUE, since they cannot be read back;
• the plain encoding shows what the caller holds: the text of a field or variable reference, the name of a function, and "pipeline".

Reading the tagged encoding

A wrapper takes exactly one form of content:

A component of $geo and $vector may be an integer literal, so {"$vector": [1, 2]} reads as two doubles.

Only the form of a wrapper is checked. An empty $vector, a latitude of 91 and a timestamp in the year 10000 read without an error, and Firestore refuses them on write. A timestamp in the year 10000 reads only in the signed form the writer produces, +10000-01-01T00:00:00Z; without the sign it is a JSON_INVALID_WRAPPER error.

A number above u64 in hand-written input reads as a double.

Errors

A failed conversion is a DeserializeError when it reads JSON and a SerializeError when it writes it. The code is in public.code, a constant of firestore::errors::codes, and the message names the JSON path of the value, such as `reviews[2].at`. A document error carries the full document name in document_path.