Working with Firebase Documents in ClojureScript

In a project I’m currently working on we’re making use of Google's Firebase to store domain data and run cloud functions.

In Firestore, which is Firebase’s database offering, every document is essentially a Javascript object. While interop in ClojureScript is pretty good we ended up converting the raw data of these documents to ClojureScript data structures using js->clj. This also meant we’d need to convert them back to JS objects before writing them to Firestore.

Because IDs are technically not part of the document the project adopted a pattern of representing documents as tuples:

[id (js->clj firestore-data)]

This works but isn’t particularly extensible. What if we also wanted to retain the “Firestore Reference” specifying a documents location inside the database? (Firestore stores data in a tree-like structure.)

It also leads to some funky gymnastics when working with collections of documents:

(sort-by (comp :join_dt second) list-of-document-tuples)

Could be worse... but also could be better.

This blogpost will compare various approaches approach to address the problems above using cljs-bean, basic ClojureScript data structures, custom protocols and :extend-via-metadata.

cljs-bean

With the recent release of cljs-bean we have an interesting alternative to js->clj. Instead of eagerly walking the structure and converting all values to their ClojureScript counterparts (i.e. persistent data structures) the original object is wrapped in a thin layer that allows us to use it as if it were a ClojureScript-native data structure:

(require '[cljs-bean.core :as cljs-bean])

(-> (cljs-bean/bean #js {"some_data" 1, :b 2})
    (get :some_data)) ; => 1

Given a Firestore QueryDocumentSnapshot we can make the JS object representing the data easily accessible from ClojureScript:

(-> (cljs-bean/->clj (.data query-document-snapshot))
    (get :some_field))

;; (cljs-bean/->clj data) is roughly the same as
;; (cljs-bean/bean data :recursive true)

The bean is immutable and can be used in client side app-state as if it is one of ClojureScript’s persistent data structures.

Caveat: Updating a bean using assoc or similar will create a copy of the object (Copy-on-Write). This is less performant and more GC intensive than with persistent data structures. Given that the data is usually quite small and that the document representations in our app state mostly aren’t written to directly this is probably ok (cljs-bean #72).

Whenever we want to use the raw object to update data in Firestore we can simply call ->js on the bean. Conveniently this will fall back to clj->js when called on ClojureScript data structures.

(.set some-ref (cljs-bean/->js our-bean))

Arguably the differences to using plain clj->js aren’t monumental but working with a database representing data as JS objects it is nice to retain those original objects.

Integrating Firestore Metadata

Now we got beans. But they still don’t contain the document ID or reference. In most places we don’t care about a documents ID or reference. So how could we enable the code below while retaining ID and reference?

(sort-by :join_dt participants)

Let’s compare the various options we have.

Tuples and Nesting

I already described the tuple-based approach above. Another, similar, approach achieves the same by nesting the data in another map. Both fall short on the requirement to make document fields directly accessible.

;; structure
{:id "some-id", :ref "/events/some-id", :data document-data}
;; usage (including gymnastics)
(sort-by (comp :join_dt :data) participants)

I’m not too fond of either approach since they both expose a specific implementation detail, that the actual document data is nested, at the call site. In a way my critique of this approach is similar to why Eric Normand advocated for getters in his IN/Clojure ’19 talk — as far as I understand anyways.

Addition of a Special Key

Another approach could be to add metadata directly to the document data.

(defn doc [query-doc-snapshot]
  (-> (cljs-bean/->clj (.data query-doc-snapshot))
      (assoc ::meta {:id (.-id query-doc-snapshot
                     :ref (.-ref query-doc-snapshot})))

This is reasonable and makes document fields directly accessible. However it also requires us to separate document fields and metadata before passing the data to any function writing to Firestore.

;; before writing we need to remove ::meta
(.set some-ref (cljs-bean/->js (dissoc document-data ::meta))

I think this is a reasonable solution that improves upon some of the issues with the tuple and nesting approach. I realize that this isn’t a huge change but this inversion of how things are nested does give us that direct field access that the nesting approach did not.

Protocols and :extend-via-metadata

An approach I’ve found particularly interesting to play with makes use of a protocol that can be implemented via metadata, as enabled by the new :extend-via-metadata option. This capability was added in Clojure 1.10 and subsequently added to ClojureScript with the 1.10.516 release:

(defprotocol IFirestoreDocument
  :extend-via-metadata true
  (id [_] "Return the ID (string) of this document")
  (ref [_] "Return the Firestore Reference object"))

(defn doc [query-doc-snapshot]
  (with-meta
    (cljs-bean/->clj (.data query-doc-snapshot))
    {`id (fn [_] (.-id query-doc-snapshot))
     `ref (fn [_] (.-ref query-doc-snapshot))}))

Using with-meta we extend a specific instance of a bean to implement the IFirestoreDocument protocol. This allows direct access to document properties while retaining important metadata:

(:name participant) ; => "Martin"
(firebase/id participant) ; => "some-firebase-id"

At call sites we use a well-defined API (defined by the protocol) instead of reaching into nested maps whose structure may need to change as our program evolves. This arguably could also be achieved with plain functions.

Sidenote: A previous iteration of this used specify!. Specify modifies the bean instance however, meaning that whenever we’d update a bean the protocol implementation got lost. In contrast metadata is carried over across updates.

Summary

Using cljs-bean we’ve enabled idiomatic property access for JS data structures without walking the entire document and converting it to a persistent data structure. We also retain the original Javascript object making it easy to use for Firestore API calls.

We’ve compared different ways of attaching additional metadata to those documents using compound structures as well as the new and shiny :extend-via-metadata. Using it we’ve extended instances of beans to support a custom protocol allowing open ended extension without hindering the ergonomics of direct property access.

While I really enjoyed figuring out how to extend beans using :extend-via-metadata it turned out that any approach storing data in “unusual places” (i.e. metadata) causes notable complexity when also wanting to serialize the data.

Serializing metadata is something that has been added to Transit quite some time ago but compared to the plug and play serialization we get when working with plain maps it did not seem worth it. Even if set up properly the protocol implementations, which are functions, are impossible to serialize.

Ultimately we ended up with plain beans and storing metadata under a well known key that is removed before writing the data to Firestore again:

(defn doc [query-doc-snapshot]
  (-> (cljs-bean/->clj (.data query-doc-snapshot))
      (assoc ::meta {:id (.-id query-doc-snapshot)
                     :ref (.-ref query-doc-snapshot)})))

(defn id [doc]
  (-> doc ::meta :id))

(defn ref [doc]
  (-> doc ::meta :ref))

(defn data [doc]
  (cljs-bean/->js (dissoc doc ::meta)))

If you're using Firebase or comparable systems, I'd be curious to hear if you do something similar on ClojureVerse.

Thanks to Matt Huebert and Mike Fikes for their feedback & ideas.