If you have been importing external content into the DocPad in-memory database, or exporting DocPad’s in-memory content into an external database, such functionality is now deprecated.
As of today, the goal of “importing external content into the DocPad database” as well as “exporting DocPad database content to external databases”, such as the way the tumblr plugin does for importers and the sync plugin does for importers and exporters, will now be considered a failed experiment and such functionality should be marked as deprecated.
This means a lot of work in the 2013 calendar year was a complete waste of productivity, with the only benefit to realise in hindsight how truly bad such an idea actually is.
DocPad like all file-based static-site-generators that I am aware of, takes files and injects them into an in-memory representation. Naturally obvious today (it surprisingly not obvious then, due to a lack of clarity about static-site-generators at the time), if one requires to inject content into such a system, it should be done by injecting files that are then read and parsed, rather than injecting directly into the in-memory database — at least it is clear now.
Let’s go over the differences in approaches:
Injecting into In-Memory Database
- Pros:
- No one is tempted to manually modify an automatic imported document, as there is no ability to do so
- Cons:
- DocPad specific, doesn’t work with other file-based systems
- No ability to edit documents, useful if all you wanted was a one off import
- Content needs to be re-fetched, parsed, and injected each and every time DocPad starts up, as in-memory database is naturally lost - there was even investment in DocPad to write the in-memory database to a serialised format on exit then load it back up on startup
Injecting into Files
- Pros:
- Can be done as generic modules that are then utilised by each SSG, removing lockin and improving collaboration and value between SSGs
- No complexity needed, already the main purpose of the system
- Content can be modified in the imported file format
- Content can be only fetched once, removing the largest time burden with importing external content
- Content can also be fetched multiple times when wanting to refresh the content, if a file was manually modified, the remote last edit time is greater than the local last edit time, the file can be updated, but if the file has since been modified, a conflict workflow can be done
- Cons:
- There really isn’t any…
All in all, I’m very surprised, that the year or so of work that was done through Bevry and other companies in making in-memory importers a thing, was ever done and made it so far - when the above is so clear. However, of course, all of this was cutting edge then, and none of it as clearly defined as it is now, due to the newness of SSG scene and whatnot. Hindsight is awesome.
The was a main driver for the in-memory method, and that was to allow remote deployments where clients could edit the content. However, that still required the external content source to the source of truth as in remote land the file system is ephemeral just like the in-memory database, so the plan was to write to the in-memory database, then save to changes to the file system and the import location. Meaning that documents have sources (that its content was imported from) and targets (when its content is saved to). However, for the multitude of complexity that was researched over the years of 2013, 2014, 2015. All the research and experimentation and time over the years can be summed up like so: if you want an external database to be imported and to be the source of truth, interact with the external database directly, not via a proxy of a particular system. That is the only fast way of doing it. It is the only simple way of doing it too.
Innovation is risky, and sometimes it works, but not always. This was a time the risk didn’t pay off. It is indeed a pity that so much time over those years went into such a failed experiment, rather than accomplishing things strictly performance related. However, we all know now. For the better of DocPad, and the SSG scene in general.
To summarise / TLDR:
- The functionality of “importing external content into the DocPad database” is deprecated and no longer encouraged nor supported
- The functionality of “exporting DocPad database content to external databases” is deprecated and no longer encouraged nor supported
- Importers and exporters are still possible, however they should just interact with the file system exclusively, rather than with DocPad’s in-memory database.
- Some plugins like the tumblr plugin will need to move over, fortunately our project chainyjs makes this an incredibly easy thing to do today
- Some plugins like the sync plugin are done for, and the reality needs to be accepted, if you needed such functionality you should heed the learnings discussed in this post.
With all that being said, the learnings from all this research actually opens a door to a new world of external-database exclusive static site generators, which will be superior in terms of performance and abilities than the entire generation of current in-memory database focused static site generators which are superior only in terms of ease of use. This is an area that I’m interested and evidently experienced in, so if your company requires such a thing, reach out.
Cheers.