DocPad Architecture Vision (2015+)

Bevry Software
, ,
1

Currently, DocPad is doing too many things:

  1. It is a static site generator
  2. It is a dynamic web server
  3. It is a document parser
  4. It is a document database
  5. It is a document query system
  6. It is a rendering system
  7. It is a plugin environment
  8. Including a home for content management plugins
  9. Including a home for content importers

That’s the reason why we’ve struggled defining DocPad for so long. It’s time we change this. The biggest change here is to revaluate our goals. For 2013 our goals were:

  1. Become a plugin environment for importers and content management systems/interfaces
  2. Create a REST API plugin so that content management systems can then interact with docpad which can then interact with the database which can then interact with the file system
  3. However, if we are on a remote environment, we also got to create a Commit Plugin that will also listen to the changes, and commit any changes to a git repo, then push that git repo to a web server - and hope that the web server isn’t on an ephemeral file system and doesn’t reset between changes and their push.
  4. Somehow develop content exporters so we can export content to other database systems (including the file system)

This is absurd. It means that DocPad also becomes an ORM on top of all that it was before.

It really should have been this:

  1. We utilise a popular external database system
  2. We develop a standard for the schema that DocPad will use, other SSGs will use, and Content Plugins can use
  3. DocPad then queries the database for changed documents and re-renders them, applying the renderings to the database

That way:

  1. Content Management Systems can be developed for the content without depending on DocPad
  2. Importers can be written without depending on DocPad
  3. Exporters can be written without depending on DocPad

This means there is the following pieces:

  1. There is a blackboxed utility for watching the filesystem and importing it into the database
  2. There is a blackboxed utility for generation, that comprises of:
  3. There is a blackboxed utility for querying the database
  4. There is a blackboxed utility for rendering the results of the query
  5. There is a blackboxed utility for writing the rendered results back to the database
  6. There is a blackboxed utility for writing the rendered results to the file system when desired
  7. There is a blackboxed utility for creating a web server on the database
  8. There is a blackboxed utility for reading the configuration
  9. There is a blackboxed utility for creating an event system
  10. There are blackboxed utilities for regenerating changes:
  11. There is a blackboxed utility for receiving a notification, updating the database, and telling the generation utility to generate
  12. There is a blackboxed utility for watching the database,and on changes, regenerating the changes
  13. There is a blackboxed utility for adding a web admin control centre to the above system, instead of just a CLI
  14. Tying these individual blackboxed utilities together, will create the dynamic static-site generator we have come to love

Then outside of these core, there will be:

  1. There is a blackboxed content management system(s) that interacts with the database directly
  2. There are blackboxed importer scripts for a whole bunch of different services that read in a feed, and convert it into the database format, then write it to the database

This makes a lot more sense, as it means rather than a highly coupled system, instead DocPad is comprised of tiny little independent modules (aka blackboxed modules).

While we can utilise a lot of the existing code for this, it will be breaking changes all over the place probably.

Considerations:

  1. When the database system is asynchronous, how do we do templating? Perhaps ES6 will have a solution here? Or we could use the way the partials plugin does it, which is inserting a place holder into the template synchronously that is then replaced with the actual data when it arrives asynchronously.
  2. This will be 1-2 years of work before we have something stable for everyday use. That’s a lot of funding and a lot of time. We need a guarantee that the investment of our time is going to be worthwhile - and that we can still afford to thrive during that time (not just live…).

Thoughts?

This proposal will deprecate the following topics/issues:

This proposal will close the following topics/issues:

  • Stream Content - depending on how the async templating is solved, this may or may not be useful
  • Forget Content - this is a half-way compromise of async data retrieval of templating - it lazily removes unused content from memory, whereas with this proposal unused content would greedily be removed asap - issue is still useful
2

Could the file-based query system be abstracted as a plugin option? I like the idea for smaller projects and it’s very novel.

Would DB options be implemented as plugins? :1234:

3

The file system utility would inject its data directly into the database if needed.

The query utility would interact with the database, not the file system.

The model utility would interact directly with the database, not the file system.

Consider the database to be mongodb at this point in time.

4

Would static deploys to GitHub Pages still work?, ie, is the database just local-dev in that case?

5

Yeah. They would be handled by the data as to file system utility. I’ll see if I can do up a sketch sometime of how it all comes together.

6

Just did this email to maxogden and domonictarr about the possibility of a distributed streams based ssg: https://github.com/maxogden/messages/issues/49

It is something to consider for a end of 2016 timeframe, which hopefully we can mould DocPad (or at least take the lessons of DocPad over 2015) to become the renderer plugin in such a system.