On one of those crazy programmer journeys

Yesterday I went on one of those crazy programmer journeys where I worked for 8-9 hours more or less non-stop. In the morning I had a discussion about all sorts of tecnical site-building-things with the crew over at Routify and this got me inspired to try some new things.

A better understanding of internals

I had been working on Bedrock for a while now, with a lof of work done in June, adding all kinds of stuff (PostCSS + PurgeCSS support for Tailwind purposes, a separate config for production etc.). This made me quite familiar with the Bedrock codebase.

(For those who don’t know what Bedrock is, it is a static site generator designed to help coding designers get up extensive HTML prototypes with hundreds of pages with surrounding styleguide and component document. We use it at Mono to prototype web apps at scale. It has been maintained as an MIT-licensed codebase for the past 6 years.)

However there were still a few parts of the code that were a bit of a mystery to me. There’s server.js which runs an Express server to get your templates on demand while working in Bedrock (avoiding static rebuilds, the way Jekyll used to work) and templates.js which builds the templates when you build a static site. Essentially they do the same, they generate the site, but one of them is at runtime based on a web request and another is when you do a static build. We will get back to this.

The site that you are reading this exact blog post on runs on WordPress (unless you are using a feed reader in which I salute you). I made this choice mostly because Bedrock never supported blogging in the first place.

The project: convert bedrockapp.org from WordPress to Bedrock itself

Now, because of the new features in Bedrock, I thought I might be able convert the Bedrock website to make it run using Bedrock. After all, since Bedrock is supposedly ready for production; why not show what we can do?

I also had reasons to dig into Express. I want to have more control for multi language setups. I want to know everything that is going on technically so I can fix problems if they arise.

So I set out in the afternoon to build new functionalities with the main focus being converting the whole website from a WordPress-driven website to one that runs on Bedrock.

For context… I am a coding designer

The initial code for Bedrock was written over 6 years ago by ex-colleague Thomas and over the years I’ve mostly maintained it, added new things but the core of it has mostly stayed the same. Thomas helped us to modernize a few things a couple of years ago too to make sure we were ready to use the next generation of JS inside of Bedrock.

I am not a programmer in my day-to-day job; but I am slowly but surely learning things. This blog post is mostly for my own reference and I doubt anyone will get through the whole thing. I like to write these kinds of things down because I know it will help me in the future.

Getting started

The first thing I did was use the this docs page’s instructions to get a recent version of Bedrock:

npx degit usebedrock/bedrock#canary bedrock-site-2

Then I looked into the WordPress repo for this website. I copied over images and SCSS and started converting the homepage. I used html2pug directly inside of my favorite editor TextMate using Filter through command to quickly convert HTML into Pug.

I got the homepage up in seconds and everything went smooth. The SCSS pipeline was similar as the WordPress site and the homepage is more or less static.

On the homepage we have a dark hero area which doesn’t exist on the other pages. So there’s a bit of code I want to post here where we are checking if we are on the homepage or if we’re not.

if !pathname || pathname.match(/^index$/)
    img(src='/images/logo-white.svg')
else
    img(src='/images/logo-black.svg')
.u-sr-accessible Bedrock

Bedrock exposes the pathname on the template level which helps to make the page react to the current URL. I realize this variable is not that logically named. Above code can be handy; for selected states please refer to practical tips.

Rebuilding the documentation part

I then got to the documentation part. I figured I wanted to keep my documentation in markdown files. Currently the docs consist out of 15-ish pages. They are categorized in the sidebar but technically they are not categorized and really have the same structure:

/documentation/page-name

In Bedrock I created the markdown files in content/templates/documentation along with a documentation template, that would use marked to process the content:

extends /templates/_layouts/default

block content

    .c-container-outer
         .c-container-inner
                .c-cols
                     .c-col-4
                         include /templates/_includes/side-nav
                     .c-col-12
                            .c-content.c-sheet!=marked(doc.body)

Note how we are using != in Pug to render an unescaped string (see Interpolation).

Now, to make the connection between the pages I first set up that on a server level, if you would hit the /documentation/ part of the website with something behind that URL e.g. /documentation/my-page a different behavior would occur.

I did this in the aforementioned server.js:

  app.get('/documentation/:doc', function (req, res) {
    const docFilename = req.params.doc;
    const doc = _.find(contentDocs.discover().allContentDocs, doc => doc.attributes.filename === docFilename);
    renderView(req, res, 'documentation/_template/doc', {
      pathname: path.join('documentation/docs/', docFilename),
      doc
    });
  });

Essentially this is a customized copy of code that already existed for visiting a styleguide component. There’s a lot going on here.

I want to zoom in on the line that says:

const doc = _.find(contentDocs.discover().allContentDocs, doc => doc.attributes.filename === docFilename);

This line essentially uses a new file called content-docs.js (#) which exports a discover() function, which essentially looks in the right path (using this a path defined in paths.js) to find the aforementioned markdown files.

The code in content-docs.js processes the markdown files using a package called front-matter. Essentially we extract the content of the markdown file to a JS object along with anything contained in the front-matter. The resulting JS object looks a bit like this:

{
  body: "My documentation page",
  attributes: {
     title: "Title"
  }
}

This can then be used to build out the docs page. I’ll repeat the final line in the template here:

.c-content.c-sheet!=marked(doc.body)

Do you see how the doc const from above circles around to the template?

In another part of already shown code above we are referencing a template documentation/_template/doc . This is in a folder with an _ and will get ignored when doing a static build.

So the documentation worked, but only when running Bedrock using npm start (where the Express server is used).

What I had to do now was also make sure that the documentation part got generated statically when running npm run build.

In general, we are using Gulp as a JS task runner. When you start up Bedrock, Gulp is used to compile Javascript, preprocess your CSS and more. When you compile the static site, Gulp is also used to compile the templates. That’s exactly the functionality that I needed to modify.

In gulpfile.js I added a new gulp task:

gulp.task('templates:compile:contentDocs', templates.compile.contentDocs);

I also made sure this task ran as part of the regular build workflow.

Now in templates.js I used the same strategy I used before – carefully looked at the code that already exists, and customize the parts I needed. In this code we are using a function called getDefaultLocals to get in the content to render it. It takes the aforementione discover functions to return JS objects that we can use to render out the content.

function getDefaultLocals() {
  const defaultLocals = locals.getDefaultLocals();
  defaultLocals.docs = docs.discover();
  defaultLocals.contentDocs = contentDocs.discover();
  return defaultLocals;
}

I made a copy of the existing docs functionality (which is for rendering out docs in the styleguide, supporting Pug and Markdown templates) and made a new version called contentDocs:

contentDocs(done) {

  const defaultLocals = getDefaultLocals();

  const tasks = defaultLocals.contentDocs.allContentDocs.map(doc => {

    return gulp.src(paths.content.contentDocs.renderTemplate)

      .pipe(data(function (file) {
        return Object.assign({}, getDefaultLocals(), {
          doc,
          pathname: file.path.replace(path.join(process.cwd(), paths.content.templates.path), '').replace('.pug', ''),
        })
      }))
      .pipe(gulpPug(config.pug))
      .pipe(rename(function (path) {
        path.basename = doc.attributes.filename;
      }))
      .pipe(gulp.dest(paths.dist.contentDocs))


  })

  const stream = es.merge.apply(null, tasks);
  stream.on('end', done);
  return stream;

},

There is a lot going on here, but essentially we are doing the same as on the server: taking a template and applying it to content. The stream part of the code is to make it work with gulp.

Here, we are not doing it based on a single user request but we are processing a series of files. paths.content.contentDocs.renderTemplate refers to the aforementioned doc.pug ; using gulp-data we make sure the markdown content is available for the template to render.

So I got this part working:

1. you could see a documentation page while developing using the server
2. You could render out all documentation pages to static HTML

Rebuilding the blog part

I then tried to tackle the blog functionality. I got that working on the server first.

I used the WordPress Export functionality to grab an XML file with all my content in there. I then used a script to convert everything to files:

npx wordpress-export-to-markdown bedrock.WordPress.2021-07-03.xml

This gave me a back a series of .md files:

2015-12-22-welcome-to-bedrock.md
2015-12-23-a-quick-homepage-for-bedrock.md
2015-12-29-jade.md
2015-12-30-a-static-site-generator-history.md
2016-01-25-color-guide.md
2016-07-07-adding-selected-state-to-a-navigation-in-bedrock.md
...+53 more posts (59 blog posts in total)

I then set up things in similar ways as the documentation, thinking about the URLs:

app.get('/blog/:year/:month/:day/:title', function (req, res) {

  const blogFilename = req.params.year + '-' + req.params.month + '-' + req.params.day + '-' + req.params.title
  let post = _.find(blogPosts.discover().allBlogPosts, post => post.attributes.filename === blogFilename);

  renderView(req, res, 'blog/_template/post', {
    pathname: path.join('blog/', blogFilename),
    post
  });
});

This code so that people can visit /blog/:year/:month/:day/:title e.g. /blog/2021/07/04/my-post .

I would take the filename of the markdown file and process it from something like 2015-12-22-welcome-to-bedrock.md to 2015/12/22/welcome-to-bedrock using some simple replace logic:

parsedFile.url = filename.replace('.md','').replace('-','/').replace('-','/').replace('-','/')

I got the blog working quite quickly and I was pleased with myself.

For the overview I applied the .sort().reverse() methods so that the data would be from new to old.

For the index page I wrote some basic pagination logic.

-
    let perPage = 10;
    let lower = 0 + parseInt(page) * perPage;
    let higher = perPage + parseInt(page) * perPage;
    let posts = blogOverview.slice(lower,higher)

    // Pagination
    let hasNewerPage = false;
    let hasOlderPage = false;
    let olderPage;
    let newerPage;
    if (parseInt(page) === 1) {
        hasOlderPage = true;
        olderPage = parseInt(page)+1
    } else if (parseInt(page) > 1) {
        hasOlderPage = true;
        hasNewerPage = true;
        olderPage = parseInt(page)+1
        newerPage = parseInt(page)-1
    }

I really need to remember what slice() does because I always have to look it up.

Then depending on the existence of a newer or older page, show the pagination button:

.c-pagination

  if hasNewerPage
      a.c-btn.previous(href='/blog/'+newerPage) Newer
  if hasOlderPage
      a.c-btn.next(href='/blog/'+olderPage) Older

For clarity, on the blog index we are looping over blogOverview as follows:

each post in posts
   .c-article
       header.c-article-header
           h2
               a(href="/blog/"+post.url) #{post.attributes.title}
           p
               | Posted on #{post.attributes.date} 
               if post.attributes.time
                   | at #{post.attributes.time} 
               if post.attributes.author
                   | by #{post.attributes.author}

       .c-article-body.c-content!=marked(post.body)

We are using the aforementioned front matter attributes. This time with a bit more metadata:

---
title: "PurgeCSS support in Bedrock 1.34"
date: "2021-06-17"
author: Wolfr_
---

I could bother to make all the dates to be a standardized format but I didn’t. Somehow the export didn’t grab the post time but only the date. Since I sometimes write multiple posts in one day, this also creates an implementation problem. Ugh.

So all in all I was happy to have a blog working. You could go from the index page to the detail page and use the pagination to see all the posts. Awesome.

Then as with the documentation pages I realized I needed to make this work for the static build, else I could never finish the project. I spent some time trying to get this to work but I realized I needed:

1. A format where the folders follow the generation logic (i.e. instead of a flat folder with a bunch of markdown files, folders for every year/month
2. OR Generate said folders with a script on build time based on the contents of my flat folders

I don’t know enough about NodeJS and fs to comfortably do (2). I could do (1) and the aforementioned WordPress export plugin supports the generation of folders.

If I would ever make this a framework for other people to use I would force people to deal with a whole folder structure to make a new blog post, which is a pain.

Conclusions

So now I am in a bit of a pickle.

First of all, how to continue this work? It’s obviously not finished, because of the missing functionality to export the blog statically.

Should I push on? Shouldn’t I just keep this WordPress site running? I learned what I wanted to learn, does the world need another blogging engine that has less features than WordPress? I don’t like static blogging in practice anyway. I need a CMS and that is a whole problem case in itself.

Second, in Bedrock, does it really make sense to reimplement all functionality twice? First for the server use case and then for the static generation?

I guess it does for the core functionality of Bedrock.

The static site generation is awesome for CI purposes. We use this all the time at work. We will deploy prototypes and their different branches and all of it will be very easy with the setup that we have in place.

I suppose I could make a fork of Bedrock that is server-side only and that -has no build. But then I would also encounter my skill ceiling.

I have no knowledge setting up Express servers to deploy on the internet. I suppose it is a new skill I could learn. Heroku is the first thing that comes to mind. Then maybe I could also experiment with posting forms to the server and saving things to a database.

Oh, all the things I could learn!

For now I am just happy to have learned more about Bedrock’s internals. You know, for if I ever need to fix something to the actual scope of the project.

If you are curious, I put the site up at https://github.com/usebedrock/bedrock-site-2 . If you want to hack on it, happy coding. If you’re interested in Bedrock in general, give it a try and if you are stuck, open an issue.

Oh and if you made it to the end, please let me know on Twitter.