A little while ago I stopped using Tumblr for this site and went back to being self-hosted using a natty bit of flat-file blogging software called Jekyll. A couple of weeks ago I had the chance to finally get it working completely how I’d like, and if you’ve noticed a massive uptick in content (uh, sorry people on Facebook when I ran a bunch of test-ish posts without switching off the RSS linkage), this is why.
(For anyone who knows me for book writing, this post is likely of little interest; go amuse yourselves elsewhere. I’m partly keeping this here as a reminder to myself of how I got everything to work, and partly as a set of vaguely helpful pointers to anyone Googling for similar systems themselves.)
Jekyll reads Markdown text files and uses them and a simple (though highly extensible) set of layout and style rules to generate a website. You can install it on your own computer, give it a directory to watch for content, and have it build you all the HTML you need. In theory, you could upload the resulting site to your own server by FTP and, hey presto, your site’s live.
That, of course, is a massive pain in the behind. Much more sensible is to install Jekyll on your server instead (either ludicrously easy if you have your own dedicated hardware, or still pretty easy if you have shared hosting and install it via RVM, which took me about five minutes on Google to figure out). Then all you need to do is feed it content and run the build command on the server, minimal uploading required. This is where Dropbox comes in.
I’d originally been going to use Marco Arment’s Second Crack, a similar flat-file engine. It’s fussy about its web root, though, and I can’t change that on shared hosting. But the suggestion to use Dropbox to sync files is equally applicable to Jekyll. Most how-tos suggest using Git to sync Jekyll, but Git doesn’t play nice with other applications, as a rule, and requires the use of actual commit commands to upload. Dropbox’s API is used by tons of applications, and the software can run without human intervention at all. The former, in particular, is extremely handy.
Install the Dropbox CLI version on your server. Create a new user account for the blog (probably the easiest option) and give it shared access to a folder on your primary user account (called, in my case, ‘Blog’). Everything you put into your regular Dropbox ‘Blog’ folder then automatically updates server-side when Dropbox runs there. You can use their own dropbox.py utility as a user, but for actual automation you want to set up a cron job to run the sync for you on a regular basis, calling a tiny script like this every couple of hours or so:
#!/bin/bash
~/.dropbox-dist/dropboxd
That’s your blog’s source files being automatically updated so long as whatever machine you’re making changes or writing material on has a connection to Dropbox. This means you can use (insert Dropbox-friendly writing app of choice) on your phone to create content without having to go near a web browser. As someone who’s tried to use WordPress’s back end on a phone before, that’s wonderful.
Because you’re using your ‘Blog’ folder (or, in my case, a named folder inside it) as Jekyll’s ‘source’ folder, you want it to have the right structure. The easiest way to get that is to run Jekyll once on your own machine, have it generate the new blog for you, and either copy the contents across or symlink the generated folder to your Blog one (if you symlink, you end up, like me, with your source folder being Blog/somename; you don’t if it’s all just copied). This folder contains the _posts, _layouts, css and _drafts directories, and all the other gubbins that Jekyll relies on.
Once this folder’s synced to the server, you can now run a Jekyll build command on the server (using your source directory as source and your public_html (or whatever your choice is) as its destination) and it’ll refresh your site with any updates received since the last time. I ended up writing a bash script, jek.sh, to do it just to save on typing:
#!/bin/bash
jekyll build --source /home/username/path/to/Dropbox/Blog/foldername --destination /home/username/path/to/public_html
(Note: if you’ve installed a plugin that adds the Maruku .md parser rather than the Redcarpet one Jekyll uses by default, make sure Redcarpet is set in _config.yml and/or remove Maruku altogether because Maruku seems to get very easily confused and throws errors all over the place.)
So all we need to do now is set up a cron job to do this every once in a while, and that’ll handle rebuilding the site whenever anything new lands, no?
Unfortunately, while that script works as a user, cron is fussier. Its limited environment means you have to be super-specific with what you tell it, and after much playing around and a helpful query on StackExchange I finally have this massive command as a cron job set to fire at three minutes past every hour (note: if you don’t need RVM, the PATH will be different):
export LANG=en_US.UTF-8 PATH=:/home/username/.rvm/gems/ruby-version-number/bin:/home/username/.rvm/gems/ruby-version-number@global/bin:/home/username/.rvm/rubies/ruby-version-number/bin:/home/username/.rvm/bin:/usr/local/jdk/bin:/home/username/perl5/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/home/username/bin && /home/username/.rvm/gems/ruby-version-number/bin/jekyll build --source /home/username/path/to/blog/folder --destination /home/username/public_html >/dev/null 2>&1
Catchy, huh?
The PATH variable has to be set for cron, and varies by server. (From memory->) You can find out what on earth yours is easily enough (echo $PATH) and then paste the result into the PATH= statement for cron to use. (/memory)
The LANG variable is also vital. Under cron, either ruby or Jekyll defaults to US-ASCII and this throws up horrible errors with invalid characters when it tries to run the build command.
The actual build command itself is no different to that user script (the /dev/null bit after just tells it to do it quietly without pestering you with messages). The fiddly bit is making cron run it without error.
So that’s Dropbox syncing your .md blog entry files (and everything else), and your server automatically regenerating the blog for you (in a process you could probably even tie in to inotify-tools for extra speed). Pretty neat. But I wanted easy link sharing, and the ability to pull photos from Instagram, and thanks to IFTTT’s ability to write to Dropbox, that’s all a one-click possibility.
I have two link sharing options: a daily round-up, and individual link posts.
The latter come via Pocket. I have an IFTTT recipe that, whenever a new link is added to my Pocket account, creates a text file in my blog’s _posts folder in Dropbox. The file is titled {{Title}} and contains the following:
---<br>
layout: post<br>
title: {{Title}}<br>
type: link<br>
link: {{Url}}<br>
tags:<br>
- pocketed<br>
- ofnote<br>
---<br><br>
> {{Excerpt}}
So for a link titled ‘10 Incredible Pictures Of Cats’, a file ‘10_incredible_pictures_of_cats.txt’ (the underscores-for-spaces and .txt format is standard and unchangeable) is created in _posts, formatted with Jekyll’s required front matter. link: is non-standard; I use it in my post.html layout file like so to generate an outbound link as the entry’s title:
{% if page.type == "link" %}
<a title="{{ page.title }}" href="{{ page.link }}">{{ page.title }} →</a>
{% else %}
<a title="{{ page.title }}" href="{{ page.url }}">{{ page.title }}</a>
{% endif %}
For daily link round-ups, two IFTTT recipes come in. The first runs daily and creates a new file in _drafts called ‘daily-links.txt’ with the following contents:
---<br>
layout: post<br>
title: Today's Links<br>
type: post<br>
tags:<br>
- links<br>
---<br><br>
The second uses link service Delicious. Whenever a new public link is added to my Delicious account, the daily-links.txt file in Dropbox has the following appended to it:
* Linkery: [{{Title}}]({{Url}}) / {{Tags}} / {{Notes}}<br><br>
(Because Tweetbot has no ability to share to Delicious, I also have a third recipe using Bitly, which Tweetbot can talk to, shunting Bitly links to Delicious so they can then be shunted again to Dropbox.)
To get the daily links out of _drafts and into _posts I have another cron job. In the dead hours of night, once a day, cron calls a post-moving bash script containing the following:
#!/bin/bash
if grep -q Linkery "/home/username/path/to/blog/_drafts/daily-links.txt"; then
mv /home/username/path/to/blog/_drafts/daily-links.txt /home/username/path/to/blog/_posts/daily-links.txt
else
rm -f /home/username/path/to/blog/_drafts/daily-links.txt
fi
What this does, in the grep command, is see if the daily links text file has had anything added to it (if so, the string “Linkery” is found inside). If it has, it’s moved into _posts for processing. If the string isn’t found and there’s no links there, the file is deleted. This cron job runs a few hours before the IFTTT recipe that creates the next day’s file, so there’s no duplication.
But… Jekyll doesn’t read .txt files, and has a strict file naming convention: ‘YYYY-MM-DD-title-of-entry.md’. Very different from our ‘10_incredible_pictures_of_cats.txt’ and ‘daily-links.txt’ now sitting in _posts.
So there’s a final bash script, namefix.sh, sitting in _posts.
#!/bin/bash
FILES=/home/username/path/to/blog/_posts/*.txt
for f in $FILES
do
pth=${f%%/*}
tit=${f##*/}
new=`echo $tit|tr '_' '-'`
echo $new
name=${new%.*}
ext=${new#*.}
today=`date +%Y-%m-%d`
newname=$today-$name.md
echo $newname
mv -f "$f" "/home/username/path/to/blog/_posts/$newname"
done
This searches the _posts directory for text files, splits the resulting title from the /path/to/it before and swaps all underscores for dashes as $new. Then it generates the date and prefixes $new with that date and bundles the whole thing together as a .md file (.txt and .md are directly swappable with no conversion) and puts it back. (Note: Jekyll can get a bit confused if titles contain double dashes or extra punctuation; I will at some point refine the script to strip all those things from names.)
As with the post mover script, this one also runs through cron (every hour, on the hour, just before the Jekyll build). Again cron has issues, and will throw up (non-fatal) errors if there are no .txt files for the script to fix. Therefore it needs a find command in the cron job to check for .txt files before calling the script. Like so:
find /home/username/path/to/blog/_posts/ -maxdepth 1 -type f -name "*.txt" 2>/dev/null | grep -q /home/username/path/to/blog/_posts/ && /home/username/path/to/blog/_posts/namefix.sh
The find command only checks the _posts directory (maxdepth 1) for .txt files, using grep to determine if any are there and then only running the script if it returns results.
And that’s it. Every time I Pocket something, it appears here without me doing a thing. Every time I read something cool I can add it to Delicious or Bitly and it’ll be in the daily round-up with no effort on my part.
With the scripts set up this way, IFTTT makes just about anything possible. Anything I tag ‘nh’ in Instagram appears here, because IFTTT again creates a new text file in Dropbox containing:
---<br>
layout: post<br>
title: {{CaptionNoTag}}<br>
type: photo<br>
tags:<br>
- instagram<br>
- photography<br>
---<br><br><img src="{{SourceUrl}}" alt="{{CaptionNoTag}}" class="photo"><br><br>
{{Caption}}<br><br>
(<a href="{{Url}}">view on Instagram</a>)
(I’ve been a bit lazy and not yet added a new if page.type statement to post.html see if the post is a photo one and display no title if it it is, but that’s on my list. The only thing to be wary of, as I’ve found out, is not to use a long caption since that’s the given post title and filename.)
Doing something similar for Flickr, 500px, Gmail, YouTube, and any of the other billion IFTTT channels would be trivially easy. With everything writing text to Dropbox, the heavy lifting is done already. The little scripts to move files around and name them properly will handle everything.
It took some diving into the vagaries of bash and cron, things I’ve barely played with in the past, but now it’s done keeping the blog running through Jekyll is as easy or easier than it ever was with Tumblr (pro tip: text expansion on computer or phone can fill in the required header information in each entry for no great effort), and a whole lot easier than with WP. Most especially of all on my phone. Using IA Writer (which, when linked to Dropbox, browses your folder tree like you’d expect and allows you to save wherever you want, unlike Byword which is confined to one folder only) I can write entries in a nice, reliable plain text editor and have them sync up automatically, rather than having to faff around in a browser window or something like Tumblr’s annoyingly crash-prone, text-eating app. And everything I share, from wherever I share it, can end up here without me having to do a damn thing.
OK. Enough techy stuff. Back to writing. And those magnificent cat pictures I keep meaning to link to.
