Data Shortcode for Hugo

In the article, I show the differences in the website analytic metrics collected on the server- and client-side. It contains several dynamic values (e.g., pageviews or visits number, the date range, etc.) scattered throughout the text. To update them, I need to pass through the content and adjust them manually. So as I plan to bring the data in this post up to date regularly (I get new input every day), this task cumulatively could consume a lot of my time. Therefore, I have decided that the data in the post should be updated automatically. In this article, I describe how I have achieved this goal.

Introduction

For my website, I use a static website generator called Hugo, so I have explored the Internet for the default options provided by this tool. I found two main approaches:

• Data Folder. It is possible to store additional data in the Data Folder in YAML, JSON, or TOML format. This data can be accessed using the .Site.Data variable.
• Param Shortcode. In the front matter of your content files, you can define a parameter and then access its value through the param built-in shortcode.

However, both these approaches have their limitations in my setup. If I use a data folder, then the data is available to the whole website and may interfere with the data on other pages. To update the data in the param shortcode approach, I have to develop a logic that would rewrite the values of the params defined in the front matter – I cannot just rewrite the file with new data. Thus, to overcome these limitations, I have developed my approach.

Approach

The approach relies on the getJSON function that Hugo provides to read local JSON files. I have developed a shortcode that substitutes a template param from a content file with the value of the variable defined in a local JSON file. That allows me to write a post with template params only once. Later, when I receive new input, I need to update the JSON file with the fresh data (that can be done automatically) and run Hugo to regenerate the corresponding webpage.

JsonData Shortcode Definition

Let’s consider the source code of the shortcode. The following listing presents the simplified version (without error processing) of the code:

{{- $json_filename := .Get "src" -}}
{{- $json_varname := .Get "var" -}}
{{- $json_format := .Get "format" -}}
{{- $json_data_filepath := path.Join "content" (path.Dir .Page.File) $json_filename -}}
{{- $json_data := getJSON $json_data_filepath -}}
{{- $var_value := index $json_data $json_varname -}}
{{- if $json_format -}}
  {{ printf $json_format $var_value }}
{{- else -}}
  {{ $var_value }}
{{- end -}}

First, I get the values of shortcode’s src, var, and format parameters. The src parameter specifies the name of the local JSON that contains the variable with the var name and its value. Thus, a post may rely on variables defined in several JSON files if required. The format parameter defines the format string.

In the fourth line, I build the relative path to the local JSON file. This file must be located in the same directory as the content file itself. Note that instead of .Page.Dir (that is deprecated and is subject to removal in the future Hugo version), I use the auxiliary function (path.Dir .Page.File) to get the path of the content file relative to the content/ directory.

In the sixth line, I call the getJSON function to read the content of the local JSON file and use the index function in the seventh line to get the value of the var variable.

Then I check if the format parameter is provided in the shortcode. If it is, then we print the value of the var variable, formatting it according to the format string (here you can read how to make a format string). Otherwise, we output the variable value.

The real code of the shortcode defined in the layouts/shortcodes/jsondata.html file looks the following way:

{{- $json_filename := .Get "src" | default "data.json" -}}
{{- $json_data_filepath := path.Join "content" (path.Dir .Page.File) $json_filename -}}
{{- if fileExists $json_data_filepath -}}
  {{- $json_data := getJSON $json_data_filepath -}}
  {{- $json_varname := .Get "var" -}}
  {{- $var_value := index $json_data $json_varname -}}
  {{- if $var_value -}}
    {{- $json_format := .Get "format" -}}
    {{- if $json_format -}}
      {{ printf $json_format $var_value }}
    {{- else -}}
      {{ $var_value }}
    {{- end -}}
  {{- else -}}
    {{ errorf "Cannot get the value of the variable %s from the data file: %s" $json_varname $json_data_filepath }}
  {{- end -}}
{{- else -}}
  {{ errorf "Cannot find the file: %s" $json_data_filepath }}
{{- end -}}

JsonData Shortcode Usage

Let’s consider how to use this shortcode on my previous post example. The content directory has the following files inside:

$ exa --tree content/post/2021/2021-09-comparison-of-cf-and-ga-data/
content/post/2021/2021-09-comparison-of-cf-and-ga-data
├── data.json
├── index.md
├── pageviews.json
└── visitors.json

As you can see, there are four files inside:

• index.md is a Hugo content file with the raw text of the article;
• pageviews.json and visitors.json contain the code for the pageviews and visitors graphs correspondingly;
• data.json contains the variables and their values used by the jsondata shortcode.

At the time of writing, the data.json contains the following content, which is generated automatically in a Jupyter notebook used to analyze visitors and pageviews data:

$ jq . content/post/2021/2021-09-comparison-of-cf-and-ga-data/data.json
{
  "min_date": "July 19, 2021",
  "max_date": "September 17, 2021",
  "cf_avg_visitors": 436.3114754098361,
  "ga_avg_visitors": 109.26229508196721,
  "avg_visitors_scale": 4.172305624197787,
  "visitors_correlation": 0.8703163848341614,
  "cf_avg_pageviews": 891.016393442623,
  "ga_avg_pageviews": 143.327868852459,
  "pageviews_scale": 6.548841590611579,
  "pageviews_correlation": 0.4551500669279511
}

Now, let’s consider how these variables are used in a content file. As I have described in the previous section, you can optionally change the representation of the value using the format string defined in the format shortcode parameter. You can use it to yield only several float digits in the article text while storing precise values in the JSON file. For instance, my previous article content file contains the following sentence:

According to Cloudflare, every day my website visits on average **{{<jsondata src=“data.json” var=“cf_avg_visitors” format="%.0f">}}** Unique Visitors.

In this sentence, I use the shortcut to get the value of the cv_avg_visitors variable from the data.json file, round it to the nearest integer, and print it in bold.

If you do not want to format the values, you can just omit the format shortcode parameter. For instance, I do this when I print the dates:

Thus, in this article I rely on the data from **{{<jsondata src=“data.json” var=“min_date”>}}** to **{{<jsondata src=“data.json” var=“max_date”>}}**.

Limitations

Of course, the described approach has some limitations. First, the shortcode currently works only with string, numeric and boolean JSON types defined on the first level. Unfortunately, it is not possible to access array and object values using this shortcode.

Second, as JSON specification does not have a “Date/Time” type, Date/Time values are stored as strings. Therefore, you cannot use Hugo’s Date/Time format functions to produce the necessary output. If you need to output a date, you need to format it before storing it in a JSON file.

Last but not least, although I am not an expert in Hugo, it seems that every time you use the shortcode in a content file, the corresponding JSON file is re-read. Indirectly, I have confirmed this assumption by tracing the Hugo process using the strace utility. Every time I store a content file that contains the jsondata shortcode, the corresponding JSON file is read the same amount of times as the number of times the shortcode appears in the text. Due to this, the generation process is slower. However, this is a bearable trade-off because you regenerate a website rarely.