Hugo Site Publishing with Respectful Caching and IndexNow Submission

Abstract

Hugo builds static pages; publishing determines whether deployed artifacts carry accurate cache validators and whether search engines receive explicit update notification. The implementation separates Hugo generated deployment artifacts, an mtime manifest and an IndexNow payload, from publication mechanics; publishing script restores filesystem mtimes, aligns .gz artifacts with their sources, deploys through rsync, and submits the current URL list.

The design is intentionally narrow: CDN policy control and general deployment framework construction are outside scope; HTTP revalidation semantics are assumed, where Cache-Control: no-cache permits storage but requires validation before reuse. Filesystem mtimes then provide sufficient Last-Modified based validation, while Hugo generated artifacts keep gzip variants and IndexNow notifications consistent.

Date Semantics

Article chronology follows a compact frontmatter rule:

date: combined publication and last modification date;
publishDate: publication date, used instead of date when the modification date differs;
lastmod: meaningful content update date, paired with publishDate.

Hugo frontmatter resolution mirrors the same rule and deliberately omits :git, keeping enableGitInfo available for deployment metadata without letting Git timestamps rewrite .Lastmod:

[frontmatter]
  date = ["date", "publishDate"]
  publishDate = ["publishDate", "date"]
  lastmod = ["lastmod", "date", "publishDate"]
[frontmatter]
  date = ["date", "publishDate"]
  publishDate = ["publishDate", "date"]
  lastmod = ["lastmod", "date", "publishDate"]

Filesystem mtime is separate deployment metadata: a cache validation timestamp for generated artifacts. Article chronology remains an editorial choice; rebuilding generated output does not rewrite article chronology, while generated artifact changes still move cache validators.

Hugo Outputs

Hugo renders three deployment files from the home page: mtimes.txt for post build touch, indexnow-<key>.json as the IndexNow submission payload, and indexnow-<key>.txt as the IndexNow verification key. The indexnow- prefix keeps the generated files identifiable; it also reduces the available alphanumeric or - key length to at most 119 characters¹.

[outputs]
  home = ["html", "rss", "indexnow", "mtimes"]

[outputFormats]
  [outputFormats.indexnow]
    mediaType = "application/json"
    baseName = "indexnow-<key>"
    isPlainText = true
    notAlternative = true
  [outputFormats.mtimes]
    mediaType = "text/plain"
    baseName = "mtimes"
    isPlainText = true
    notAlternative = true
[outputs]
  home = ["html", "rss", "indexnow", "mtimes"]

[outputFormats]
  [outputFormats.indexnow]
    mediaType = "application/json"
    baseName = "indexnow-<key>"
    isPlainText = true
    notAlternative = true
  [outputFormats.mtimes]
    mediaType = "text/plain"
    baseName = "mtimes"
    isPlainText = true
    notAlternative = true

IndexNow Output

layouts/index.indexnow.json derives the host from baseURL, computes the IndexNow key from the output filename, publishes the key file with resources.FromString, and emits the sorted HTML permalink list as urlList.

{{- $baseURL := urls.Parse .Site.BaseURL -}}
{{- $key := "" -}}
{{- with .Site.Home.OutputFormats.Get "indexnow" -}}
  {{- $key = strings.TrimSuffix ".json" (path.Base .RelPermalink) -}}
{{- end -}}
{{- with resources.FromString (printf "%s.txt" $key) (printf "%s\n" $key) -}}
  {{- $noop := .Publish -}}
{{- end -}}
{{- $urls := slice -}}
{{- range .Site.Pages -}}
  {{- with .OutputFormats.Get "html" -}}
    {{- $urls = $urls | append .Permalink -}}
  {{- end -}}
{{- end -}}
{{- dict "host" $baseURL.Host "key" $key "urlList" ($urls | uniq | sort) | jsonify (dict "indent" "  ") -}}
{{- $baseURL := urls.Parse .Site.BaseURL -}}
{{- $key := "" -}}
{{- with .Site.Home.OutputFormats.Get "indexnow" -}}
  {{- $key = strings.TrimSuffix ".json" (path.Base .RelPermalink) -}}
{{- end -}}
{{- with resources.FromString (printf "%s.txt" $key) (printf "%s\n" $key) -}}
  {{- $noop := .Publish -}}
{{- end -}}
{{- $urls := slice -}}
{{- range .Site.Pages -}}
  {{- with .OutputFormats.Get "html" -}}
    {{- $urls = $urls | append .Permalink -}}
  {{- end -}}
{{- end -}}
{{- dict "host" $baseURL.Host "key" $key "urlList" ($urls | uniq | sort) | jsonify (dict "indent" "  ") -}}

Modification Time Manifest

layouts/index.mtimes.txt requires enableGitInfo = true and emits one relative path and timestamp pair for each generated output. It folds the later value of page AuthorDate and CommitDate into an optional HUGO_MINIMUM_MTIME lower bound, maps each output format to its generated file path, and formats UTC timestamps.

HUGO_MINIMUM_MTIME should be an ISO 8601 timestamp, for example the active theme’s last commit time; when absent, the template falls back to the Unix epoch beginning.

Hugo’s binary build date, exposed as hugo.BuildDate in fixed upstream versions, belongs to the same lower bound for Hugo generated files: generator changes can alter artifacts without changing site sources.

The manifest covers page outputs and page bundle resources. Static files stay outside it because Hugo preserves their source mtimes when copying them; global assets remain outside its scope.

{{- $minimumMTime := time.AsTime "1970-01-01T00:00:00Z" -}}
{{- with os.Getenv "HUGO_MINIMUM_MTIME" -}}
  {{- $minimumMTime = time.AsTime . -}}
{{- end -}}
{{- with hugo.BuildDate -}}
  {{- $hugoMTime := time.AsTime . -}}
  {{- if gt $hugoMTime.Unix $minimumMTime.Unix -}}
    {{- $minimumMTime = $hugoMTime -}}
  {{- end -}}
{{- end -}}
{{- $siteMTime := $minimumMTime -}}
{{- range hugo.Sites -}}
  {{- range .Pages -}}
    {{- with .GitInfo -}}
      {{- $gitMTime := .AuthorDate -}}
      {{- if gt .CommitDate.Unix $gitMTime.Unix -}}
        {{- $gitMTime = .CommitDate -}}
      {{- end -}}
      {{- if gt $gitMTime.Unix $siteMTime.Unix -}}
        {{- $siteMTime = $gitMTime -}}
      {{- end -}}
    {{- end -}}
  {{- end -}}
{{- end -}}
{{- range hugo.Sites -}}
  {{- range .Pages -}}
    {{- $mtime := $siteMTime -}}
    {{- with .GitInfo -}}
      {{- $mtime = $minimumMTime -}}
      {{- $gitMTime := .AuthorDate -}}
      {{- if gt .CommitDate.Unix $gitMTime.Unix -}}
        {{- $gitMTime = .CommitDate -}}
      {{- end -}}
      {{- if gt $gitMTime.Unix $mtime.Unix -}}
        {{- $mtime = $gitMTime -}}
      {{- end -}}
    {{- end -}}
    {{- $touchTime := $mtime.UTC.Format "200601021504.05" -}}
    {{- $pagePath := strings.TrimPrefix "/" .RelPermalink -}}
    {{- $bundleDir := "" -}}
    {{- with .File -}}
      {{- $bundleDir = .Dir -}}
    {{- end -}}
    {{- range .OutputFormats -}}
      {{- if ne (strings.ToLower .Name) "mtimes" -}}
        {{- $path := strings.TrimPrefix "/" .RelPermalink -}}
        {{- if or (eq $path "") (strings.HasSuffix $path "/") -}}
          {{- $path = printf "%sindex.html" $path -}}
        {{- end }}
        {{- printf "%s %s\n" $path $touchTime -}}
      {{- end -}}
    {{- end -}}
    {{- range .Resources -}}
      {{- $resourceName := strings.TrimPrefix "/" .Name -}}
      {{- $resourcePath := path.Join $pagePath $resourceName -}}
      {{- $resourceMTime := $mtime -}}
      {{- with $bundleDir -}}
        {{- $sourcePath := path.Join . $resourceName -}}
        {{- if os.FileExists $sourcePath -}}
          {{- with os.Stat $sourcePath -}}
            {{- $resourceMTime = .ModTime -}}
          {{- end -}}
        {{- end -}}
      {{- end -}}
      {{- printf "%s %s\n" $resourcePath ($resourceMTime.UTC.Format "200601021504.05") -}}
    {{- end -}}
  {{- end -}}
{{- end -}}
{{- $minimumMTime := time.AsTime "1970-01-01T00:00:00Z" -}}
{{- with os.Getenv "HUGO_MINIMUM_MTIME" -}}
  {{- $minimumMTime = time.AsTime . -}}
{{- end -}}
{{- with hugo.BuildDate -}}
  {{- $hugoMTime := time.AsTime . -}}
  {{- if gt $hugoMTime.Unix $minimumMTime.Unix -}}
    {{- $minimumMTime = $hugoMTime -}}
  {{- end -}}
{{- end -}}
{{- $siteMTime := $minimumMTime -}}
{{- range hugo.Sites -}}
  {{- range .Pages -}}
    {{- with .GitInfo -}}
      {{- $gitMTime := .AuthorDate -}}
      {{- if gt .CommitDate.Unix $gitMTime.Unix -}}
        {{- $gitMTime = .CommitDate -}}
      {{- end -}}
      {{- if gt $gitMTime.Unix $siteMTime.Unix -}}
        {{- $siteMTime = $gitMTime -}}
      {{- end -}}
    {{- end -}}
  {{- end -}}
{{- end -}}
{{- range hugo.Sites -}}
  {{- range .Pages -}}
    {{- $mtime := $siteMTime -}}
    {{- with .GitInfo -}}
      {{- $mtime = $minimumMTime -}}
      {{- $gitMTime := .AuthorDate -}}
      {{- if gt .CommitDate.Unix $gitMTime.Unix -}}
        {{- $gitMTime = .CommitDate -}}
      {{- end -}}
      {{- if gt $gitMTime.Unix $mtime.Unix -}}
        {{- $mtime = $gitMTime -}}
      {{- end -}}
    {{- end -}}
    {{- $touchTime := $mtime.UTC.Format "200601021504.05" -}}
    {{- $pagePath := strings.TrimPrefix "/" .RelPermalink -}}
    {{- $bundleDir := "" -}}
    {{- with .File -}}
      {{- $bundleDir = .Dir -}}
    {{- end -}}
    {{- range .OutputFormats -}}
      {{- if ne (strings.ToLower .Name) "mtimes" -}}
        {{- $path := strings.TrimPrefix "/" .RelPermalink -}}
        {{- if or (eq $path "") (strings.HasSuffix $path "/") -}}
          {{- $path = printf "%sindex.html" $path -}}
        {{- end }}
        {{- printf "%s %s\n" $path $touchTime -}}
      {{- end -}}
    {{- end -}}
    {{- range .Resources -}}
      {{- $resourceName := strings.TrimPrefix "/" .Name -}}
      {{- $resourcePath := path.Join $pagePath $resourceName -}}
      {{- $resourceMTime := $mtime -}}
      {{- with $bundleDir -}}
        {{- $sourcePath := path.Join . $resourceName -}}
        {{- if os.FileExists $sourcePath -}}
          {{- with os.Stat $sourcePath -}}
            {{- $resourceMTime = .ModTime -}}
          {{- end -}}
        {{- end -}}
      {{- end -}}
      {{- printf "%s %s\n" $resourcePath ($resourceMTime.UTC.Format "200601021504.05") -}}
    {{- end -}}
  {{- end -}}
{{- end -}}

Publishing

The publishing proceeds in ordered stages. Before the build, git-restore-mtime from git-tools² restores Git backed source mtimes, including submodules; Hugo then preserves those timestamps when copying static files.

git restore-mtime -q -c
git submodule foreach --recursive 'git restore-mtime -q -c'
git restore-mtime -q -c
git submodule foreach --recursive 'git restore-mtime -q -c'

The first stage then builds a clean Hugo output and passes the optional minimum mtime as the last commit time touching .gitmodules or themes/*:

env HUGO_MINIMUM_MTIME=$(git log -1 --format=%cI -- .gitmodules themes/*) \
    hugo \
    --cleanDestinationDir \
    --gc
env HUGO_MINIMUM_MTIME=$(git log -1 --format=%cI -- .gitmodules themes/*) \
    hugo \
    --cleanDestinationDir \
    --gc

The generated manifest is applied as filesystem metadata and then removed:

while read -r file mtime; do
    env TZ=UTC touch -t "$mtime" "public/$file"
done < public/mtimes.txt

rm -f public/mtimes.txt
while read -r file mtime; do
    env TZ=UTC touch -t "$mtime" "public/$file"
done < public/mtimes.txt

rm -f public/mtimes.txt

mtimes.txt turns Hugo’s logical output list into HTTP validators. The clean destination removes stale files; the manifest pass prevents build time from becoming deployed modification time. Removing the manifest keeps deployment helper data outside the synchronized artifact set.

find public -type f -not -iname '*.gz' \
    -exec gzip -9 -k -n -f {} \; \
    -exec touch -m -r {} {}.gz \;
find public -type f -not -iname '*.gz' \
    -exec gzip -9 -k -n -f {} \; \
    -exec touch -m -r {} {}.gz \;

Precompression follows mtime restoration. Because find predicates are evaluated left to right, the second -exec runs only after gzip succeeds for the same file; it assigns each .gz artifact the source mtime, keeping compressed and uncompressed variants aligned for web servers that serve precompressed files directly³.

rsync \
    -az \
    --delete-after \
    public/ \
    <some.server.com>:/var/www/htdocs/~user
rsync \
    -az \
    --delete-after \
    public/ \
    <some.server.com>:/var/www/htdocs/~user

The generated IndexNow JSON is submitted only after rsync completes:

curl \
    -XPOST \
    -d "@$(ls -1 public/indexnow-*.json | head -n 1)" \
    -H 'Content-Type: application/json' \
    https://api.indexnow.org/indexnow
curl \
    -XPOST \
    -d "@$(ls -1 public/indexnow-*.json | head -n 1)" \
    -H 'Content-Type: application/json' \
    https://api.indexnow.org/indexnow

The full URL list suits low volume, rarely updated sites; larger or frequently updated sites require selective change sets.

After publication, public/indexnow-*.json remains an operational artifact: delete the file after submission, store it on the server as a publication record, or ignore it through web server configuration.

If named publish.sh, the script can be installed as a Git pre-push hook with ln -sf ../../publish.sh .git/hooks/pre-push.

IndexNow’s documentation limits keys to 8 to 128 characters and requires the hosted key file name to match the key value. A Hugo site uses lowercase key material, or disablePathToLower = true, to avoid default page URL lowercasing. ↩︎
MestreLion’s git-tools includes git-restore-mtime, a Git subcommand for restoring file mtimes from commit history. ↩︎
ngx_http_gzip_static_module sends precompressed .gz files instead of regular files, and its documentation recommends that original and compressed file modification times be the same. ↩︎