Deployment with Circle CI, AWS S3, AWS CloudFront

[stephenpiscura]

Hi everyone,

Referencing:

• Automated Deployment with Circle CI + Hugo
• https://forestry.io/blog/automate-deploy-w-circle-ci/

I’ve attempted to setup my deployment pipeline with using Circle CI, GitHub, AWS S3, and AWS CloudFront.

My issue: Circle CI is listing my builds as “successful”, but no pages are being generated in my S3 bucket, save three pieces of “Categories”-related content:

• /categories/index.html
• /categories/index.xml
• /categories/page/1/index.html

I don’t use any taxonomies in my project—compounding the strangeness.

My static files are all being successfully generated (css, img, etc.) in my S3 bucket.

Here’s my .circleci/config.yml file:

version: 2
jobs:
  build:
    docker:
      - image: cibuilds/hugo:latest
    working_directory: ~/hugo
    environment:
      HUGO_BUILD_DIR: ~/hugo/public
    steps:
      - run: apk update && apk add git
      - checkout
      - run: git submodule sync && git submodule update --init
      - run: curl -L https://github.com/bep/s3deploy/releases/download/v2.3.0/s3deploy_2.3.0_Linux-64bit.tar.gz | tar xvz
      - run: HUGO_ENV=production hugo -v -d $HUGO_BUILD_DIR
      - run: htmlproofer $HUGO_BUILD_DIR --allow-hash-href --check-html --empty-alt-ignore --disable-external
      - deploy:
          name: deploy
          command: |
            if [ "${CIRCLE_BRANCH}" = "master" ]; then
              ./s3deploy -source=$HUGO_BUILD_DIR -region=us-east-2 -bucket=mybucketname -distribution-id=_____MyID_____
            else
              echo "Not master branch, dry run only"
            fi

Here’s a snapshot of what Hugo generates when it’s running locally:

   | EN  
+------------------+----+
  Pages            | 78  
  Paginator pages  |  1  
  Non-page files   |  1  
  Static files     | 88  
  Processed images |  0  
  Aliases          |  4  
  Sitemaps         |  1  
  Cleaned          |  0  

And here’s my log from Circle CI on the run: HUGO_ENV=production hugo -v -d $HUGO_BUILD_DIR command:

#!/bin/bash -eo pipefail
HUGO_ENV=production hugo -v -d $HUGO_BUILD_DIR
INFO 2019/01/31 18:29:52 No translation bundle found for default language "en"
INFO 2019/01/31 18:29:52 Translation func for language en not found, use default.
INFO 2019/01/31 18:29:52 i18n not initialized; if you need string translations, check that you have a bundle in /i18n that matches the site language or the default language.
INFO 2019/01/31 18:29:52 Using config file: 
Building sites … INFO 2019/01/31 18:29:52 syncing static files to /root/hugo/~/hugo/public/
INFO 2019/01/31 18:29:52 Alias "/tags/page/1/index.html" translated to "tags/page/1/index.html"
INFO 2019/01/31 18:29:52 Alias "/categories/page/1/index.html" translated to "categories/page/1/index.html"
INFO 2019/01/31 18:29:52 Found no layout for "404", language "en", output format "HTML": create a template below /layouts with one of these filenames: 404.html

                   | EN  
+------------------+----+
  Pages            |  6  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     | 83  
  Processed images |  0  
  Aliases          |  2  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 26 ms
htmlproofer $HUGO_BUILD_DIR --allow-hash-href --check-html --empty-alt-ignore --disable-external00:00
Exit code: 0
#!/bin/bash -eo pipefail
htmlproofer $HUGO_BUILD_DIR --allow-hash-href --check-html --empty-alt-ignore --disable-external
Running ["ScriptCheck", "ImageCheck", "LinkCheck", "HtmlCheck"] on ["~/hugo/public"] on *.html... 


Ran on 5 files!


HTML-Proofer finished successfully.
deploy

Please note that none of my 78 pages are generating.

I’m totally unfamiliar with backend processes and, as such, most of what i’ve done here is rote copying and pasting and hoping for the best. (For example, i don’t understand the purpose of the working_directory in my .circleci/config.yml file and if i should be changing this to something else for my use case.). I’m sorry to paste so much into this post. Rest assured i’ve got more to paste, if need be.

This may be way out of the scope of Hugo support, but i appreciate any thoughts anyone may be willing to share.

[stephenpiscura]

Sorry, i should add, “Tags” content and root-level index files are also being generated:

Again, i’m not using Tags or Categories or any taxonomies.

[zwbetz]

Since you’re setting

HUGO_ENV=production

Do you have any template logic that renders pages differently depending on the env set?

[stephenpiscura]

Thanks, @zwbetz, i wasn’t even aware of environmental variables until i read your post and started looking into them. I searched my project and found the only reference to the Hugo environment is in my forestry.io settings.yml file, which relates to how forestry.io builds previews for users. In this file, i have kept forestry.io’s defaults, which are:

build:
  preview_command: hugo -E -F -D -b $DEPLOY_URL -d public
  publish_command: hugo -d public
  preview_env:
  - HUGO_ENV=staging
  - HUGO_VERSION=0.53
  publish_env:
  - HUGO_ENV=production
  - HUGO_VERSION=0.53
  preview_output_directory: public
  output_directory: public

Other than that, there is—to my knowledge—no other reference in any of my files to the Hugo environment.

Thanks again for offering your thoughts.

[zwbetz]

Yeah your env var config is fine then.

So I see your preview build command is:

preview_command: hugo -E -F -D -b $DEPLOY_URL -d public

Do you perhaps have pages that are (1) drafts, or (2) future dated? Because if so they wouldn’t get published with your current production build command

[stephenpiscura]

Every single piece of content on my site was set to draft: true.

I set my “About” page to draft: false, pushed it to GitHub, and—lo and behold!—now it exists! Now just need to change the others.

Thank you @zwbetz, now i have a solid deploy method which i’ll be able to use from this point forward!

[zwbetz]

Good deal

On another note, I checked out the site linked in your profile. Your photography section is nice

[stephenpiscura]

Hey thanks very much! That site is getting a little long in the tooth, but better than nothing.

One loose-ends follow up question. No sweat if you don’t have time for it.

Before opting for the stack i outlined above, i was using Travis CI with the following .travis.yml configuration file:

install:
- pip install --user awscli
- wget https://github.com/gohugoio/hugo/releases/download/v0.53/hugo_0.53_Linux-64bit.deb
- sudo dpkg -i hugo*.deb
- hugo version
before_script:
- rm -rf public
- hugo
script:
- aws s3 sync public/ s3://mybucketname --acl public-read --delete
- aws cloudfront create-invalidation --distribution-id _____MyID_____ --paths '/*'
branches:
  only:
  - master

(Credit to Nate Barbettini and his excellent Hugo training videos.)

Using this configuration, i don’t need any AWS Bucket Policy in oder for my site to be publicly accessible.

But using CircleCI and bep’s s3deploy, as i outlined in my original post, i need to set a Bucket Policy like this in order to avoid a 403 error:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucketname/*"
        }
    ]
}

I even tried adding -public-access to the deploy command like so:

./s3deploy -source=$HUGO_BUILD_DIR -region=us-east-2 -bucket=mybucketname -distribution-id=_____MyID_____ -public-access

but it doesn’t seem to make a difference—the public Bucket Policy is still needed.

Any guesses about what might be making the difference?

[zwbetz]

Hmm. Am not sure on that one