Books, Cloud Infra, HashiCorp, SysAdmin

Book Review – Terraform: Up & Running: Writing Infrastructure as Code

Terraform: Up & Running: Writing Infrastructure as Code

On Terraform: Up & Running: Writing Infrastructure as Code author Yevgeniy Brikman does an excellent job defending the use of Terraform, a tool designed to make the deployment of infrastructure as code. The book comes with many hands-on examples with all the code available on github for download allowing the reader to follow along.

The author provides answers to questions such as What is Infrastructure as code and What are the benefits of using Infrastructure as code. He provides context of why we are seeing a proliferation of a new set of tools – think Terraform, Packer, Docker, Ansible. He finally show how ties them back to the rise of the DevOps movement.

On DevOps, he uses a definition shared by many, myself included: “The goal of DevOps is to make software delivery more efficient”. The emphasis here is on efficiency and Terraform delivers it.

Throughout the rest of the book the many examples and exercises are set to run on Amazon AWS. Although you can run them someplace else, I would tell you to create an account on AWS and do it there. The examples do follow a logical path. You start with a single server and ending with a cluster of web servers.

To make use of new Terraform concepts, the author makes changes to the initial examples, adopting building blocks available on AWS. These concepts can to different cloud providers and the choice of AWS illustrates the power of Terraform.

One thing to keep in mind, is the fact that there isn’t a simple “brain transfer” between one cloud provider and another. This means that you can’t expect that a specific configuration on one platform will work on another. For example, an Elastic Load Balancers is a specific solution from Amazon and although a similar offer might exist in another place, Terraform can’t guarantee that it will work or act the same.

Terraform uses the HashiCorp Configuration Language (HCL), a declarative language intended to create the infrastructure. “Why should I learn yet another language” you might ask and the answer is: “To make your life easier” and here is why. Terraform can create infrastructure on different platforms such as public and private cloud providers (e.g.: AWS, Azure, DigitalOcean, OpenStack) and virtualization solutions like vSphere. This a core concept in Terraform called providers. Terraform knows how to ‘talk’ to different providers. This takes you away from having to know how to automate these platforms. Instead you can focus on learning one declarative language, HCL. This saves you from having to keep up with CloudFormation, ARM, HEAT, vRealize Automation and others.

Using Infrastructure as Code brings some important considerations. How to provide file isolation, file locking and safeguard the state of the infrastructure managed by Terraform? This brings another point, the trade-offs when using Infrastructure as Code. There are advantages of using Terraform to deploy an infrastructure. Its clear, concise, reproducible method are a plus. The drawback to have in mind is this: when not careful, a commit can break your entire infrastructure. This could be your production and staging environment.

To mitigate potential issues like that, the author shows a few practical approaches. To start, he talks about file isolation and proposes a layout to better organize your code. There’s a chapter deidcated to the use of modules (to avoid code duplication) and version control your code.

If you are new to Terraform this book will bring you up to speed and teaching you about this new and powerful tool. If you have used Terraform before, chances are you can still learn from the vast experience that Yevgeniy brings.

Terraform is a cutting edge tool and it is here to address a complex problem. This book allows you to get the information you need in no time with many examples. Recommended!

To Yevgeniy Brikman: Thank you for writing this book. Staying up to date with a tool still in it’s early days is a challenge. Writing about it must be very hard, so thank you.

Cloud Infra, FreeBSD, OpenStack

Using release(7) to create FreeBSD images for OpenStack – Yes, you can do it!


During a recent BSDNow podcast, Allan and Kris mentioned that it would be nice to have a tutorial on how to create a FreeBSD image for OpenStack using the official release(7) tools.

With that, it came to me that: #1 I do have access to an OpenStack environment and #2 I am interested in having FreeBSD as a guest image in my environment. Looks like I was up for the challenge.

Previously, I’ve had success running FreeBSD 11.0-RELEASE on OpenStack with success but more could/should be done. For instance, as suggested by Allan,  wouldn’t be nice to deploy the latest code from FreeBSD ? Running -STABLE or even -CURRENT ? Yes, it would. Also, wouldn’t it be nice to customize these images for a specific need? I’d say ‘Yes’ for that as well.

With that, I thought I would give it a try, so without further ado, here’s how you
can bake a FreeBSD image using the official releng toolset.

After some research I found that the current openstack.conf file, located at /usr/src/release/tools/ could use some extra tweaks to get where I wanted. I’ve created and attached that to a bugzilla on the same topic. You can read about that here.

This diff brings the following changes:

  •  resize FreeBSD to the size of the OpenStack flavor (growfs).
  • speeds up the boot process by disabling sendmail.
  • allows an user to ssh as root with a public key.
  • make ssh respond faster by disabling DNS lookups (can be reverted).
  • enable DHCP on the vtnet interface.

Here’s our goal:

Create a FreeBSD image for OpenStack (KVM based) using the release(7) tools.

and as a bonus:

  • Be able to upload the image to OpenStack’s Glance and to manipulate a HEAT stack _from FreeBSD_


  • Make sure you have a system that can do a build(7) – disk space is important here, more CPU power helps too.
  • Patch the openstack.conf file with the new changes – you can get the file from here.
# $FreeBSD: releng/11.0/release/tools/openstack.conf 277458 2015-01-20
# 23:56:04Z gjb $

# Set to a list of packages to install.
export VM_EXTRA_PACKAGES="net/cloud-init
 devel/py-pbr devel/py-iso8601 \
 net/py-eventlet net/py-netaddr comms/py-serial devel/py-six \
 devel/py-babel net/py-oauth net/py-netifaces"

# Set to a list of third-party software to enable in rc.conf(5).
export VM_RC_LIST="cloudinit"


vm_extra_pre_umount() {
 #env ASSUME_ALWAYS_YES=yes pkg -c ${DESTDIR} delete -f -y pkg
 #rm ${DESTDIR}/var/db/pkg/repo-*.sqlite

 echo 'growfs_enable="YES"' >> ${DESTDIR}/etc/rc.conf

 # Enable sshd by default
 echo 'sshd_enable="YES"' >> ${DESTDIR}/etc/rc.conf

 # Disable DNS lookups by default to make SSH connect quickly
 echo 'UseDNS no' >> ${DESTDIR}/etc/ssh/sshd_config

 echo 'PermitRootLogin without-password' >>

 # Enable DHCP for the OpenStack instance
 echo 'ifconfig_DEFAULT="SYNCDHCP"' >> ${DESTDIR}/etc/rc.conf

 # Disable sendmail
 echo 'sendmail_enable="NO"' >> ${DESTDIR}/etc/rc.conf
 echo 'sendmail_submit_enable="NO"' >> ${DESTDIR}/etc/rc.conf
 echo 'sendmail_outbound_enable="NO"' >> ${DESTDIR}/etc/rc.conf
 echo 'sendmail_msp_queue_enable="NO"' >> ${DESTDIR}/etc/rc.conf

 # Openstack wants sudo(8) usable by default without a password.
 rm -f ${DESTDIR}/etc/resolv.conf

 # The console is not interactive, so we might as well boot quickly.
 echo 'autoboot_delay="-1"' >> ${DESTDIR}/boot/loader.conf
 echo 'beastie_disable="YES"' >> ${DESTDIR}/boot/loader.conf

 # Reboot quickly, Don't wait at the panic screen
 echo 'debug.trace_on_panic=1' >> ${DESTDIR}/etc/sysctl.conf
 echo 'debug.debugger_on_panic=0' >> ${DESTDIR}/etc/sysctl.conf
 echo 'kern.panic_reboot_wait_time=0' >> ${DESTDIR}/etc/sysctl.conf

 touch ${DESTDIR}/firstboot
 return 0


  • Fetch the FreeBSD source code and extract it under /usr/src
  • Once the code is in place, follow the regular process of build(7) and perform a `make buildworld buildkernel`
  • Change into the release directory (/usr/src/release) and perform a make cloudware

That’s it! This will generate a qcow2 image with 1.4G in size and a raw image of 2G.
The entire process uses the release(7) toolchain to generate the image and should work with newer versions of FreeBSD.

Tested with 11.0-RELEASE-p1 and also tested on OpenStack Mitaka.

Bonus Tips: Installing OpenStack cli tools.

Instal pip:

pkg install py27-pip

Install the OpenStack Python Client:

pip install python-openstackclient

Install the legacy OpenStack Clients (might be needed for some environments).

pip install python-heatclient
pip install python-nova client

Next, create an openrc file and source it. Here’s an example:

unset ${!OS_*}
export OS_AUTH_URL=
export OS_USERNAME=admin
export OS_PASSWORD=NotMyPassw0rd
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=default

Then, source it:

source ~/

Right now, you have all you need to operate OpenStack. To test, let’s try the following:

  1. Upload our newly created FreeBSD image to OpenStack’s Glance.
  2. Spin up a stack that uses our FreeBSD image by invoking a Heat template
  3. Test the connectivity to the new stack
  4. Clean the stack

Let’s go!

With the image now fully created, upload it to glance.

[root@freebsd /usr/src/release]# cd /usr/obj/usr/src/release
[root@freebsd /usr/obj/usr/src/release]# glance image-create \ 
 --progress              \ 
 --disk-format qcow2     \
 --name FreeBSD-OpenStack\
 --file openstack.qcow2  \
 --container bare

Create a heat stack called “freebsd-image-test” using the freebsd-openstack.yaml file.

[root@freebsd ~]# openstack stack create -t ./freebsd-openstack.yaml freebsd-image-test

| Field               | Value                                      |    
| id                  | e217ba55-6892-4012-aebb-e8b2f7032df7       |    
| stack_name          | freebsd-image-test                         |    
| description         | This template generates a FreeBSD instance |    
|                     |                                            |    
| creation_time       | 2016-12-07T01:26:46                        |    
| updated_time        | None                                       |    
| stack_status        | CREATE_IN_PROGRESS                         |    
| stack_status_reason | Stack CREATE started                       |    

Now that the stack is created, let’s get the floating IP of the instance and ssh into it. For this example, I’m using a key named “cse-pubkey”. You can/should change that on the template to reflect your environment.

[root@freebsd ~]# openstack stack output show --all freebsd-image-test 
| Field               | Value                                   |             
| instance_private_ip | {                                       |             
|                     |   "output_value": "",          |             
|                     |   "output_key": "instance_private_ip",  |             
|                     |   "description": "No description given" |             
|                     | }                                       |             
| instance_public_ip  | {                                       |             
|                     |   "output_value": "",     |             
|                     |   "output_key": "instance_public_ip",   |             
|                     |   "description": "No description given" |             
|                     | }                                       |             

Test the connection to the instance.

[root@lppa240 ~]# ssh -i cse_id_rsa
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:LX06d48gTGV+SCrl3DJlWw5i4j7+IjaHDW6J2s/JwzQ.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
FreeBSD 11.0-RELEASE-p1 (GENERIC) #1: Sat Dec  3 10:54:58 EST 2016

Welcome to FreeBSD!

Release Notes, Errata:
Security Advisories:
FreeBSD Handbook:
Questions List:
FreeBSD Forums:

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@freebsd-image-test-instance-qk6lpulwbxp4:~ # uptime
 1:30AM  up 3 mins, 1 users, load averages: 0.27, 0.30, 0.14

All set!

Cleaning the stack

[root@freebsd ~]# heat stack-delete freebsd-image-test


Cloud Infra, OpenStack

OpenStack Neutron Diagrams

<!– –>

TL;DR: Neutron diagrams depicted below are available on my Github repo.

During my presentation on Troubleshooting Neutron Networks (OpenStack Summit 2016 in Barcelona – details and video) there were many network diagrams used to illustrated the various scenarios with Neutron networks.

Many stackers were asking where to get these diagrams and as promised, here they are:


east-west DVR traffic


north-south traffic



east-west traffic with VXLAN

GitHub – dcasati/OpenStack/tree/master/OS2016BCN-Troubleshooting-Neutron

The diagrams can be open using and are made available under the Creative Commons License (CC by 4.0). You can show/hide the different layers and well as edit each component individually.

Have fun!

Cloud Infra, OpenStack, Uncategorized

OpenStack Summit Barcelona 2016


OpenStack Summit in Barcelona is now officially over. The Summit happened between October 25 and 28th in Barcelona, Spain.

This being my first Summit, I was excited about what to see and who I would meet during the conference 4 days.  The fact that I had the chance to deliver two presentations alongside my team members.

Since this was the  OpenStack Summit, we could not go on without, well having  OpenStack, so we brought one own. And it was great ! With us this year, we brought the Dell EMC VxRack Neutrino, a hyperconverged turn-key OpenStack solution. Neutrino really shined during the Summit as more and more Stackers were curious wether or not we  really had a running rack or just a gimmick blinking lights. For folks who are still wondering, we did have the real deal with running workloads on it.

Presentations –

With all of the activities during the Summit, it was hard to see all of them while on site, but I did manage to attend a few of them while also presenting.

Without further ado, here’s what I’ve presented and also some of the interesting presentations that I saw:

Accelerate the OpenStack RCA with Kibana and ElasticSearch (Details)

During this presentation, me and David Sanchez went on to talk about how difficult it is for teams and OpenStack practioners to fully have a Root Cause Analysis (RCA), in time and in a reproducible fashion. We’ve delve  the different aspects of a successful RCA and also how to move from the firefighting mode into a more proactive one.

Troubleshooting Neutron – Physical and Virtual (Details)

In this presentation we’ve went through two real scenarios of troubleshooting Neutron, showing the different components involved by following the data traffic trought the network.

Hands-On Workshops –

Hands-On Workshop: Learn About Microservices Architectures with Docker Swarm, Etcd, Kuryr, Neutron (Details)

    This hands-on workshop was presented by Fawad Khaliq from PLUMgrid who essentially did a great job explaining how Microservices work through theory and examples. During the course of this workshop, Fawad showed showed how one would increase and decrease a swarm cluster and how Docker networking and OpenStack Neutron interface through Kuryr.

Neutron Ninja 101: A Hands-On Workshop with Neutron Networking (Details)

    Becoming a Ninja seems very challening, but Phil Hopkins did a great job here to ensure that we would get there. This presentation was packed with good content and Phil was able to drive the conversation by making sure we all had the necessary networking and Linux foundation. I truly enjoyed his style of presenting and his very clear and concise examples. To my surprise, the last set of slides on this workshop were related to explaing some of the more complex aspects of the OpenStack overlay networking, which as great as our Troubleshooting Neutron: Physical and Virtual Networks presentation started right at this point.

A final cherry-on-top-of-the-cake moment happened after this presentation as I met James Denton whose book, Learning OpenStack Networking (Neutron), was mentioned as a excelent source of reference during our presentation.FullSizeRender.jpg

The highlight of an event like this must be the opportunity one has to meet with fellow Stackers and share stories, knowledge and experience. It’s the whole conversation that happens between the presentations and 1×1 engagement with others.

Happy to have the opportunity to travel to the Summit. See you next time.

Cloud Infra, Uncategorized

OpenStack Images default users


After chasing this information too often, I’ve decided to give a break to my brain and compile a list of the most common OpenStack Images (at least for my day-to-day activities). Without further ado, here’s the list:

Distribution Default User
Ubuntu 14.04 Trusty ubuntu
CentOS 6/7 centos
Red Hat Enterprise Linux 6/7 cloud-user
CirrOS cirros
Debian admin

A complete and up to date list can be found at the OpenStack site.

You can then login into an instance with the provided SSH keys or optionally, you could also change the default user password with the following cloud-config, which will allow you to ssh into an instance but force the password to be changed on the first login:

password: ChangeMe
chpasswd: { expire: True }
ssh_pwauth: True

Let’s understand the options from the above configuration.

# you can set passwords for a user or multiple users
# this is off by default.
# to set the default user's password, use the 'password' option.
# if set, to 'R' or 'RANDOM', then a random password will be
# generated and written to stdout (the console)
# password: passw0rd
# also note, that this will expire the password, forcing a change
# on first login. If you do not want to expire, see 'chpasswd' below.
# So, a simple working example to allow login via ssh, and not expire
# for the default user would look like:
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True

and finally,

# in order to enable password login via ssh you must set
# 'ssh_pwauth'.
# If it is set, to 'True' or 'False', then sshd_config will be updated
# to ensure the desired function.  If not set, or set to '' or 'unchanged'
# then sshd_config will not be updated.
# ssh_pwauth: True

More examples can be found here.

Cloud Infra, SysAdmin, Uncategorized


Have you ever been in a situation where you had to go down to your home lab datacenter to reset a stubborn system? I bet you did and that’s not cool (at all). Luckily, many servers have a way to help you out with this. For me, salvation came in the form and shape of the Intel RMM (Remote Management Module) board.

According to Intel here are the key features:

  • Full remote access keyboard, video and mouse (KVM)
  • USB media redirection
  • Remote power actions
  • Proactive system health monitoring
  • Secure, embedded web server
  • Dedicated network connection

Which translates into pretty much the same experience as one would have by sitting in front of the machine (BIOS access and such). This also means that one could install an operating system remotely and even reboot/shutdown/poweron a server (again, remotely).

The access to this board is usually done through ssh. After you login you will be prompted into some form of shell (a bit more crude), specified by the Systems Management Architecture for Server Hardware Command-Line Protocol (aka SMASH CLP), specified bt the DTMF.

Continue reading