Efficient consumption of cloud services is all about automating the up and down-scaling of resources according to the ever-changing needs. The list of tools, their properties and their fitness for the purpose can be daunting.
On top of that comes the changing landscape in licensing and subscription, sometimes with astonishing effects on your existing cloud strategy, recently exemplified by Hashicorp’s unexpected license change to the Business Source License (BSL). In this landscape, we must always be prepared to adapt and change, thus, it is good to have acquaintance with both old established tools and some alternatives. This post is the first in a series on how to utilize Pulumi to automate service consumption towards the Safespring cloud APIs.
Prerequisites
- Basic understanding of Python programming language and the concepts of declarative infrastructure as code (IAC). This presentation by Lee Briggs held in the Configuration management camp in Ghent 2023 may help to straiten out some concepts.
- An account with Pulumi SaaS for storing and managing infrastructure stacks (state).
- A Pulumi-supported operating system with API access to one of Safespring’s datacenters, typically Oslo or Stockholm.
API access
The APIs are firewalled, so either you need to operate from an instance (jumphost) which is already in a Safespring data center (then firewall openings are already in place). If not you must send an email to support@safespring.com and ask to whitelist the source IP of the host/CIDR you will be using Pulumi from.
The command curl ifconfig.me will tell you
your public IP as seen by the API firewall. This is useful especially if
you access the API through NAT. Also, please tell us if the address(es) changes
so we can open the new one and close the old one.
Setting up Pulumi
We recommend the manual installation for Linux because we advise against piping content from the Internet directly to a shell. Of course, the manual installation is pretty easy to automate according to your security polices too.
When having a working Pulumi executable on your local machine (in this case a Ubuntu 22.04 jump host), you must log in to the Pulumi SaaS in order to store track records of your resources and their history. In a later blog post, we will see how to change the stack storage to a Safespring S3 bucket, thus, running a standalone setup of Pulumi.
ubuntu@demo-jumphost:~$ pulumi login
Manage your Pulumi stacks by logging in.
Run `pulumi login --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account/tokens
or hit <ENTER> to log in using your browser :
Welcome to Pulumi!
Pulumi helps you create, deploy, and manage infrastructure on any cloud using
your favorite language. You can get started today with Pulumi at:
https://www.pulumi.com/docs/get-started/
Tip: Resources you create with Pulumi are given unique names (a randomly
generated suffix) by default. To learn more about auto-naming or customizing resource
names see https://www.pulumi.com/docs/intro/concepts/resources/#autonaming.
Logged in to pulumi.com as JarleB (https://app.pulumi.com/JarleB)
ubuntu@demo-jumphost:~$
Before proceeding, we need to install a dependency for the Pulumi OpenStack Python project template:
$ sudo apt update && sudo apt install python3.8-venv
Then create a new Pulumi project and search for OpenStack and choose
openstack-python:
ubuntu@demo-jumphost:~/pulumi$ pulumi new
Please choose a template (37/220 shown):
opens [Use arrows to move, type to filter]
openstack-go A minimal OpenStack Go Pulumi program
openstack-javascript A minimal OpenStack JavaScript Pulumi program
> openstack-python A minimal OpenStack Python Pulumi program
openstack-typescript A minimal OpenStack TypeScript Pulumi program
openstack-yaml A minimal OpenStack Pulumi YAML program
project name: (pulum) pulumi-demo
project description: (A minimal OpenStack Python Pulumi program)
Created project 'pulumi-demo'
Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)
Created stack 'dev'
Installing dependencies...
Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
Collecting pip
(...)
Next, we need to export the necessary environment variables and/or clouds.yaml
config file. For now, the simplest way is to log in to the web GUI of OpenStack
(Horizon) and download the OpenStack rcfile from the menu Project -> API access -> Download OpenStack RC file.
Then source that file:
ubuntu@demo-jumphost:~/pulumi$ source ~/.sandbox.safespring.com-openrc.sh
Please enter your OpenStack Password for project sandbox.safespring.com as user jarle@safespring.com:
ubuntu@demo-jumphost:~/pulumi$
You can also use application credentials if you don’t want to expose your personal password in the environment.
At this moment, we are ready to start creating resources by means of a Pulumi
program written in Python. However, Pulumi programs need to know some
parameters to maintain resources. The easiest way of quickly getting
this information is to use the OpenStack CLI. OpenStack CLI already uses the
sourced environment as configuration so we only need to install the OpenStack
CLI in order to use the openstack command for this purpose.
ubuntu@demo-jumphost:~/pulumi$ sudo apt-get install virtualenvwrapper
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
mkvirtualenv oscli
(oscli) ubuntu@demo-jumphost:~/pulumi$ pip install --upgrade piping
(oscli) ubuntu@demo-jumphost:~/pulumi$ pip install python-openstackclient python-neutronclient
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2023-08-25T18:54:44+0000
(...)
The token issue commend verifies that the OpenStack CLI is configured correctly towards the OpenStack API. Pulumi will use the same configuration.
Creating an instance with Python/Pulumi
After setting Pulumi with the OpenStack template, we see a file called
__main__.py in the Pulumi working directory. This is sample file created by
the templating process.
"""An OpenStack Python Pulumi program"""
import pulumi
from pulumi_openstack import compute
# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('test',
flavor_name='s1-2',
image_name='Ubuntu 16.04')
# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)
In order to create an instance in Safespring, we just need to substitute the example parameters with values that match the Safespring platform. You can use the OpenStack CLI to get this information.
To get a list of available flavors, use:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack flavor list
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| 02e49e88-dfd3-4a41-b6f6-5c95ef8364cf | b2.c16r32 | 32768 | 0 | 0 | 16 | True |
| 117ccf62-1758-4300-ab1b-b5ba78948346 | l2.c8r16.100 | 16384 | 100 | 0 | 8 | True |
| 121bfee4-8d70-4668-81b6-0a755e022fd1 | l2.c16r32.500 | 32768 | 500 | 0 | 16 | True |
| 1a58ba25-8444-4fb9-a2a8-4f8027cd0f59 | l2.c8r16.1000 | 16384 | 1000 | 0 | 8 | True |
| 1da5b6f6-d722-4e83-b507-2d4ea6d3ca7d | l2.c16r32.1000 | 32768 | 1000 | 0 | 16 | True |
| 52d06354-6cd0-42fa-a7e8-2998647db65d | l2.c2r4.1000 | 4096 | 1000 | 0 | 2 | True |
| 52f4bc1c-973b-409e-aea1-23f7e2d14b27 | b2.c2r4 | 4096 | 0 | 0 | 2 | True |
| 5748eadf-a45f-41f3-ba4e-dd03973ceed3 | b2.c1r4 | 4096 | 0 | 0 | 1 | True |
| 601cf092-db6d-4faa-b456-0e8613a0c9dc | l2.c2r4.500 | 4096 | 500 | 0 | 2 | True |
| 62502c5c-9441-4546-8859-c243a506da31 | b2.c2r8 | 8192 | 0 | 0 | 2 | True |
| 79a8fc06-7385-490f-86b7-4daf178b6590 | l2.c16r32.100 | 32768 | 100 | 0 | 16 | True |
| 7cade287-87a1-4bdf-a92b-a4208101895d | b2.c1r2 | 2048 | 0 | 0 | 1 | True |
| 8574373f-b266-400c-80b1-49027c97bdcb | l2.c32r64.1000 | 65536 | 1000 | 0 | 32 | True |
| 8f84ceab-89c1-4dfb-9ef6-97504475bd3a | l2.c4r8.500 | 8192 | 500 | 0 | 4 | True |
| 9268de17-2d5b-4885-bc53-155f093aed6d | l2.c2r4.100 | 4096 | 100 | 0 | 2 | True |
| a697753c-12ef-4abf-8c1d-f3bef761ffb7 | l2.c4r8.100 | 8192 | 100 | 0 | 4 | True |
| b10d4f41-6ca4-4dae-8fec-7580cdd2a1dd | b2.c8r32 | 32768 | 0 | 0 | 8 | True |
| b4d75d91-f3b4-4ad1-b859-0306064856d8 | l2.c8r16.500 | 16384 | 500 | 0 | 8 | True |
| d2fc99a7-85da-49dd-9725-6670086a1aa9 | b2.c16r64 | 65536 | 0 | 0 | 16 | True |
| e91ff4b7-cf9e-4d95-8374-aa3b1d765200 | l2.c4r8.1000 | 8192 | 1000 | 0 | 4 | True |
| eb1d6bec-60ab-4a6b-95e9-313e33dd6712 | b2.c4r8 | 8192 | 0 | 0 | 4 | True |
| f448fae2-135d-4865-a8d3-8306cc1a119e | b2.c4r16 | 16384 | 0 | 0 | 4 | True |
| f578ce2f-2a60-4803-8274-a4b92a44a227 | b2.c8r16 | 16384 | 0 | 0 | 8 | True |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
(oscli) ubuntu@demo-jumphost:~/pulumi$
To get a list of available images, use:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack image list
+--------------------------------------+------------------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+------------------------------------------------+--------+
| d007510b-908a-44a5-a1a5-b2dc8e751260 | debian-10 | active |
| f2ef69eb-2856-4319-95f5-902f43fccef8 | debian-11 | active |
| a7394047-8f79-4b2c-92aa-d4a818ef42c2 | debian-12 | active |
| ee81e161-d04c-40c0-a848-582750f9903b | ubuntu-18.04 | active |
| cfc0ca97-780b-4fa5-aa87-44e4f41a0766 | ubuntu-20.04 | active |
| aac74808-9dba-4f49-a530-70a23b4163f3 | ubuntu-22.04 | active |
+--------------------------------------+------------------------------------------------+--------+
(oscli) ubuntu@demo-jumphost:~/pulumi$
You can also upload your own image.
And finally for networks:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack network list
+--------------------------------------+------------------+----------------------------------------------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+------------------+----------------------------------------------------------------------------+
| 14ff54e0-80e4-492b-a54a-8c4d4097ed8f | default | 1ae2aebe-542a-410a-8fea-bf1f941d6d6a, 34489b94-634a-45cd-bac9-61deea3daf5a |
| 33dc493f-f4d5-4ab4-bf8e-43bee3faf3ef | public | 368db41d-c77f-4759-8113-d702818702fd, 5d1e4008-7a1a-4c88-9b0c-7d0faf54a9d8 |
| 67892ac3-1dcd-4bba-bd60-28b5d037f6ff | private | 059d94a0-0fc1-40dd-9814-eb00571c6a4d, 6ff36feb-deb9-4cc0-aa09-8007006988bb |
+--------------------------------------+------------------+----------------------------------------------------------------------------+
So we choose to change the __main__.py file like this:
"""An OpenStack Python Pulumi program"""
import pulumi
from pulumi_openstack import compute
# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('pulumi-demo',
flavor_name='l2.c2r4.100',
networks=[{"name": "public"}],
image_name='ubuntu-22.04')
# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)
Note that we only attach one network even though it is technically possible to
attach several networks with the networks parameter being a list data type.
This is because in the Safespring platform, there is no need for attaching
multiple interfaces; it could even create instability and problems.
To read about why this is the case, please read the blog post about
Safespring’s network model
Finally, we can run pulumi up in order to build the resource graph, and
subsequently apply that graph to the OpenStack API with Pulumi.
(oscli) ubuntu@demo-jumphost:~/pulumi$
(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/ec8d68fc-15a6-4dd3-8add-0b5076170bfd
Type Name Plan
pulumi:pulumi:Stack pulumi-demo-dev
+ └─ openstack:compute:Instance pulumi-demo create
Outputs:
+ instance_ip: output<string>
Resources:
+ 1 to create
1 unchanged
Do you want to perform this update? yes
Updating (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/3
Type Name Status
pulumi:pulumi:Stack pulumi-demo-dev
+ └─ openstack:compute:Instance pulumi-demo created (15s)
Outputs:
+ instance_ip: "212.162.146.151"
Resources:
+ 1 created
1 unchanged
Duration: 17s
(oscli) ubuntu@demo-jumphost:~/pulumi$
So, let’s see if the instances was created by using the OpenStack CLI:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pulu
| 94593df2-eae9-40cf-bfba-7f8078bae970 | pulumi-demo-0d3a61e | ACTIVE | public=212.162.146.151, 2a09:d400:0:1::1d9 | ubuntu-22.04 | l2.c2r4.100 |
(oscli) ubuntu@demo-jumphost:~/pulumi$
And sure, it was! Note, that since we did not specify a name, Pulumi created
one for us using pulumi-demo as prefix and a random string as suffix.
If we care about the name of the instance, we can just add it to the Pulumi program like this:
"""An OpenStack Python Pulumi program"""
import pulumi
from pulumi_openstack import compute
# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('pulumi-demo',
name = 'pulumi-demo', # < ---- here
flavor_name='l2.c2r4.100',
networks=[{"name": "public"}],
image_name='ubuntu-22.04')
# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)
And then we apply the change:
(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/7a022268-94e2-4f79-9b20-9f0010563b57
Type Name Plan Info
pulumi:pulumi:Stack pulumi-demo-dev
~ └─ openstack:compute:Instance pulumi-demo update [diff: ~__defaults,name]
Resources:
~ 1 to update
1 unchanged
Do you want to perform this update? details
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:dev::pulumi-demo::pulumi:pulumi:Stack::pulumi-demo-dev]
~ openstack:compute/instance:Instance: (update)
[id=94593df2-eae9-40cf-bfba-7f8078bae970]
[urn=urn:pulumi:dev::pulumi-demo::openstack:compute/instance:Instance::pulumi-demo]
[provider=urn:pulumi:dev::pulumi-demo::pulumi:providers:openstack::default_3_13_3::4cf9816e-9eb8-447f-9c15-4d5614b3c329]
~ name : "pulumi-demo-0d3a61e" => "pulumi-demo"
Do you want to perform this update? yes
Updating (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/5
Type Name Status Info
pulumi:pulumi:Stack pulumi-demo-dev
~ └─ openstack:compute:Instance pulumi-demo updated (2s) [diff: ~__defaults,name]
Outputs:
instance_ip: "212.162.146.151"
Resources:
~ 1 updated
1 unchanged
Duration: 4s
(oscli) ubuntu@demo-jumphost:~/pulumi$
And now the name was changed:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pulu
| 94593df2-eae9-40cf-bfba-7f8078bae970 | pulumi-demo | ACTIVE | public=212.162.146.151, 2a09:d400:0:1::1d9 | ubuntu-22.04 | l2.c2r4.100 |
(oscli) ubuntu@demo-jumphost:~/pulumi$
Connecting to the instance
In order to access the instance, though, we need to open up some ports using security groups and rules.
Let’s add this code to the Pulumi python program:
"""An OpenStack Python Pulumi program"""
import pulumi
from pulumi_openstack import compute
from pulumi_openstack import networking
sg = networking.SecGroup('pulumi-sg',
name = 'pulumi-sg')
ssh_rule = networking.SecGroupRule("pulumi-ssh-ingress",
direction="ingress",
ethertype="IPv4",
protocol="tcp",
port_range_min=22,
port_range_max=22,
remote_ip_prefix="0.0.0.0/0",
security_group_id=sg.id)
instance = compute.Instance('pulumi-demo',
name = 'pulumi-demo',
flavor_name='l2.c2r4.100',
networks=[{"name": "public"}],
security_groups=[sg.name], # <- New parameter for security group membership
image_name='ubuntu-22.04')
And then run pulumi up again:
(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/5ee8bfa0-7cf3-4e54-9c38-534de190f776
Type Name Plan Info
pulumi:pulumi:Stack pulumi-demo-dev
+ ├─ openstack:networking:SecGroup pulumi-sg create
~ ├─ openstack:compute:Instance pulumi-demo update [diff: ~securityGroups]
+ └─ openstack:networking:SecGroupRule pulumi-ssh-ingress create
Resources:
+ 2 to create
~ 1 to update
3 changes. 1 unchanged
Do you want to perform this update? yes
Updating (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/17
Type Name Status Info
pulumi:pulumi:Stack pulumi-demo-dev
+ ├─ openstack:networking:SecGroup pulumi-sg created (1s)
+ ├─ openstack:networking:SecGroupRule pulumi-ssh-ingress created (0.57s)
~ └─ openstack:compute:Instance pulumi-demo updated (4s) [diff: ~securityGroups]
Resources:
+ 2 created
~ 1 updated
3 changes. 1 unchanged
Duration: 8s
(oscli) ubuntu@demo-jumphost:~/pulumi$
Now we can connect on port 22:
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pul
| be203992-22ce-4f60-900b-d3b47b39262f | pulumi-demo | ACTIVE | public=212.162.147.112, 2a09:d400:0:1::321 | ubuntu-22.04 | l2.c2r4.100 |
(oscli) ubuntu@demo-jumphost:~/pulumi$ nc -w 1 212.162.147.112 22
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
(oscli) ubuntu@demo-jumphost:~/pulumi$
Summary
We have seen that we can use Python (or other imperative programming languages) to deploy and maintain infrastructure resources in a similar fashion to Terraform. One thing to be aware of, though, is the mix between declarative and imperative approaches. In some sense, the Pulumi program behaves like a sequential recipe of tasks even though it eventually builds a similar resource graph (state) as Terraform. This means that if we change the order of events in the program by placing the instance creation before the security group creation, it will fail because the definition of the security group, that the instance will be a member of, is not yet defined in the program sequence. This is different in Terraform, where the ordering of the code statements is completely irrelevant.
In the next blog post, we will be scaling things up and separating the program configuration data out into yaml-files..
