Code Deployments and Security

A simple post focusing on what CI/CD methods I think are worth spending time on
and things you can do to avoid letting it out in the open for hackers

The post is specific to server based apps, I will make a separate post about
mobile app setups in a different post.

Here's a few CI/CD methods that I know of and the one's we'll be talking about

Self Hosted CI/CD and Deployments via SSH

This is the traditional way of doing deployments. The idea is to have a bastion
host and then this host has access to your deployment environments and takes
care of running builds and pushing the builds to the needed server or even
better, triggering a build on the needed server and monitoring the progress.

A good example is the a Jenkins setup

Quick Note: Never, I mean never let the bastion host be open via default
ports, and obviously, shut down the 80 port as well, use a random set of ports
for both SSH and HTTP.

The Cons

The Pros

Docker Containers and Deployments via SSH

With everything moving to cloud native setups this is a lot more common around
devs and this mainly involves the Runners / Container that we can find with most
CI/CD services today, examples would be Buddy Works, Gitlab CI, Github Actions,
Circle CI etc etc

These basically give you a isolated container that you can run your script in
and thus reduces the overlap of past caches (does support caching if you really
need it) effecting a fresh build. These are amazing and obviously a lot more
scalable since each runner can run a deployment and you are still limited as to
how many you can based on the service you use but considering Github Actions and
Gitlab Runners I've had 3-4 run in parallel and I can't really complain with
that count on a free plan.

The Cons

The Pros

AWS's Systematic deployments

The nightmare of deployment setups, you start with 100 services and then link
those services and then hope everything works but no you forgot to change the
security group for one service and now you'll have to fix that but oh no you
forgot which service was to be restarted to make it run so you restart one by
one and it's been 4 hours.

Sarcasm aside. A major contender in terms of a Dev Ops environment and
technically they've taken care of all security concerns I normally worry about
but still the amount of setup it takes is better off done with terraform config.
Manually is really going to take a lot of time, there's a reason terraform
exists and devops love it.

anyway, no cons and pros here because it's basically a good standard as long as
the ports are left on default. Mail me stuff you think should be added to
improve security here and I'll test it out and edit the post as needed.

Git style deployments (Heroku, Dokku, etc)

This is one of my favorite ways of doing deployments since there's no involvment
of keys , heroku can hook onto your git repository to and take care of deploying
on changes.

On the other hand, there's Ansible which can also handle this while using a
local ssh key so your private key isn't gifted to any service.

Dokku is another alternative and works in a similar fashion and you can push to
a dokku repository and it'll build it there for you.

The only flaw is heroku since it is protected with email and password and people
use amazing passwords and don't add 2fa so .... for that i'll blame the
developer and not the service but if someone hacks on the service, you pretty
much loose your source code and not someone who hacks on all your data because
in these cases the data is a separate service.

One other problem is the rollback mechanism but it's only on dokku since I have
to trigger another rebuild, I can use plugins to cache the rollback build but
doesn't provide it by default so I'm going to rant about it. Kidding, it's a
simple fix plus has enough plugins to take care of a lot of things.

You can't modify a simple line and restart server to test it so this is going to
need a commit and push to work which might not be ideal for services that need
to quickly monkey patch security vulnerabilities but works for most other
scenarios.

The Cons

The Pros

Security

While I would love for everyone to go ahead with heroku and/or dokku a lot of
people already have a setup and wouldn't like to change it just because I said
that something else is better, so instead lets try to fix the minor issues we
have with these setups.

The first setup with bastion has the problem of being able to do anything and
everything with the connected resources. This can be solved by

"Reaper, they can hack into the password too!", yes they can.

But, the password will take a significant time to crack , you can have logs of
those attempts stored to monitor if someone unauthorized tried to log in. Also,
the password attempts can be limited on most ssh daemons so that adds in even
more blockage.

The same solution can be used for Docker Container Runners like Gitlab and
Github where you provide your private key to the runner.

HOW DO I LIMIT IT TO EXECUTE ONE SCRIPT !?

Good question.

There's a file called ~/.ssh/authorized_keys where you add public keys that
can access the ssh daemon and connect to the user. What you can do is add a few
parameters before this key to limit what the key has access too. You can find
all the parameters on the
authorized_keys manual
from openssh and one of them is command . Basically you can provide a string
of command(s) that whenever the ssh-key connects, the system is to execute it.
So, in our case we are going to add a command that runs the execute-deploy.sh
script.

So, it'll look a little something like

command="echo 'hello'" ssh-rsa [.....]

Now everytime the holder of this key tries to ssh, he will just see hello and
the session will end , if he sends any other command, that'll also just execute
hello and end the ssh session.

In a real life scenario you are going to have something in the lines of :

command="bash -ic /home/reaper/execute-deploy.sh" ssh-rsa [.....]

To break it down

-c : run the commands from the following string

-i: create an interactive shell, which if you don't provide, will try to run
the script in a plain tty environment and all the commands/binaries and custom
path setups you made on the server might not work in the script you execute.

Now all I need to do is pass this private key to the Runners and if the runner
tries to connect, it executes a deploy. If gitlab or github ever leak the
private key, the max he can do is execute the script unless he brute forces his
way into the server and then decides to kill everything, sadly, I don't have the
knowledge to prevent that right now.

You'll know when I have more ways to help improve things like these.

As for now,

Adios!