Going on from my developer-led setups, I have recently decided at my organisation to challenge the current setup, and I will talk through it here.
The current status
A while ago when my current employer had onboarded a recent merger, we had some DevOps team members join and they ended up creating a “temporary” development environment. But the problem is, nothing is more permanent than a temporary solution that works, even if it is poorly executed.
This setup has always annoyed me, because when I looked at it there was a few major red flags that people dealt with just as a comment on ‘this is how it is’.
- Every single service is managed within this one “environment” repository.
- Maintaining requires extensive knowledge of makefiles.
- Structure is assumed
My proposed alternative
A few weeks back, between other pieces of work, I decided to throw together a new alternative. A challenger to the status quo. The idea swings around a few particulars, no makefiles, no custom generating any docker compose yaml files and finally flexible structure.
Though there are some things that I enforced as “these are non-negotiables” that I decided to go with
- Using
.internalas the local development domain (this comes from ICANN reserving.internalfor this case.) - During the early beta phase, some structure assignments are fine, though this is not in final state.
I stuck with the zero makefile structure, but ended up finding out that I needed to have some other way to handle a bunch of tasks. It ended up turning out that when I needed to have some common tasks that the projects needed, but I ultimately wanted to ensure that each project had its own control of how it is ran.
Locality of change
One problem that comes up with scale, when developers use their own setups, is if one major change happens, we should not need to go to other setups to update for everyone. If a service in itself does not own how it is ran, this is where the problem rears its head the most.
So the simple answer is, make the service own its own setup, its own configuration and own ways of working.
Why not adapt the status-quo
Adapting is something that came to mind, but this, to me became more an architectural challenge, as I would need to go through and unwind the current setup with how it was done, then introduce it correctly, and with the fact we already had a lot of developer experience related issues, why not take the moment to provide an alternative and propose a better way of working. Or at least that’s what I was telling myself.
Why not Sail?
While most of our applications might use Laravel, we don’t only use it, some of our applications are a hybrid of legacy PHP and Laravel, as well as Python and other projects, so we needed to ensure that we had a solution that covered everything, something as capable as Sail, but flexible enough for other languages & environments.
My approach
In the end I decided to utilise the very powerful tool Mise-en-place, this was the tool that ultimately provided the shared set of tools to use.
But other than tooling there was some assumptions to look at.
- We need to match production & end goals as closely as possible.
- How will services communicate?
- How can we maintain local developer experience.
Where I work, we have been trending towards dockerising our major deploys slowly in future environments, they are currently VM based workloads that are internally managed, but new things have started to become docker managed.
Given that we have docker, the communication method would be easiest to utilise docker’s inter-container networking, so defining a common network to use on developer machines would be the simplest choice.
This would also allow containers that need to be isolated from the main stack to define their own network and opt in for joining when needed.
We have a core stack of services, internal, customer portals a & b, and also a payment portal, so we define a ‘core’ network.
architecture-beta group main(simple-icons:docker)[Core] service traefik(simple-icons:traefikproxy)[Traefik] in main service internal(simple-icons:laravel)[Internal] in main service customer-a(simple-icons:laravel)[Customer Portal A] in main service customer-b(simple-icons:laravel)[Customer Portal B] in main service payments(lucide:circle-dollar-sign)[Payments] in main service host(lucide:computer)[Computer] traefik:R -- L:host traefik:L -- R:internal traefik:L -- R:customer-a traefik:L -- R:customer-b traefik:L -- R:payments align column internal customer-a customer-b payments align row traefik host
You will note that all services ultimately are accessed via Traefik as the edge ingress within docker, which its primary use is TLS termination via a local step-ca instance running within docker that provides just-in-time and automatically renewing certificates.
If a service, needs to communicate within another service, they don’t route through Traefik just for SSL termination, it will directly communicate between containers using the standard docker inter-container networking, as they all stay in one network.
Developer Experience
Developer experience was probably my biggest usage of mise-en-place and why I decided to do this. Utilising Laravel and Vue primarily, meant that we have a bunch of developers that are very familiar with the terminal and utilising composer install and php artisan as CLI tooling.
I didn’t want to have them learn a whole new way of doing docker compose exec -it <service> composer install ... since that is a pain to remember. So my approach was to ensure I included a whole bunch of mise interactive tasks that could take over, so composer install became mise composer install and php artisan became mise artisan.
I didn’t neglect php directly, as I created tasks for that, but I wanted to ensure that all things had a task to provide this experience.
New stacks
If another stack of services comes up to play, they have a few choices. They could join the main network, allowing any service to communicate This here is the simplest side of things, but it doesn’t give true isolation and name separation, alternatively, they can define their own network name & join only services they want exposed through Traefik into the “core” network.
Demo phase
This turned out to be my biggest question and challenge, when I was actually wanting to demo this to my colleagues, I didn’t want to have to go into long-standing, existing projects, force my new proposed way of working into their systems just for something that might be pushed back.
So I came up with an “Overlay” concept. This is primarily inspired by the ideas that OverlayFS uses, where inside the docker containers, the resulting build is the actual code checkout plus my new proposed set of changes.
This ended up coming down to a mise.toml containing all the tasks that would be used. Dockerfile with how everything will be built, and some compose.yaml files.
This does enforce a temporary failure of my current state where I wanted absolute flexibility to where files might exist, but for a demo period, I consider this fine.
The other winning factor I consider in my demo, is the first time setup, one issue always happened to be for setting up a new developer, we needed to hand them both a documentation page with “here’s how you set up the entire world” as well as a new, fresh laptop. And if you have started a new job lately, that starting of a new job is a daunting period of time, so I utilised this time, to create a Go & Bubble Tea application that guided people with the setup, telling them where to get any authentication details, local CA trust, local DNS, setting up access tokens & pulling down the repos.
Additionally to my onboarding flow, I made it so that we can define a dependency graph of what one service requires. Since not every developer will
work on your whole platform, why would we need to make it so they are running it? If you need to run frontend a then make sure only the dependency
chain down is installed. internal service c may not be needed - so why install it then?
Feedback
The first thing I did was communicate with a couple of colleagues who I knew and could get open communication with letting them know I have been working on this. These colleagues were using the setup for a few weeks before the first demo that I gave the wider team. This, to me was the most critical, you can make assumptions to how you work, but that does not apply to everyone. My request for feedback was very broad, I asked for “anything that doesn’t feel right” or even anything that was unintuitive, give me this feedback and we started making strides.
I also had those colleagues as the first collaborators in the project, where they were able to help the resilience and tease out some bugs & quirks of local setups.
The whole team
Once I had worked out the initial quirks and experience problems and therate of bug fixes had slowed down, I announced it to the rest of the team with “Hey! I’ve been working on this, anyone else want to try it out?” Once announced, this came back with some amazing positive feedback and some colleagues jumped on it immediately, some would take their time when they found it, but in the most part, due to my initial phasing, this made it so almost zero changes were needed, other than adding new services.
I deliberately chose not to clone the entire stack, but focused on a core set of services for this case. To assist with adoption and trialling, I made sure that there was enough documentation and my own personal availability there to explain the concepts behind the setup and how to adopt this in new servicesu.
Conclusion
It’s hard to have a complete conclusion since this is still an active change that I have been effecting at the business, but so far I have detailed some of the rationale and choices behind it. The next stages here are wider adoption and once the team is happy, slowly unshipping that OverlayFS inspired setup for direct-in-repo changes.
So far the feedback I have had from the wider team has been very positive and they are liking this new way of working, so the gamble seems to have paid off.