This story dives into the hidden cost and lessons learned in rushing the move to production, and a failure to invest in the underlying architecture that protects your critical workloads. Through an assumption that the new environment is as good as the old one, I spent an entire summer patching issues, chasing runaway objects, and making little progress on anything else.
The Initial Deployment
The thing about goats are, they are the ultimate example of a poorly written application. Left to their own devices, they will go all the places you don’t want them, consume all the plants you want protected, and once they find a weakness in your infrastructure, they will continously to exploit that until it is fixed. They are also the ultimate example of Garbage Cleanup. They will eat almost anything, and especially seek out the foilage that other animals won’t touch, like multiflora rose, poison ivy, and briar bushes. You can place them into your environment, and they will clear out all the unwanted junk that chokes out the useful plants for your other “workloads”.
We had this exact problem, and needed to clear pasture for our eventual cattle. Pefect use case for goats. On top of that, we had existing infrastrucutre that we could put them in. A runtime environment that was proven solid, and we know would be up to the task.
The workloads
In the exploration of deploying our weed-eating workload, my dad told me about an adage with goats, “If It Can Hold Water, It Can Hold A Goat”. The meaning of this was, you plan for infrastructure needs to be rock solid. For certain types of animals, you can skimp on the boundaries, because they are easy going. A small pony, you could put up a peice of string and they’re likely to stay. We felt confident though, and moved forward with purchasing the goats and placing them in the pen. They were deployed to our preconfigured, robust environment with little concern.
Starting out, we had a nice little area for our goats, strong fence, and lots for them to eat. The goats did their job, the small field we had for them was cleared, and everyone was happy. However, as time went on, we need to expand their pen as they were eating their way through the available resources we had they had. Much like expanding the resource limits for a containerized application, we had to provide more room for the goats to continue to eat, and we needed to deploy out newest workloads, cattle, into our freshly cleared field. I had successfully deployed my goats to my pre-existing environment, and was planning move them to the new environment. I started to think “I know more about keeping goats in than my dad!”. He may have 40 years of farming experience, but I kept my goats in easy…… that was a mistake.
Investment, or lack of, in the New Environment
As I mentioned, the failure was in not understanding the workload. The goats would not so easily give up their old pen, where they slept every night and where they had eaten for the past year. When evaluating building infrastructure to run our applications, it’s not enough to understand the process of building, or the material that is needed. You must also understand the underlying applications, the application needs, schedules, and tendancies.
Too often, we see Platform Engineers concentrate on the infrastructure, and ignore the application. It is important to remember that the Platform serves the Application, and success of the Platform is directly tied to a successful deployment and execution of the Applications that exist ontop of it. I would learn the hard lesson that rushing the infrastructure, and failing to spend the time to plan and build robust guardrails would cost me in both time and money.
Poor Infrastructure
So, in my rush to get the goats into their new environment, bought the cheap fence, cheap posts, and strung up a fence on the hillside. It took me a few days to get what I thought would be a sufficient solution in place. Unlike their previous environent, this one was built from the ground up, with no pre-existing solutions in place. I believe that because they ran in the old enviornment, surely this one will work too. I moved the goats from the pre-existing environment to the new one. Two strings of electric fence, some water, surely I’ll be good, right?
It took about 2 whole hours for my newly deployed workload to escape the boundaries that I had set for it. They were out in my yard before I could go back in for dinner. And once they found that they could expliot the weakness in my infrastructure, it would keep happening…. over and over and over and over.
As the summer goes on, they kill my wife’s flowers, they poop on our porch, they sleep in our driveway. I woke up in the morning one time to a goat looking through my window at me. The ultimate noisy neighbor, killing the plants, dirtying up the environment, and crashing when they’re not supposed to. I had a runaway workload, and no possible way to easily retrofit the environment to counteract the holes they were exploiting. Once an Application is in production, it is must more difficult to make sweeping changes in how those Applications are managed. Deploying sweeping changes Production comes with friction, the need for change control, the risk of impacting other things in the environment, the cost of thoroughly testing and validating those changes.
It became perosnal. I spent that entire summer trying to patch fence, add electric strands to decrease the exposed fence line, spending hundreds on electric chargers. Adding more fence posts, more strands, more anything just to get this workload under control. Eventually, you couldn’t tell if I was trying to stop goats from escaping, or the T-Rex from Jurassic Park. I had completely over engineered the environment, but to little success.
The reason is, I was trying to retrofit a now pre-existing environment for a workload that it was never properly designed to handle. Once you’ve opened the environment up, and then try to bolt on additional guardrails because you’ve under engineered it, you simply expose the ugly under belly of the solution. My solution was never designed to withstand the requirements that this workload came with. I never understood how stubborn, how creative, and how persistent they would be at testing the edges of my solution. I knew how to build A fence, but I didn’t think about the proper fence for what I was going to be placing into this environment. Then, trying to turn that solution into the proper solution without ripping it out just led to disjointed and illfitted add-ons that cost money but did nothing for efficiency.
Bad Neighbor
Now picture this, you’re the wife of a farmer, drinking your afternoon coffee. It’s 93 degrees outside, and you look in the yard. You see a goat sprinting by, and then your farmer husband sprinting with a net above his head. He catches one, puts it back into the pen, and another escapes behind him.
Eventually, I just gave up, they did what they wanted, and I tried to mitigate the damage. I moved them back to the old environment, we ripped out the entire solution, and I built it properly from scratch.
No Kidding, the Bahhhhd Truth
The old adage of, you can do it fast, cheap, or good…. pick 2. I chose fast and cheap, and it ended up being an absolute poor fit that ended up costing me more in the long run. There are many lessons we can take away from this story. Platform Engineering isn’t just about building infrastructure. It’s about understanding the differences in what you need to provide from environment to environment. What Applications will be running in your Platform, what capabilities do they need, and how much room do they need to expand. My new pen looked great, until I placed a workload in it. Had I been putting a horse in there, it probably would have been sufficient. However, I wasn’t, and my failure to take time to understand the workload requirements meant I couldn’t account for their tendancies in my initial design.
There is a time investment that must occur up front, before any workloads ever get scheduled. Do you have the proper controls in place to enforce standardization, such as Policy Enforcement around workload declaration to enforce resource limits? Do you have proper observability, so that you don’t find out your workload is running amok due to your wife finding her tulips as just stumps on the ground? Is the automation in place to seamlessly move that workload from the old environment to the new, to add new workloads in a controlled and standardized way, and to simplify the processes that manage and maintain that environment? You can build a pretty looking environment without much substance, but you will truly test those boundaries when you start putting workloads into it, and any weakness in the architecture is exposed. You’ll never know where those workload may test your infrastructure if you don’t take the time to understand their tendancies.
Too often, leads look to implement the new thing before taking time to prepare the infastracture and teams building that infrasructure. Whether that is failing to invest in the proper hardware, or failing to provide adequate time to train the engineers who will support the new environment, or even allowing for the process to build the guardrails to provide a stable and consistent environment with which to run. When that happens, the engineers spend all their time chasing errors, runaway processes, and issues in the environment. Add to that the additional money spent on trying to patch a losing solution, all that time wasted trying to find quick fixes to a situation that inevitably would never work at scale. As a leader and decision maker, we must make sure to present the downfall of a lack of investment and preparation for a big move, while making sure to evangelize the longterm benefit in proper investing in planning and execution. We must fight for that investment in time, effort, and hardware. On top of what we’re already dealing with, AI workload implementation will challenge our understanding of what is necessary in the infrastructure, and test new boundaries and configuration to expose those weaknesses. Whether that is new scheduling requirements, changes in howe we observe and react, or execution concurrency on hardware.
There are hidden opportunity costs to this lack of investment as well. There is A LOT I can accomplish on this farm. Improving the pig load out area, increasing the number of cattle we have, improving my water distribution system in the barn, or pasture management to improve grass quality. Instead of doing any of those, which would either increase automation to free up time, or improve quality and output from a financial perspective, I was chasing fixes for a solution that was not generating any net benefit at the time. This is the true cost, taking development and engineering time away from new features, and having them directed towards just trying to keep the existing runtimes operational. Lowering the overall speed to market due to rushing a solution to production that wasn’t ready.
How many of your engineers are sprinting across the yard with a net trying to catch that runaway workload simply because we avoided investing in the time and cost of properly configured environment? How many developers have given up on solving the problem and are simply just trying to limit the damage? Your infrastructure isn’t just an environment, but it’s the boundary that keeps the workloads running where they should, when they should, and how they should. The lesson here is, buy the right fence early, build it strong, save yourself the headache, and allow your teams to move on to the next big thing.
Results of Proper Planning