[{"content":"This story dives into the hidden cost and lessons learned in rushing the move to production, and a failure to invest in the underlying architecture that protects your critical workloads. Through an assumption that the new environment is as good as the old one, I spent an entire summer patching issues, chasing runaway objects, and making little progress on anything else.\nThe Initial Deployment The thing about goats are, they are the ultimate example of a poorly written application. Left to their own devices, they will go all the places you don\u0026rsquo;t want them, consume all the plants you want protected, and once they find a weakness in your infrastructure, they will continously to exploit that until it is fixed. They are also the ultimate example of Garbage Cleanup. They will eat almost anything, and especially seek out the foilage that other animals won\u0026rsquo;t touch, like multiflora rose, poison ivy, and briar bushes. You can place them into your environment, and they will clear out all the unwanted junk that chokes out the useful plants for your other \u0026ldquo;workloads\u0026rdquo;.\nWe had this exact problem, and needed to clear pasture for our eventual cattle. Pefect use case for goats. On top of that, we had existing infrastrucutre that we could put them in. A runtime environment that was proven solid, and we know would be up to the task.\nThe workloads\nIn the exploration of deploying our weed-eating workload, my dad told me about an adage with goats, \u0026ldquo;If It Can Hold Water, It Can Hold A Goat\u0026rdquo;. The meaning of this was, you plan for infrastructure needs to be rock solid. For certain types of animals, you can skimp on the boundaries, because they are easy going. A small pony, you could put up a peice of string and they\u0026rsquo;re likely to stay. We felt confident though, and moved forward with purchasing the goats and placing them in the pen. They were deployed to our preconfigured, robust environment with little concern.\nStarting out, we had a nice little area for our goats, strong fence, and lots for them to eat. The goats did their job, the small field we had for them was cleared, and everyone was happy. However, as time went on, we need to expand their pen as they were eating their way through the available resources we had they had. Much like expanding the resource limits for a containerized application, we had to provide more room for the goats to continue to eat, and we needed to deploy out newest workloads, cattle, into our freshly cleared field. I had successfully deployed my goats to my pre-existing environment, and was planning move them to the new environment. I started to think \u0026ldquo;I know more about keeping goats in than my dad!\u0026rdquo;. He may have 40 years of farming experience, but I kept my goats in easy\u0026hellip;\u0026hellip; that was a mistake.\nInvestment, or lack of, in the New Environment As I mentioned, the failure was in not understanding the workload. The goats would not so easily give up their old pen, where they slept every night and where they had eaten for the past year. When evaluating building infrastructure to run our applications, it\u0026rsquo;s not enough to understand the process of building, or the material that is needed. You must also understand the underlying applications, the application needs, schedules, and tendancies.\nToo often, we see Platform Engineers concentrate on the infrastructure, and ignore the application. It is important to remember that the Platform serves the Application, and success of the Platform is directly tied to a successful deployment and execution of the Applications that exist ontop of it. I would learn the hard lesson that rushing the infrastructure, and failing to spend the time to plan and build robust guardrails would cost me in both time and money.\nPoor Infrastructure\nSo, in my rush to get the goats into their new environment, bought the cheap fence, cheap posts, and strung up a fence on the hillside. It took me a few days to get what I thought would be a sufficient solution in place. Unlike their previous environent, this one was built from the ground up, with no pre-existing solutions in place. I believe that because they ran in the old enviornment, surely this one will work too. I moved the goats from the pre-existing environment to the new one. Two strings of electric fence, some water, surely I\u0026rsquo;ll be good, right?\nIt took about 2 whole hours for my newly deployed workload to escape the boundaries that I had set for it. They were out in my yard before I could go back in for dinner. And once they found that they could expliot the weakness in my infrastructure, it would keep happening\u0026hellip;. over and over and over and over.\nAs the summer goes on, they kill my wife\u0026rsquo;s flowers, they poop on our porch, they sleep in our driveway. I woke up in the morning one time to a goat looking through my window at me. The ultimate noisy neighbor, killing the plants, dirtying up the environment, and crashing when they\u0026rsquo;re not supposed to. I had a runaway workload, and no possible way to easily retrofit the environment to counteract the holes they were exploiting. Once an Application is in production, it is must more difficult to make sweeping changes in how those Applications are managed. Deploying sweeping changes Production comes with friction, the need for change control, the risk of impacting other things in the environment, the cost of thoroughly testing and validating those changes.\nIt became perosnal. I spent that entire summer trying to patch fence, add electric strands to decrease the exposed fence line, spending hundreds on electric chargers. Adding more fence posts, more strands, more anything just to get this workload under control. Eventually, you couldn\u0026rsquo;t tell if I was trying to stop goats from escaping, or the T-Rex from Jurassic Park. I had completely over engineered the environment, but to little success.\nThe reason is, I was trying to retrofit a now pre-existing environment for a workload that it was never properly designed to handle. Once you\u0026rsquo;ve opened the environment up, and then try to bolt on additional guardrails because you\u0026rsquo;ve under engineered it, you simply expose the ugly under belly of the solution. My solution was never designed to withstand the requirements that this workload came with. I never understood how stubborn, how creative, and how persistent they would be at testing the edges of my solution. I knew how to build A fence, but I didn\u0026rsquo;t think about the proper fence for what I was going to be placing into this environment. Then, trying to turn that solution into the proper solution without ripping it out just led to disjointed and illfitted add-ons that cost money but did nothing for efficiency.\nBad Neighbor\nNow picture this, you\u0026rsquo;re the wife of a farmer, drinking your afternoon coffee. It\u0026rsquo;s 93 degrees outside, and you look in the yard. You see a goat sprinting by, and then your farmer husband sprinting with a net above his head. He catches one, puts it back into the pen, and another escapes behind him.\nEventually, I just gave up, they did what they wanted, and I tried to mitigate the damage. I moved them back to the old environment, we ripped out the entire solution, and I built it properly from scratch.\nNo Kidding, the Bahhhhd Truth The old adage of, you can do it fast, cheap, or good\u0026hellip;. pick 2. I chose fast and cheap, and it ended up being an absolute poor fit that ended up costing me more in the long run. There are many lessons we can take away from this story. Platform Engineering isn\u0026rsquo;t just about building infrastructure. It\u0026rsquo;s about understanding the differences in what you need to provide from environment to environment. What Applications will be running in your Platform, what capabilities do they need, and how much room do they need to expand. My new pen looked great, until I placed a workload in it. Had I been putting a horse in there, it probably would have been sufficient. However, I wasn\u0026rsquo;t, and my failure to take time to understand the workload requirements meant I couldn\u0026rsquo;t account for their tendancies in my initial design.\nThere is a time investment that must occur up front, before any workloads ever get scheduled. Do you have the proper controls in place to enforce standardization, such as Policy Enforcement around workload declaration to enforce resource limits? Do you have proper observability, so that you don\u0026rsquo;t find out your workload is running amok due to your wife finding her tulips as just stumps on the ground? Is the automation in place to seamlessly move that workload from the old environment to the new, to add new workloads in a controlled and standardized way, and to simplify the processes that manage and maintain that environment? You can build a pretty looking environment without much substance, but you will truly test those boundaries when you start putting workloads into it, and any weakness in the architecture is exposed. You\u0026rsquo;ll never know where those workload may test your infrastructure if you don\u0026rsquo;t take the time to understand their tendancies.\nToo often, leads look to implement the new thing before taking time to prepare the infastracture and teams building that infrasructure. Whether that is failing to invest in the proper hardware, or failing to provide adequate time to train the engineers who will support the new environment, or even allowing for the process to build the guardrails to provide a stable and consistent environment with which to run. When that happens, the engineers spend all their time chasing errors, runaway processes, and issues in the environment. Add to that the additional money spent on trying to patch a losing solution, all that time wasted trying to find quick fixes to a situation that inevitably would never work at scale. As a leader and decision maker, we must make sure to present the downfall of a lack of investment and preparation for a big move, while making sure to evangelize the longterm benefit in proper investing in planning and execution. We must fight for that investment in time, effort, and hardware. On top of what we\u0026rsquo;re already dealing with, AI workload implementation will challenge our understanding of what is necessary in the infrastructure, and test new boundaries and configuration to expose those weaknesses. Whether that is new scheduling requirements, changes in howe we observe and react, or execution concurrency on hardware.\nThere are hidden opportunity costs to this lack of investment as well. There is A LOT I can accomplish on this farm. Improving the pig load out area, increasing the number of cattle we have, improving my water distribution system in the barn, or pasture management to improve grass quality. Instead of doing any of those, which would either increase automation to free up time, or improve quality and output from a financial perspective, I was chasing fixes for a solution that was not generating any net benefit at the time. This is the true cost, taking development and engineering time away from new features, and having them directed towards just trying to keep the existing runtimes operational. Lowering the overall speed to market due to rushing a solution to production that wasn\u0026rsquo;t ready.\nHow many of your engineers are sprinting across the yard with a net trying to catch that runaway workload simply because we avoided investing in the time and cost of properly configured environment? How many developers have given up on solving the problem and are simply just trying to limit the damage? Your infrastructure isn\u0026rsquo;t just an environment, but it\u0026rsquo;s the boundary that keeps the workloads running where they should, when they should, and how they should. The lesson here is, buy the right fence early, build it strong, save yourself the headache, and allow your teams to move on to the next big thing.\nResults of Proper Planning\n","permalink":"https://pigs-pods.pages.dev/posts/holding-goats/","summary":"\u003cp\u003eThis story dives into the hidden cost and lessons learned in rushing the move to production, and a failure to invest in the underlying architecture that protects your critical workloads. Through an assumption that the new environment is as good as the old one, I spent an entire summer patching issues, chasing runaway objects, and making little progress on anything else.\u003c/p\u003e\n\u003ch2 id=\"the-initial-deployment\"\u003eThe Initial Deployment\u003c/h2\u003e\n\u003cp\u003eThe thing about goats are, they are the ultimate example of a poorly written application. Left to their own devices, they will go all the places you don\u0026rsquo;t want them, consume all the plants you want protected, and once they find a weakness in your infrastructure, they will continously to exploit that until it is fixed. They are also the ultimate example of Garbage Cleanup. They will eat almost anything, and especially seek out the foilage that other animals won\u0026rsquo;t touch, like multiflora rose, poison ivy, and briar bushes. You can place them into your environment, and they will clear out all the unwanted junk that chokes out the useful plants for your other \u0026ldquo;workloads\u0026rdquo;.\u003c/p\u003e","title":"The Great Goat Breakout"},{"content":"Pigs and Pods What is this? I\u0026rsquo;ve always wanted to be one of those people that wrote interesting and impactful tech articles. The thought of having something to contribute to the Cloud-Native and Platform Engineering community was always an enticing goal.\nHowever, like many of you, I\u0026rsquo;ve always struggled with believing that I could contribute something novel to the conversation. There are so many talented and insightful viewpoints out there, how can I fit an original thought with all those experts.\nThe thought hit me one day as I cut my 4 acre yard. I have a farm, and I\u0026rsquo;ve often thought about the comparison between what I do on the farm, and what I do professionally at work. And although I don\u0026rsquo;t have the official count, the number of individuals creating OPA Gateway Policies and L7 Ingress Routes in the early afternoon, and shoveling pig poop in the evening is likely very low. And much like I have to clean up after my animals, professionally Platform Engineers are often times cleaning up after bad implementations, poorly written applications, and impacts of limited resources.\nThe farm and Platform Engineering comparison goes very deeply once you take the time to understand both.\nWorkloads\nThe Workload The application is whole reason to build a Platform. A space to run the application, boundaries to protect the application, and day to day management to keep the application running at optimal performance for optimal output.\nOn the farm, this is the animals, the livestock. We build fence to keep them within the area we\u0026rsquo;ve designated for them. We feed and provide them water, the resources they need to run. We clean up after them, removing the aftereffects of them \u0026ldquo;running\u0026rdquo; on the farm.\nThe Infrastructure\nThe Infrastructure The underlying infrastructure provides the runtime for the workload. The Memory, the CPU, Disk IO and throughput, Networking, and even the software infrastructure like Kubernetes. We build the infrastructure in a way that is optimal for the workload, not too large that we waste resources, but not too small that we can handle the workloads that we need run. The infrastructure is a investment, that you get back by running your workloads. The farm itself is all infrastructure. The fence that we build to keep the livestock in the area we designated for them. The feed and water that we provide them so they can continue to grow and operate, much like memory and cpu to an application. The barn to provide security to the animal from the weather, or the electric that keeps your water from freezing at night. These are all costs, that enable you, as a farmer, to maintain your livestock. The Automation\nThe Automation Without Automation, we would be buried in toil. Whether it\u0026rsquo;s Ansible, Terraform, Github Actions, Jenkins, ArgoCD and GitOps, all these tools and methodologies allow us to take manual tasks and complete them in a uniform and efficient pattern. You couldn\u0026rsquo;t scale your platform, you couldn\u0026rsquo;t guarantee consistency, and you couldn\u0026rsquo;t maintain the environment with the usage of these tools. On the farm, these are the equipment. Without a tractor, there would be no way for me to move the payload that I need move. Cutting hay and feeding it by hand would never let me get beyond a small number of chickens, purchasing the hay baler and hay mower allows me to provide for a much larger number of livestock. Though, I only have one piece of equipment that was manufactured before I was born, it serves the same purpose\u0026hellip;. just requires a whole lot more maintenance, no opensource contributors are remounting the wobble box on my New Holland 477 Hay Bine. The Engineer\nThe Engineer The engineer is the most critical part. They must be experts in security, networking, automation, policy, and cloud-native technologies. There is always something new to learn, always a new technique or technology. That is what a lot of Platform Engineers love, including myself. The daily challenge, the need to learn and grow, and the feeling of never being bored. The engineer is the farmer. I\u0026rsquo;ve had to learn to read the hay to know when it\u0026rsquo;s ready to bale, to understand the feed makeup for finishing market hogs, learning to weld to fix broken equipment, or understanding how to manage selling product to family, friends, and community members. That same desire to learn and be challenged exist in the farmer, and it\u0026rsquo;s what makes this such a great comparison. For all of those that dream of giving up the IT life to become a goat farmer, this is your chance to see what that is truly like, from one of your own. Read about the challenges that I\u0026rsquo;ve faced on the farm, the lessons learned, and how I connect them to the daily struggle we see in the life of a Platform Engineer. Along with two rambunctious kids, and one wife who, for some reason, tries to adopt anything with wings, this is Pigs and Pods.\nCredit This idea came about one day, riding my tractor after finishing the grass. More than anything, the credit should go to all those that have supported me in my life and career. From my parents, who own a much more impressive farm and have helped me start my own. My co-workers who have mentored me, supported me, and helped me grow as a professional. My friends who laugh when I share my experiences. Most of all though, my loving wife who has supported me all along the way, and I would never be able to do this without her.\nTags List of Tags\n","permalink":"https://pigs-pods.pages.dev/posts/pigs-and-pods/","summary":"\u003ch1 id=\"pigs-and-pods\"\u003ePigs and Pods\u003c/h1\u003e\n\u003ch2 id=\"what-is-this\"\u003eWhat is this?\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;ve always wanted to be one of those people that wrote interesting and impactful tech articles. The thought of having something to contribute to the Cloud-Native and Platform Engineering community was always an enticing goal.\u003c/p\u003e\n\u003cp\u003eHowever, like many of you, I\u0026rsquo;ve always struggled with believing that I could contribute something novel to the conversation. There are so many talented and insightful viewpoints out there, how can I fit an original thought with all those experts.\u003c/p\u003e","title":"Pigs and Pods"},{"content":"","permalink":"https://pigs-pods.pages.dev/about/","summary":"About Pigs and Pods","title":"About Pigs and Pods"},{"content":"The groundwork for this site was laid by the team behind FedEx Cookbook, so thank you to that team for both requesting the usage of HUGO and providing access to their code base so we could build a comparable site for the edge.\n","permalink":"https://pigs-pods.pages.dev/credits/","summary":"Credits","title":"Credits"},{"content":"© Isaac Omaits\n","permalink":"https://pigs-pods.pages.dev/_footer/","summary":"\u003cp\u003e© Isaac Omaits\u003c/p\u003e","title":""}]