DZone Spotlight

Recently, I made a comment about the idea of there being a “best” monitoring tool: In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case. The comment got some pushback, both privately and in a few forums, and so I wanted to dig into what I meant and why I feel that way. But before I do that, I want to set the record straight: I stand by what I said about there being no “best” monitoring tool. But I was flat-out lying about the other stuff. The hills I’m willing to die on are: The best programming language is PerlThe best car is the 1967 Ford Mustang 390 GT/AThe best pizza style is deep dish, and I’m partial to getting it from Tel Aviv Kosher Pizza in Chicago With that cleared up, let’s get back to monitoring and observability. Zoom Zoom What really got me thinking about the false concept of a “best” monitoring tool was a video my son shared with me, comparing a Lucid Air Sapphire, a Bugatti Chiron, and a Tesla Plaid to see which was the fastest production car of all time. Disclaimer: I am NOT a car person. My son Kaleb (who is 21, in his 2nd year of university to become a mechanical engineer, and on two different Baja SAE teams) very much is. After watching the video, Kaleb pointed out that if the race (which was on a quarter-mile track) had been a half-mile instead, the Bugatti would have won, hands down. This was because (reasons. I honestly couldn’t follow the things he was saying at this point. I leave it to the reader to imagine Star Trek-like technobabble) My point in all this is that “best” — the fastest car in this case — is highly subject to other variables. The type of track (this was on a track that had been pre-treated with VHT, which makes it super sticky and affects traction), the distance, even things like altitude and weather — these all can impact the ultimate outcome. But I’m talking about more than external factors. What is “best” can be affected by the ultimate use case. Cost is the easiest one that comes to mind. Yes, a Bugatti might be fastest. But possibly the “best” car is a Honda Civic because you value cost and reliability over speed. Or perhaps a Kia Sedona might be “best” because you need more seats. Or a Ford F150. Or a Ryder 16-wheeler. This all relates to monitoring and observability in ways that are both important and, sadly, novel to a lot of IT practitioners. We get so caught up in the “speeds and feeds” aspect of tools and solutions — how many flows per second, how many traces per collector, maximum ingest before backpressure occurs — that we often fail to stop and say “Do I need that? Will I ever need that?” I shared this with my friend Kevin Sparenberg (another car guy, who writes occasionally on ), and he added: Monitoring tools are like any other tool. You have your favorite hammer, but it’s not appropriate in every scenario. You don’t (or shouldn’t) use a sledge for setting a nail, nor would you use a claw hammer for forging. You might have a favorite, but you need to consider the type of work you need accomplished. (Besides being a car guy, he’s a home repair weekend warrior. This is just one of the many reasons why we’re friends.) “Best” Isn’t Always Best Part of the blame can be placed at the feet of vendors. I’ve worked at a few, and it’s rare to find one that has the tools to help customers quantify volume and cost before implementation. Most simply say “let’s just get this installed, and we’ll see where it lands, and we can tune from there,” blithely ignoring the way the level of effort (not to mention political maneuvering in the C-suite) to implement a new tool makes sunk-cost fallacy a near-certainty. But that’s only part of the blame. The other part rests at our feet — the monitoring engineers who need to better shoulder the responsibility of understanding and speaking for the needs of our organization. Because if we don’t, who will? Much of this comes back to things I’ve already ranted about: If we don’t have a plan for the monitoring and observability data being collected, any cost is going to seem to be too much. If you have a plan, you’ll know exactly how much the data is worth, and be able to evaluate the cost of a tool. Learn to speak the language of business, to frame things NOT in the technical terms that you find familiar and comfortable, but in terms that make it clear to the business why a new tool is needed and the value it will provide. Solve the problems your organization is actually having. Once again, it’s easy to get swept up by a vendor’s vision. But if your company isn’t having any of the problems that vendor vision describes, it’s all wasted time and money. Kevin added another nuance. Hyper-focusing on a single metric isn’t just sloppy; it can lead to real problems down the road: I would even go so far as to revisit that zero-to-60 metric. What about zero-to-60-to-zero? Sure any car can get to 60, but which car is able to do that AND return to status quo quickly. If you need a good “bad” example, look at the Gen 1 for Dodge Viper. As my dad said, “plenty of giddy-up, virtually no whoa.” How does that translate to monitoring and observability? Think about alerting. LOTS of tools are able to detect and trigger alerts based on extremely specific (and sensitive) triggers. But far fewer have the controls to detect and stop alert storms. My point in all this is to remember that “best” always has to be weighed against YOUR values: Your real-world, actual business needsYour cost-to-benefit ratioYour team’s skillsetsYour tolerance for toil and effort during the transitionYour willingness to support one more tool in perpetuity…and so on. Good Enough is Usually Good Enough There is a story that is equal parts old, hilarious, and fake. It involves a hapless young man (it’s ALWAYS a dude) who decides to mount a JATO rocket to his car to see just how fast he can go. And in the ensuing chaos, this young man (supposedly) earned himself a posthumous Darwin Award. (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) However, one of the aspects of the story that makes it funny (at least in my opinion) is the sheer ridiculousness of it all. Sure, lots of folks want a fast car. Many of those folks are willing to spend a little extra for a car that’s a little faster than the norm. Fewer (but not zero) people might also be willing to go to great lengths to acquire not only a fast car, but “the fastest” car. But strapping a rocket to the top of an old car — one that was clearly never intended to be used this way? That is some serious janky automotive slapstick. It tickles our funny bone by evoking those grainy sepia-tinted images of early 20th century “flying machines” that were nothing more than 2 umbrellas strapped to a piston. As the final image of the JATO story fades in our mind, we can almost see the end-card saying “You just couldn’t leave well-enough alone, could you?” Likewise, we need to foster a habit of self-restraint and technical reflection in our monitoring discipline. We have to recognize when our excitement about a tool’s ability to process 8 million log messages a second clouds our own ability to step back and say “why do I even HAVE 8 million log messages a second?” And if there’s a good reason for that volume, follow up with the question “Why do I need to send every single one of those messages across the internet to a vendor’s storage?” I’m not saying there’s nobody in the world who might need that. You might have a good reason. I just want to suggest you take a second to consider things before you end up creating your own JATO-powered observability disaster. The Solution Is Both-And, not Either-Or This is the thing most vendors don’t say, at least admit within earshot of investors and members of the board, because it flies in the face of the marketing hype and sales pitches they’ve worked so hard to craft. You need more than one monitoring and observability solution. You need to decide which systems (and more fundamentally, which data on those systems) use each tool. You’re going to have to split your budget (time, effort, skills, money) between those tools. This is an ugly but unavoidable truth. Over 27 years working with monitoring and observability tools, the number of times I’ve seen a company that had exactly one monitoring solution is: zero. Even the ones that insist they do, after a little digging, have at least a few pockets of the environment that use something else, whether that’s a class of system (mainframes, minis, Windows NT servers, 6509’s); an organization or team that just went their own way or was acquired but never fully integrated; or a location that — due to their distance from the main organization, either functionally or geographically — has to maintain their own set of tools. And most orgs don’t have “at least a few pockets”. They have a full suite of overlapping solutions. You are going to have — more likely you already DO have — multiple monitoring solutions in place. This is one of those tech realities which is simple, but not easy — like supporting more than one operating system (whether servers or desktops); or moving from branch-based development to feature flags; or building a multi-language, multi-cloud app. It’s a cost of doing real business in the real world. Observability, like life itself, is messy. It also, to quote Ian Malcom, finds a way. So will you. Here’s how: Plan and prepare to identify data types based on complex criteria: it might be a combination of location, system type, data type, and even time frame. Expect that, based on those parameters, you’ll then filter and transform the data before sending it in the correct direction. Also expect that some data types are so valuable, you’ll end up sending the same data in more than one direction. But also prepare to put boundaries in place so you aren’t doing that all the time, because that gets expensive fast. As I said, you need to be ready to support multiple tools, but you should have a plan in place for how you’ll keep track of those tools, identify their primary use case, and even set boundaries on the things they will NOT be permitted to monitor, so your stable of solutions doesn’t explosively get out of control. One way to do that is to set, for every data type or use case, a definitive choice for a primary tool that handles that data, and a secondary that you use as a gut-check. For larger organizations or more important data sets, you might have a tertiary, but draw the line there. Another way (complementary to the first) is to understand that some tools are cheap and do a lot of things mostly ok, so you can spread them like peanut butter across the enterprise; while others are expensive (in time, effort, or money) and only do certain things well. So you should spread THOSE like caviar — only on the systems where they’ll do the most good. Finally, differentiate between management tools that also have a monitoring component and true monitoring and observability solutions. You shouldn’t get rid of management tools, but you also shouldn’t make them the primary source of truth for enterprise monitoring information because they are usually so vendor- or system-specific, and it will lead (again) to that explosion of tools I cautioned against earlier. A Brief Buyer’s Guide A natural question to ask next is “how do I know WHICH tools to get?” Once again, my buddy Kevin has some wise observations: Is it even worth mentioning bake-offs? Do people even do that anymore? Maybe tool A has these features, but tool B has these other ones. And they both share some other capabilities. We need all of it, but can’t get it in one package. There’s nothing wrong with that. Leon’s point about having multiple tools is on target. Bias your decisions to picking the right tool for the right job (but at the same time, try not to collect too many tools. This ain’t Pokémon). Don’t buy for “this tool has this neat feature we don’t need, but maybe someday we’ll want.”. Buy (or deploy) for what you need now and in the near future. IT (and the business) is always growing and evolving. What you THINK is important today may be irrelevant in six months. Hopefully not, but thinking too far ahead — the infamous “five year plan” is just a waste of your effort and time. Taking a Victory Lap The point of this blog is pretty simple: There’s no such thing as “best”, and that goes for everything from cars to programming languages all the way to observability solutions. But more essential is my point about WHY there isn’t a single, specific “best” — it’s because context matters. Use case matters. To be more nuanced, there probably is a “best,” but what is best is extremely particular to you and your circumstances. So the lesson in all this is to make sure you are clear about those circumstances, and that you’re always weighing them against the “we’re the best” marketing hype you’ll hear from many vendors.

Last year, I was working on deploying an agentic AI system to help manage cloud infrastructure at scale. The idea was straightforward: give the agent access to AWS APIs, let it observe infrastructure state, and allow it to take remediation actions autonomously. Scale a deployment here, restart a service there, update a configuration when metrics cross a threshold. What I did not fully appreciate at the time was how differently an AI agent fails compared to a traditional automation script. When a shell script goes wrong, it fails in a bounded, diagnosable way. You get an error code. You trace it. You fix it. When an agentic AI system fails, it can fail in ways you never anticipated, hallucinating resource states, misinterpreting instruction scope, or acting on adversarial inputs buried in a monitoring alert. These failures do not produce clean stack traces. They produce production damage. That realization sent me down a path of building a guardrail system. What I eventually learned, and this took four calibration cycles to prove empirically, is that no single guardrail layer can solve this problem. You need multiple complementary layers, and you need to design them to compensate for each other's blind spots. Here is what I built, what broke along the way, and what I would do differently from the start. Why the Obvious Solutions Did Not Work My first instinct was to use AWS Bedrock Guardrails. Configure a topic denial policy for destructive operations, set the content filters to HIGH, block PII like access keys. Simple, managed, done. I ran it against 100 representative agent prompts, a mix of read operations, staging changes, risky production changes, destructive operations, and adversarial jailbreak variants. The result stopped me cold. Tuned for zero false negatives, meaning I wanted to catch every genuinely dangerous action, the guardrail produced a 40% false positive rate. It was blocking list operations. It was blocking staging scale-outs. It was blocking service configuration updates that had nothing to do with deletion or destruction. That is not a deployable guardrail. That is a system that would make the AI agent useless within a day. The second problem was structural, not tuning-related. A Bedrock guardrail intercepts the model's text output. But an agent does not only produce text; it invokes tool calls. An agent can generate a perfectly compliant response like "I will scale the deployment safely" and then immediately invoke a delete API as a tool call. The guardrail never sees the tool call. It evaluated the wrong boundary. The third issue came when I looked at policy-as-code frameworks. OPA with Gatekeeper is excellent at Kubernetes admission time, evaluating manifests before they are deployed. But a DevOps agent is not deploying manifests. It is generating action proposals at runtime against live infrastructure that changes by the hour. A static policy that denies writes to "production resources" is useless unless it knows, at this exact moment, which resources are tagged as production. That information is not in a manifest. It is in live EC2 tags pulled from the AWS API. These were not flaws in the tools. There were boundary mismatches. Each tool was designed for a different problem. None of them was designed for the problem of governing an autonomous agent at the tool-call execution boundary. The Architecture I Landed On After a lot of iteration, I settled on a five-layer pipeline that intercepts at the tool-call boundary the moment the agent transitions from thinking to acting. Any layer can terminate the pipeline. The default is blocked. Blast-Radius Scoring Before any layer fires, every proposed action gets a blast-radius score between 0 and 1. Read-only operations (list, get, describe, monitor) score 0.1. Reversible mutations (restart, scale, update, patch) score 0.4. Potentially irreversible operations (delete, destroy, terminate, purge, wipe) score 0.9. This score drives routing. Low-risk actions auto-approve without touching the full pipeline. High-risk actions require human approval. Everything in between goes through policy evaluation. This is what keeps the system from adding 8 seconds of latency to every "list all EC2 instances" call. Layer 1: Bedrock Guardrail With a Bypass I kept Bedrock Guardrails as the first layer but added something critical: a selective bypass for low-risk and staging operations. Before the guardrail fires, the pipeline checks the blast-radius score and environment context. If the action is read-only or explicitly targeting a staging environment, the guardrail is skipped entirely. This one change took the false positive rate from 40% down to 18%. That occurred not by tuning the guardrail, but by changing the architecture around it. Layer 2: OPA Against Live State The second layer runs Open Policy Agent, but not against a static manifest. It pulls live AWS context via boto3 immediately before each evaluation: EC2 inventory with environment tags, S3 buckets, IAM roles. That live context becomes part of the input document that OPA evaluates. Now the policy can answer the question that actually matters: "Is this specific resource, right now, a production resource?" A rule that reads is_production(resource)` checks the live tag, not a manifest field. This is what catches the actions that Layer 1 misses, like "purge all messages from the SQS queue," that use vocabulary outside the guardrail's topic examples but clearly target production infrastructure. In my evaluation, Layer 2 was the sole blocking layer for 43% of correctly blocked prompts. Without it, those would have been false negatives. Layer 3: Confidence Delta Scorer This layer does something a little unconventional. It invokes the model twice, once with the guardrail attached and once without, and measures how much the guardrail changed the model's response. It looks at hedging vocabulary frequency: words like "should," "might," "could," "consider," "careful." A big divergence between the guardrail and unguarded responses means the guardrail substantially changed the model's behavior. That is a signal; the action is near the policy boundary and worth extra scrutiny. It is a probabilistic safety margin for borderline cases. Layer 4: HITL Gate With DynamoDB TTL For high-risk actions, a Lambda function writes a DynamoDB record with a 5-minute TTL and sends an SNS email with approve and reject URLs. The design principle that matters here: **silence equals rejection**. If the TTL expires without a response, DynamoDB automatically deletes the record, and the pipeline denies the action. An unavailable approver never produces an implicit approval. I built this using DynamoDB's native TTL feature, which means there is no polling loop, no Lambda checking every 30 seconds whether the token expired. It just works. Layer 5: Audit and Rollback Every pipeline execution writes a structured JSON record to CloudWatch: the prompt, risk classification, blast-radius score, per-layer decisions, latency, cost, and outcome. A CloudWatch alarm fires if the block rate drops below 70% in any 5-minute window, which would indicate something has degraded in the guardrail configuration. Before every approved action, boto3 captures a snapshot of the relevant resource state to S3. If health metrics degrade within 5 minutes after execution, automated rollback restores the pre-action state from that snapshot. What the Numbers Actually Showed I ran the complete pipeline against 100 prompts on live AWS infrastructure in us-east-1. Here is what came back: CategoryAccuracyFP RateFN RateAvg LatencyRead operations (20)95%5%0%910msSafe staging changes (20)100%0%0%7,920msRisky production changes (20)95%0%5%13,949msDestructive operations (20)100%0%0%9,590msAdversarial jailbreaks (20)90%0%10%8,125msTotal (100)96%1%3%8,099ms Three things stood out. Zero false negatives on destructive operations: Not one delete, destroy, terminate, or purge operation made it through. That was the most important number for me.22x latency reduction for blocked actions: Intercepted actions averaged 347ms. Allowed actions averaged 8,099ms. Because Layer 1 fires before the full LLM invocation, blocking a harmful action is actually faster than allowing a safe one. The safety mechanism is also a performance optimization for the adversarial case.Total cost: $0.0017 USD for 100 prompts: At 10,000 agent actions per day, that is roughly $0.17 daily. The cost of not having this is measured in incidents. The Calibration Study Nobody Talks About The finding I keep coming back to is the calibration progression: VersionAccuracyFP RateWhat Changedv1: Single Bedrock guardrail60%40%Baselinev2: Added low-risk bypass79%18%Architectural changev3: Added staging context in OPA89%8%Live state integrationv4: Expanded service config keywords96%1%Allow-list expansion What strikes me is that each improvement required a fundamentally different mechanism. The bypass addressed a structural mismatch. The staging context detection required live infrastructure data that no static guardrail can access. The keyword expansion fixed a vocabulary coverage gap. None of these is achievable by turning a dial on a single layer. This is the empirical case for layered defense-in-depth. Not as a philosophical preference. As a measurable engineering necessity. Practical Takeaways If you are building agentic DevOps tooling, here is what I would tell myself from a year ago: Intercept at the execution boundary: Your safety mechanism must fire when the agent calls a tool, not when it generates text.Pull live state before every policy evaluation: A policy that cannot see which resources are actually in production right now is not protecting production.Make your HITL gate fail closed: Design it so an unresponsive approver produces a denial, not a permit. DynamoDB TTL handles this elegantly without polling.Run your calibration study before going live: Measure FP and FN rates separately. They trade off against each other in ways that are not obvious until you measure them.Snapshot before every approved action: Automated rollback is not glamorous, but it is the safety net you will want when something approved turns out to be harmful. The Code Everything described here is open source: https://github.com/ManvithaP-hub/agentic-devops-guardrails That includes the Lambda functions, OPA Rego policies, boto3 state fetching, DynamoDB approval gate, CloudWatch audit, and a Terraform deployment module. You can run the full evaluation on your own AWS account for under a dollar.

For years, the MediatR package has been a default inclusion in almost every new .NET project template. It is frequently hailed as the gold standard for decoupling controllers or minimal APIs from business logic and implementing clean architecture or CQRS patterns. However, as ASP.NET Core has matured, many of the architectural justifications for adding MediatR as a heavy third-party dependency have faded. If you are building standard CRUD or even moderately complex enterprise APIs, it might be time to ask yourself, Why am I routing my HTTP requests through an abstract in-memory bus when native alternatives exist? Let’s explore why MediatR might be an unnecessary abstraction in your codebase and how you can replace its most beloved feature, pipeline behaviors, using pure .NET Dependency Injection (DI) plumbing and the decorator pattern. The Core Critique of MediatR While MediatR provides an elegant mechanism for decoupling, it introduces specific friction points into a codebase, Obfuscated Control Flow: Because handlers are resolved dynamically, it is impossible to use standard “Go to Definition” (F12) in your IDE to trace execution directly from an endpoint to its handler. You are forced to search for the corresponding IRequestHandler type implementation.Debugging Overhead: Stepping through code becomes a tedious exercise of bypassing internal library code generator stacks instead of moving sequentially through your business logic.Unnecessary Architecture: In standard Web APIs, the mapping between an endpoint and its handler is almost always 1:1. Introducing a mediator pattern for a direct request-response cycle over complicates the system with minimal structural payback. If you are using minimal APIs or standard controllers, you already have powerful endpoint routing. So why use MediatR? The answer almost always boils down to one critical feature, namely, pipeline behaviors. The Killer Feature: Cross-Cutting Concerns Developers love MediatR because it makes addressing cross-cutting concerns, such as centralized logging, validation pipelines (e.g., using FluentValidation), metrics collection, and OpenTelemetry tracking, incredibly clean. Instead of cluttering every single business handler with repetitive try-catch blocks or explicit validation calls, MediatR lets you define a generic middleware pipeline around your requests: C# // The classic MediatR approach builder.Services.AddMediatR(cfg => { cfg.RegisterServicesFromAssembly(typeof(Program).Assembly); cfg.AddOpenBehavior(typeof(LoggingBehavior<,>)); cfg.AddOpenBehavior(typeof(ValidationBehavior<,>)); }); It is a fantastic architectural model, but you do not need a third-party package to achieve it. Modern .NET allows you to implement this exact pattern natively using compile-time type-safe decorators. The Native Alternative: DI-Based Decoration Instead of an abstract pipeline behavior model, we can leverage the decorator pattern natively supported by the Microsoft.Extensions.DependencyInjection container. This allows us to transparently wrap any standard interface registration with cross-cutting behaviors. 1. Defining a Domain-Driven Request Handler Interface First, let’s establish our own lightweight, explicit handler contract. This preserves your CQRS separation without binding your domain to external packages. C# public interface IRequestHandler<in TRequest, TResponse> { Task<TResponse> HandleAsync(TRequest request, CancellationToken cancellationToken = default); } 2. Creating a Concrete Business Handler Your concrete query or command handlers remain pristine. They are entirely unaware of logging, telemetry, or validation logic, adhering perfectly to the Single Responsibility Principle. C# public record CreateProductCommand(string Name, decimal Price) : IRequest; public class CreateProductHandler : IRequestHandler<CreateProductCommand, Guid> { private readonly IProductRepository _repository; public CreateProductHandler(IProductRepository repository) { _repository = repository; } public async Task<Guid> HandleAsync ( CreateProductCommand request, CancellationToken cancellationToken = default) { var id = Guid.NewGuid(); // Core business logic goes here... return id; } } 3. Implementing the Cross-Cutting Decorator Now, let’s implement a generic decorator that intercepts the request execution. It implements the same interface but accepts the inner concrete handler as a dependency, wrapping it with infrastructure logic. C# public class LoggingHandlerDecorator<TRequest, TResponse> : IRequestHandler<TRequest, TResponse> { private readonly IRequestHandler<TRequest, TResponse> _inner; private readonly ILogger<LoggingHandlerDecorator<TRequest, TResponse>> _logger; public LoggingHandlerDecorator( IRequestHandler<TRequest, TResponse> inner, ILogger<LoggingHandlerDecorator<TRequest, TResponse>> logger) { _inner = inner; _logger = logger; } public async Task<TResponse> HandleAsync (TRequest request, CancellationToken cancellationToken = default) { var requestName = typeof(TRequest).Name; _logger.LogInformation("Executing request: {RequestName}", requestName); try { var response = await _inner.HandleAsync(request, cancellationToken); _logger.LogInformation("Successfully executed request: {RequestName}", requestName); return response; } catch (Exception ex) { _logger.LogError(ex, "Request failed: {RequestName}", requestName); throw; } } } Wiring It Together Natively in Program.cs To register these decorators without external help, we can write a clean extension method using the native DI container’s service factory capabilities. This gives us full architectural control over which handlers get decorated and in what order. C# public static class RequestHandlerRegistrationExtensions { public static IServiceCollection AddDecoratedRequestHandler<TRequest, TResponse, THandler>( this IServiceCollection services) where THandler : class, IRequestHandler<TRequest, TResponse> { // 1. Register the concrete handler with its own concrete type services.AddTransient<THandler>(); // 2. Register the interface using a factory method that constructs the decorator chain services.AddTransient<IRequestHandler<TRequest, TResponse>>(sp => { var concreteHandler = sp.GetRequiredService<THandler>(); var logger = sp.GetRequiredService<ILogger<LoggingHandlerDecorator<TRequest, TResponse>>>(); // Wrap the core handler with our logging decorator return new LoggingHandlerDecorator<TRequest, TResponse>(concreteHandler, logger); }); return services; } } Pro-Tip: If manual registration feels too verbose for thousands of handlers, you can use assembly scanning utilities like Scrutor to dynamically apply the .Decorate() method across your entire service collection automatically. Your application endpoints remain perfectly clean, directly consuming the generic interface via native dependency injection: C# app.MapPost("/products", async ( CreateProductCommand command, IRequestHandler<CreateProductCommand, Guid> handler) => { var result = await handler.HandleAsync(command); return Results.Ok(result); }); Architectural Advantages By shifting from an in-memory mediator library to native DI decoration, you unlock major engineering benefits: No Black Boxes: Your execution pathway is clear. If you place a breakpoint inside your API endpoint and step into handler.HandleAsync(), you will sequentially enter the LoggingDecorator, step through any validation layers, and land cleanly inside your actual business logic handler.Zero Third-Party Vendor Lock-in: Your core business logic blocks do not depend on external packages. Upgrading your framework version will never be delayed due to a breaking change in an open-source messaging mediator.Granular Structural Control: Unlike globally forced middleware pipelines, you can explicitly choose which command or query handlers receive specific decorators. If a high-throughput endpoint requires raw performance without validation overhead, you can register it without its decorator wrap. Summary MediatR has served the .NET community exceptionally well over the last decade. However, frameworks evolve. With modern ASP.NET Core DI plumbing, we can maintain clean separation of concerns, enforce CQRS principles, and leverage pipeline behaviors seamlessly using standard patterns built straight into the runtime. Evaluate your current architecture. Is MediatR solving a foundational problem for you, or is it merely acting as boilerplate code that your native framework can already handle?

The Failure You Have Probably Already Seen An enterprise AI agent is deployed against production data. It answers the first ten questions confidently and correctly. Then, on the eleventh question, it produces an answer that looks reasonable but is completely wrong. The team investigates. The model is fine. The prompt is fine. The tool integrations are fine. The problem is buried in the data itself. A field the agent relied on has drifted. A join it assumed existed no longer holds. A quality signal that used to be reliable has silently degraded. This is not a rare edge case. It is becoming one of the most common failure patterns in enterprise AI systems moving from prototype to production. And it points to a simple, uncomfortable truth: most enterprise data infrastructure was built for a consumer we no longer have. I have spent the past couple of years designing agentic AI systems against production data at Fortune 500 scale. What follows is the runtime governance pattern I now design around, and the failure modes it protects against. Who this article is for: This article is for data engineers, platform architects, AI engineers, and governance teams building enterprise agents that depend on production data. It focuses less on prompt design and more on the runtime data controls required to make agent answers reliable. Twenty Years of Data Built for Humans Every large enterprise data platform in production today was designed for human consumption. Analysts, business users, data scientists, and BI teams. Those consumers share a common trait: they exercise judgment. A human analyst looking at a broken dashboard notices it. A data scientist opening a table with unusual distributions asks a colleague. A finance user reviewing a report questions the number when it does not match their gut. Enterprise data governance evolved to support this consumer. Documentation lives in wikis. Quality is enforced by expected-value alerts that a human triages. Lineage is captured at the ETL job level, not the field level. Access is granted through role-based permissions and refined by manual data stewardship. All of this works when a human is at the end of the pipeline. An AI agent is not that consumer. An agent has no judgment. It processes what it is given and returns an answer. If the data is stale, the agent produces a stale answer with high confidence. If the lineage is broken, the agent cannot trace why. If a quality signal exists only as a wiki page, the agent cannot use it. The Four Gaps Most Enterprises Have Across the AI-in-production work I have seen, the same four gaps show up almost every time. Gap 1: Machine-Readable Data Contracts Most contracts exist as documentation, not as programmatic constraints. An agent cannot ask a Confluence page whether it is safe to trust a field. Data contracts need to be enforced at the platform layer, with schema, type, freshness, and quality guarantees expressed as executable rules. Gap 2: Use-Case-Aware Quality Fitness A dataset that is 95 percent complete may be fine for a marketing dashboard and completely wrong for a clinical AI model. Traditional data quality checks are use-case-agnostic. Agentic AI requires quality signals that answer a different question: is this data fit for this specific decision, right now? Gap 3: Field-Level Lineage That Updates in Real Time When a pipeline changes, human consumers get an email. Agents get a wrong answer. Lineage systems need to update as pipelines evolve and expose change signals in a form agents can consume, not just visualize. Gap 4: A Discovery Layer Agents Can Query Most catalog systems are designed for humans to browse. Agents need a machine interface to ask questions like which tables contain the concept I care about, and which of them is authoritative for this domain. Design Principles for Agentic Data Governance Closing these gaps does not require rebuilding the entire data platform. It requires making governance executable in the same path where the agent retrieves data, evaluates context, and produces an answer. Three design principles matter most. Start with the decision, not the data. For each production AI use case, define what a wrong answer looks like and work backward to the data requirements that would prevent it. This surfaces the specific quality signals, lineage nodes, and freshness constraints that matter. Make governance runnable, not readable. Every governance artifact your agents depend on should be programmatically executable at inference time. If a rule cannot be checked in code, an agent cannot use it. Documentation is useful for humans, but for agents it is invisible. Instrument for continuous evaluation. A governance framework that only fires at deployment is not enough. Models drift, data drifts, and use cases evolve. The governance layer needs to continuously evaluate agent outputs against real-world outcomes and flag drift before it becomes damage. Reference Architecture: Runtime Data Governance for AI Agents A practical implementation usually introduces a lightweight runtime governance layer between the agent and the underlying data platform. The goal is not to slow the agent down. The goal is to give the agent a reliable way to ask whether the data behind an answer is safe to use. At a minimum, this pattern includes five components: a data catalog that exposes authoritative sources, a contract registry that stores schema and business rules as executable checks, a lineage service that tracks upstream dependencies at the field and metric level, a quality service that publishes freshness and fitness signals, and an agent guardrail service that evaluates these signals before the agent responds. Runtime flow: User question → Agent → Semantic/data resolver → Governance service → Catalog, contract registry, lineage service, and quality service → Pass/Warn/Block decision → Agent response. Layer Responsibility Example Signal Catalog Identify authoritative datasets and business definitions. Certified source for booked deal value. Contract registry Validate schema, data types, null thresholds, and business rules. Discount variance must use the approved baseline method. Lineage service Track upstream source, transformation, and metric dependencies. Metric changed because a new source was added. Quality service Publish freshness, completeness, anomaly, and fitness scores. Dataset refreshed within SLA and passed threshold checks. Agent guardrail Block, warn, or allow the answer based on governance signals. Answer allowed only if lineage and contract checks pass. The agent should not directly trust a dataset simply because it can access it. Before answering, it should evaluate the data path, the contract status, the freshness window, the lineage change history, and the use-case-specific fitness score. If any critical check fails, the agent should either decline to answer or return the answer with an explicit data reliability warning. How the Runtime Governance Check Works In practice, the check is a short pre-answer step. The agent does not need to understand every governance rule directly. It needs a stable contract with a governance service that can evaluate the data path and return a decision. The user asks a business question.The agent resolves the requested metric, entity, dataset, or semantic concept.The agent calls the governance service with the resolved data assets and intended use case.The governance service checks catalog certification, contract status, lineage changes, freshness, completeness, and use-case fitness.The service returns a pass, warn, or block decision with machine-readable reasons.The agent answers, adds a caveat, escalates, or declines based on that decision. What a Machine-Readable Data Contract Actually Looks Like The abstract idea of a data contract only becomes real when you can point to one that an agent can actually consume. Here is a compact YAML example for a deal variance metric, expressing schema constraints, business rules, freshness expectations, and quality thresholds in a single artifact: YAML contract: dataset: deal.discount_variance schema: - field: discount_variance_pct type: decimal(18,2) required: true calculation: approved_discount_baseline_v2 - field: source_system type: string allowed_values: [crm_v3, revenue_hub] freshness: sla_hours: 24 breach_action: warn quality: completeness_threshold: 0.95 anomaly_score_max: 3.0 lineage: change_window_days: 30 on_upstream_change: require_review With this in place, an agent can call a single governance endpoint before responding, receive a machine-readable pass, warn, or block decision, and either answer confidently, answer with a caveat, or decline. The rule is not buried in a wiki page. It is live at inference time. Example Runtime API Pattern The runtime call does not need to be complicated. A minimal request can identify the metric, dataset, use case, and decision context. The response should be small enough for the agent to use directly in its control flow. JSON POST /governance/evaluate Request: { "metric": "deals.discount_variance_pct", "dataset": "deals.discount_variance", "use_case": "deal_desk_agent_review", "decision_context": "discount_variance_explanation" } Response: { "decision": "warn", "reasons": ["upstream_lineage_changed", "freshness_within_sla"], "agent_action": "answer_with_caveat" } In the agent workflow, this response becomes a control decision. A pass allows the agent to answer normally. A warn allows the answer but requires a reliability caveat. A block prevents the answer and routes the request to review, remediation, or a safer fallback path. Pseudocode: Turning Governance Into Agent Control Flow Python decision = governance.evaluate(metric, dataset, use_case) if decision.status == "block": return decline_with_reason(decision.reasons) if decision.status == "warn": return answer_with_caveat(query, decision.reasons) return answer(query) This is the core shift: governance is no longer a document the team reads during design review. It becomes a runtime dependency that the agent uses to decide whether to answer, qualify the answer, or stop. Runtime Checks an AI Agent Should Perform Before Answering Is this dataset or metric certified for the requested business domain?Has the schema changed since the agent workflow was last validated?Did all required fields meet completeness and validity thresholds?Is the data fresh enough for the decision being requested?Has any upstream lineage changed within a defined risk window?Does the requested answer depend on a metric with multiple calculation methods?Should the agent answer, warn, escalate, or decline based on the governance outcome? This does not require a heavyweight approval workflow for every query. In many cases, the runtime check can be a fast metadata call that returns a simple decision: pass, warn, or block. The important design principle is that governance must be available in the same execution path as the agent response, not in a separate documentation process that only humans can interpret. Failure Modes and Runtime Controls Failure mode What causes it Runtime control Stale answer Dataset missed its refresh SLA. Freshness check with warn or block behavior. Wrong metric Multiple calculation methods exist for the same business concept. Contract and semantic registry validation. Silent lineage change An upstream source or transformation changed after validation. Field-level lineage check within a defined risk window. Misused dataset The dataset is accessible but not certified for the requested domain. Catalog certification and use-case fitness check. Incomplete evidence Required fields fail completeness or validity thresholds. Quality service decision with explicit failure reasons. A Concrete Example From the Field On one enterprise AI project in a regulated environment, we deployed an agentic assistant to help analysts explore a large deal registration and booking dataset. Early testing looked solid. Several weeks into production, the agent began returning confidently wrong answers about a specific discount variance metric. The model had not changed. The prompt had not changed. What changed was an upstream ingestion job that added a new source that computed discount against a different price baseline. A human analyst would likely have questioned the number because it felt off. The agent did not. It saw a valid number in a valid field and reported it as authoritative. The fix was not in the model. We added a machine-readable contract for the approved discount baseline, a lineage signal for recent upstream changes, and a runtime check the agent could call before answering. After that, the same failure could not recur silently. The agent either answered correctly or flagged that the underlying data had changed and required review. The lesson was not that agents are unreliable. It was that agent reliability is a property of the data layer, not the model layer. Once we treated the governance layer as an active runtime dependency instead of static documentation, the entire class of silent-failure risk collapsed. Implementation Considerations Cache low-risk governance decisions to reduce latency, but recheck high-risk metrics at runtime.Separate warn rules from block rules so agents can still answer safely when risk is explainable.Version data contracts alongside pipelines, semantic models, and metric definitions.Log every agent answer with the governance decision, reasons, dataset version, and lineage snapshot used.Start with high-risk metrics and regulated workflows before expanding the pattern across the broader data estate. Why This Belongs in the Architecture, Not the Prompt Prompt engineering can reduce some surface-level errors, but it cannot solve a missing contract, stale dataset, broken lineage path, or ambiguous metric definition. Those failures sit below the model. They need to be handled in the platform architecture, where data access, metadata, quality, lineage, and policy decisions are available at runtime. For teams building enterprise AI agents, the practical takeaway is straightforward: treat runtime governance as part of the agent stack. If an agent can call a retrieval service, vector index, SQL endpoint, or workflow tool, it should also be able to call a governance service before committing to an answer. The next generation of enterprise AI reliability will not come only from better models. It will come from data platforms that can tell agents, in real time, whether an answer is safe to give. About the Author. Avinash Maddineni is a Lead Data Engineer with 15 years of enterprise data infrastructure experience across healthcare, financial services, energy, and travel. He builds agentic AI and data governance systems at Fortune 500 scale and is founder of PureStrokeAI (USPTO provisional patent filed May 2026).

Arrays in Java are fundamental data structures used to store elements of the same type sequentially in memory. They provide a convenient way to manage collections of data where each element is accessed by its index. Basics of Arrays An array in Java is a fixed-size container that holds a specific number of elements of the same data type. This means all elements in an array must be of the same type such as integers (int), floating-point numbers (double), characters (char) or objects (Object). Declaring and Initializing Arrays To declare an array in Java we specify the type of elements followed by square brackets [] and the array name: Java dataType[] arrayName; For example, to declare an integer array named numbers: Java int[] numbers; Arrays in Java are objects and like all objects they must be instantiated with the new keyword before they can be used: Java arrayName = new dataType[arraySize]; For instance, to create an integer array numbers with a size of 5: Java int[] numbers = new int[5]; This initializes an array numbers that can hold 5 integers with indices ranging from 0 to 4. Accessing Elements in Arrays Array elements are accessed using their index, which starts at 0 for the first element and goes up to arraySize - 1 for the last element. For example, to access and modify elements of the numbers array: Java int[] numbers = {10, 20, 30, 40, 50}; int firstElement = numbers[0]; // Retrieves the first element (10) int thirdElement = numbers[2]; // Retrieves the third element (30) numbers[1] = 25; // Modifies the second element to 25 Array Length The length of an array in Java, which is the number of elements it can hold can be obtained using the length property: Java int arrayLength = numbers.length; // Returns 5 for the 'numbers' array The length property is a final variable defined in the array object itself and it cannot be changed after the array is created. Iterating Through Arrays Arrays can be traversed using loops such as for or foreach to access and manipulate each element sequentially: Java int[] numbers = {10, 20, 30, 40, 50}; for (int i = 0; i < numbers.length; i++) { System.out.println("Element at index " + i + ": " + numbers[i]); } Alternatively, Java provides an enhanced for-each loop also known as the enhanced for loop to iterate through elements of an array without explicitly using an index: Java for (int number : numbers) { System.out.println(number); } Multidimensional Arrays Java supports multidimensional arrays which are arrays of arrays. we can declare and initialize them as follows: Java dataType[][] arrayName = new dataType[rows][columns]; For example, to create a 2D integer array matrix with 3 rows and 3 columns: Java int[][] matrix = new int[3][3]; Accessing elements in a 2D array requires specifying both row and column indices: Java int element = matrix[1][2]; // Retrieves element at row 1, column 2 Arrays Class Methods The Arrays class in Java provides utility methods for working with arrays such as sorting, searching and comparing arrays: Java import java.util.Arrays; int[] numbers = {5, 3, 8, 2, 7}; Arrays.sort(numbers); // Sorts the 'numbers' array in ascending order int index = Arrays.binarySearch(numbers, 8); // Searches for '8' in the sorted array Other useful methods include copyOf(), fill() and equals(). Common Operations on Arrays Sorting: Arrays can be sorted using Arrays.sort().Searching: Use Arrays.binarySearch() to search for an element in a sorted array.Copying: Arrays can be copied using Arrays.copyOf() or System.arraycopy().Filling: Arrays can be filled with a specific value using Arrays.fill(). Applications of Arrays Arrays are used extensively in various applications, such as: Storing and manipulating collections of data in algorithms and applications.Implementing data structures like lists, queues, and matrices.Handling input/output operations in Java programs.Passing arrays as parameters to methods for processing and manipulation. Conclusion Arrays are fundamental data structures in Java that provide efficient storage and access mechanisms for homogeneous collections of data. They play a crucial role in Java programming offering versatility and performance in managing and manipulating data elements. FAQs 1. What is an array in Java? An array in Java is a fixed-size collection of elements of the same type stored sequentially in memory.2. How do you declare an array in Java? You declare an array in Java by specifying the type of elements followed by square brackets [] and the array name, like int[] numbers;.3. Can arrays in Java store elements of different data types? No, arrays in Java can only store elements of the same data type. Once declared the data type of an array is fixed.4. What is the difference between length and length() in arrays? length is a final variable in arrays that denotes the number of elements it can hold. length() is a method used with the strings and other objects to get the number of characters or elements.5. How do you initialize an array in Java? You can initialize an array in Java using the new keyword followed by the array type and size like int[] numbers = new int[5];.6. What are multidimensional arrays in Java? Multidimensional arrays in Java are arrays of arrays. They allow you to store data in multiple dimensions such as rows and columns in a matrix.7. How can you iterate through an array in Java? we can iterate through an array in Java using a for loop or an enhanced for-each loop to access each element sequentially.8. Can you resize an array in Java once it's created? No, once an array is created with a specific size its size cannot be changed. we would need to create a new array with the desired size and copy elements if resizing is needed.9. What are the common operations you can perform on arrays in Java? Common operations include sorting arrays (Arrays.sort()) searching for elements (Arrays.binarySearch()), copying arrays (System.arraycopy()) and filling arrays (Arrays.fill()).10. What are the applications of arrays in Java? Arrays are used for implementing data structures like lists and queues storing data in algorithms handling input/output operations and passing data to methods efficiently.

Need to perform asynchronous operations and support multitasking in your app? Async/await is at your service — simple and elegant. The cooperative thread pool efficiently switches threads between tasks, while the compiler ensures thread safety at the type level. You can even seamlessly bridge older parts of your codebase written in GCD! But then, for some reason, your app starts hanging in production… Below, we will explore specific examples (complete with diagrams) of how not to mix async/await code with DispatchQueue (the same rules apply to other blocking primitives). The Root of the Problem The system doesn’t allocate a dedicated thread for every Task. Instead, tasks are executed on a cooperative thread pool, where the number of available threads never exceeds the number of active CPU cores. Therefore, you can cheaply spawn thousands of tasks — they are merely small allocations on the heap, not separate threads. However, a blocking GCD call or an infinite task (a loop) is not a suspension point; they occupy the thread and do not return it to the pool. The more of these tasks there are, the higher the chance of depleting the pool. Each of the methods below leads to this situation in its own way. Method #1. Saturating the Pool With Blocking Tasks The simplest way is to occupy every thread in the pool with a task that blocks it until its execution is complete. An example using DispatchQueue.sync: Swift // The pool size is 2. We launch 2 blocking tasks, each on its own queue. for i in 0..<2 { Task { DispatchQueue(label: "blocking-\(i)").sync { // blocks the thread // some heavy work } print("done") } } Task { print("See you later...") } // stuck in the queue, won't execute anytime soon The task inside sync does not suspend. The thread from the pool waits until the block finishes executing on the queue. If you do this simultaneously on every thread in the pool, its throughput will drop to zero. The pool recovers after the blocks complete. But while they are executing, nothing that lives on it makes progress: non-isolated async functions, regular actors, TaskGroup (@MainActor and GCD queues continue to work in the meantime — the main actor has its own executor on the main thread, and GCD has its own pool). The heavier the task — a synchronous network request, heavy computation, file I/O — the longer the stall. How can this slip through tests? If you only test on powerful devices. If, say, 4 blocking tasks occur simultaneously at runtime, the code might run normally on 8 cores, but then fail on a 2-core CI runner or a low-end device. Additionally Pool exhaustion due to blocking calls is discussed in the Swift Forums thread Deadlock When Using DispatchQueue from Swift Task, where a reader-writer subsystem managed by a TaskGroup deadlocks as soon as a sufficient number of tasks simultaneously block their threads in the pool. The Problem With the Vision Framework A blocking call can be inside third-party code, and you won’t see it in your own. The Swift Forums thread Cooperative pool deadlock when calling into an opaque subsystem describes such a case: a seemingly synchronous Apple API (VNImageRequestHandler.perform from Vision) internally drops down into GCD and blocks the calling thread. Just a few concurrent tasks calling it are enough to exhaust the cooperative pool and hang the entire application. Method #2. Creating a Deadlock Between Queues Thread starvation is temporary if the blocking call eventually finishes. To make it permanent, you need to arrange it so that two blocked threads wait for each other. Swift let queueA = DispatchQueue(label: "A") let queueB = DispatchQueue(label: "B") Task { queueA.sync { // holds the pool thread on A... queueB.sync { } // ...then waits for B } } Task { queueB.sync { // holds the pool thread on B... queueA.sync { } // ...then waits for A → circular wait } } The queueA block won't complete until queueB is freed, and the queueB block won't complete until queueA is freed. Important: This is not a guaranteed deadlock. It only happens if both outer sync calls manage to capture their queues before the inner sync calls execute. If the first task completely finishes before the second one starts, nothing will happen. This can lead to intermittent (flaky) bugs. Method #3. Creating a Deadlock on a Single Queue Variant 1. Two Nested sync Calls A familiar situation: Swift let queue = DispatchQueue(label: "serial") Task { queue.sync { // blocks the cooperative thread // ...work... queue.sync { } // sync on the same serial queue } } In practice, this will more likely result in a crash rather than a hang. libdispatch recognizes the simple case — the thread already owns the queue and calls sync on it again — and intentionally crashes the application with EXC_BAD_INSTRUCTION and the message BUG IN CLIENT OF LIBDISPATCH: dispatch_sync called on queue already owned by current thread. This applies to a serial queue. A nested sync on a concurrent queue will not cause a deadlock, but it will still hold the pool thread. sync deadlocks between queues and on a single queue are well-known GCD "pitfalls"; it is easy to fall into them in a cooperative pool of limited size. Variant 2. A Hidden Reentrant sync and a Single Queue for Everything The blocking call can be hidden behind an innocent helper function. For example, in a seemingly safe synchronous accessor like this: Swift let queue = DispatchQueue(label: "store") func currentUser() -> User { // used throughout the code queue.sync { _user } // fine — as long as you are not on `queue` } And now someone somewhere starts work on the same queue and calls this helper from within: Swift Task { queue.sync { // now executing ON `queue` let user = currentUser() // currentUser() calls queue.sync again apply(user) // the same serial queue → crash } } Each call looks normal on its own. The problem only arises when they are combined, and its two halves might reside at opposite ends of the codebase. As a result, the application crashes with the same libdispatch message as in Variant 1, but the stack trace doesn't immediately reveal that two "normal" halves of code from different files are to blame. Method #4. Not Keeping Track of @MainActor The main thread is not part of the cooperative pool; @MainActor has its own executor on the main thread. But the scheduling model is the same — cooperative — and a blocking sync breaks it in exactly the same way: Swift @MainActor func onTap() { let worker = DispatchQueue(label: "load") worker.sync { // blocks the main thread, the UI freezes let data = loadDataSync() DispatchQueue.main.sync { // worker is now waiting for main... render(data) // ...but main is blocked above → deadlock } } } Blocking the main thread stops rendering, gesture processing, and run loop events. The user sees a frozen screen, and the watchdog might kill the application. Method #5. Not Suspending Heavy Synchronous Tasks Without GCD or any primitives. A task performing long synchronous work between await points also does not yield its thread back: Swift Task { while true { heavySynchronousWork() // never reaches an await } // holds its thread forever } In a cooperative pool, the runtime can only reassign a thread at a suspension point. No await means no yielding. From the pool's perspective, a tight CPU loop without an await is indistinguishable from a blocking call; it just does useful work while 'starving' everyone else. A possible solution is to break the long-running work into chunks with await Task.yield() between them: Swift Task { while !Task.isCancelled { heavySynchronousWork() await Task.yield() } } Apple’s documentation for Task.yield() describes it as suspending the current task to allow other tasks to execute. But this is not an ideal solution, because between yield points, the work still occupies a pool thread. There is another option: moving the heavy work out of the pool entirely, for example, via GCD + continuation or a separate executor. How Not to Break Swift Concurrency Do not call long-running tasks under blocking primitives or queue.sync inside a Task. Short critical sections under a fast lock (os_unfair_lock, NSLock, an instantaneous queue.sync around a field read) are acceptable: the thread holding the lock will perform the work itself and release it immediately.Call callback APIs using continuations. To turn a GCD API with a completion handler into an async function, wrap it in withCheckedContinuation (or withCheckedThrowingContinuation when an error is possible). The continuation suspends the task and resumes it from the callback without blocking the thread.Keep blocking sync calls from the same queue in one place. If a public function blocks the thread, indicate this explicitly (via its signature or a comment) or use async.Watch out for calls within @MainActor methods. Do not call heavy tasks under sync on the main thread, with the exception of a short sync for the sake of an atomic read. Launch heavy work in a separate Task or queue and update the UI asynchronously.Use suspension in heavy loops. Insert await Task.yield() so that a long (or infinite) task does not hijack a pool thread for itself, or move the work out of the cooperative pool.Test on low-end devices and under load. In an environment with 1–2 cores or on a pool saturated with concurrent tasks.

In Part 1, we solved one direction of the multi-cloud connectivity problem: a workload running in Google Cloud interacting with an AWS cloud resource. A GKE pod read a Google-issued OIDC token from the metadata server, handed it to AWS STS via AssumeRoleWithWebIdentity, and received short-lived AWS credentials, with no static access keys stored anywhere. MultiCloudJ wrapped the token dance behind a portable client so the application code never touched a provider SDK directly. This article covers the return trip: a workload running in AWS calling into Google Cloud — specifically, an Amazon EKS pod reading and writing a Google Cloud Storage (GCS) bucket — again with zero long-lived credentials. The zero-trust principle is identical. The mechanism is a bit different. And that asymmetry is the single most important thing to understand before you build it. Authentication Flow from AWS to GCP 1. Build a SigV4-signed GetCallerIdentity request (signed with STS credentials): It's assumed that the EKS pod already holds temporary AWS credentials. 2. Call sts.googleapis.com for token exchange: The pod sends that signed request to Google Cloud as the input to an OAuth 2.0 token exchange. It is asking Google, "Here is proof of who I am on AWS - please give me a Google token to access cloud resources." 3. Replay the GetCallerIdentity signed request: Google does not trust the request blindly. It runs the signed request against AWS STS on the caller's behalf. 4. Response with ARN: AWS checks the signature and replies with the caller's ARN (the AWS role identity) as part of the GetCallerIdentity response. Now Google knows exactly which AWS identity is asking - proven by the signature, with no shared secret. 5. Validate the ARN with the pool: Google checks that ARN against the Workload Identity Pool rules - which AWS account and which role are allowed in, and how the ARN maps to a Google identity. 6. Access token: Once the ARN passes, Google returns a short-lived access token to the EKS pod. 7. Access the resource with the access token: The pod uses that token to read and write Cloud Storage. When the token expires (usually within an hour), the flow repeats. Nothing long-lived is ever stored. Summary: AWS proves the pod's identity by answering Google's replayed request, and Google issues a short-lived token based on that proof. No access keys, no service-account key files - just a signed request and a temporary token crossing the trust boundary. Please note that this authentication flow can be used for any cloud service and is not specifically for cloud storage. Direct Pool Access vs. Service Account Impersonation Once Google has verified the caller's AWS identity through the signed request, it still has to map that AWS identity to something that actually holds permissions on the bucket. There are two ways to do this mapping, and you should pick one before you grant any IAM role. Option 1: Direct Pool Access You grant the Cloud Storage role straight to the federated identity. In IAM, the member looks like this: principalSet://iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/* The permission sits on this pool principal, not on the AWS role. The AWS role never holds any GCP permission. Its only job is to prove identity: it answers Google's replayed GetCallerIdentity request so Google knows which AWS identity is asking. Google then checks that identity against the pool rules and, if it is allowed in, treats the caller as this pool principal. The bucket role, such as roles/storage.objectAdmin, is bound to that principal, so that is where the actual access comes from. No service account sits in the middle. In your code, the value you pass is the pool provider resource name (the audience), and Google issues a token that represents the pool identity directly. Option 2: Service Account Impersonation You create a GCP service account, grant that service account the bucket role, and then let the federated identity impersonate it. The federated identity needs roles/iam.serviceAccountTokenCreator on that service account, and the exchange gets a second hop: first a pool token, then an impersonated service-account token. In your code, the value you pass is the service account email. Which to Choose For a straight AWS EKS to GCS case like this one, direct pool access is the better default: Fewer moving parts. No service account to create, no token-creator grant to manage, and no second token hop.Tighter blast radius. The bucket permission is tied to identities coming through this specific pool, not to a service account that other workloads might also be able to impersonate. You can narrow it further to a single AWS role with an attribute condition on the principal.Less to audit. One IAM binding on the bucket tells the whole story. Reach for impersonation only when you actually need what a service account gives you: You must reuse an existing service account that already carries permissions across many GCP resources.A downstream Google API or tool only understands service-account identities and cannot evaluate a principalSet:// member.Your organization standardizes on service accounts as the single unit of access, to stay consistent with other human and machine grants. In short, direct pool access is simpler and safer, so use it unless a concrete requirement forces impersonation. Set Up Workload Identity Pool on GCP Before any code runs, you configure the trust relationship on Google Cloud once. Three things: a pool, an AWS provider inside it, and an IAM grant on the bucket. Create the Workload Identity Pool: The pool is the identity container that your AWS workloads will be represented as.gcloud iam workload-identity-pools create aws-pool --location="global" --display-name="AWS workloads"Create the AWS provider inside the pool: The provider is the entry gate. It tells Google to trust GetCallerIdentity results from a specific AWS account, how to map the caller's ARN into a Google attribute, and which callers are allowed in.Two important parts here: The attribute mapping turns the caller's raw ARN into a stable attribute.aws_role value with the session name stripped, so grants survive session rotation.The attribute condition is the first gate: only callers from your AWS account are admitted, before any IAM binding is even checked. Shell gcloud iam workload-identity-pools providers create-aws aws-provider \ --location="global" \ --workload-identity-pool="aws-pool" \ --account-id="123456789012" \ --attribute-mapping="google.subject=assertion.arn,attribute.aws_role=assertion.arn.contains('assumed-role') ? assertion.arn.extract('{account_arn}assumed-role/') + 'assumed-role/' + assertion.arn.extract('assumed-role/{role_name}/') : assertion.arn,attribute.account=assertion.account" \ --attribute-condition="assertion.account == '123456789012'" Grant the bucket role to the pool principal: This is the direct pool access model. The permission binds to the AWS role (via the mapped attribute), not to a service account. Shell gcloud storage buckets add-iam-policy-binding gs://my-archive-bucket \ --role="roles/storage.objectAdmin" \ --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/aws-pool/attribute.aws_role/arn:aws:sts::123456789012:assumed-role/my-eks-role" After this, the EKS pod's role can federate into the pool and read/write the bucket, and the application code in the next section never touches any of this setup again. Implementation With MultiCloudJ MultiCloudJ exposes the same BucketClient abstraction you saw in Part 1; you build it for the "gcp" provider and attach a CredentialsOverrider that carries the federated identity. The library handles the SigV4 signing, the STS token exchange, and (on the impersonation path) the generateAccessToken call internally; your code just does blob operations (full example). Java private static final String REGION = "us-west-2"; // The audience is the full Workload Identity Pool provider resource name. // We grant the bucket role directly to this pool principal (direct pool // access), so no service account sits in the middle. private static final String AUDIENCE = "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider"; // The supplier runs on every GCP token refresh. Each time it signs a fresh // GetCallerIdentity request with the pod's AWS role (IRSA, picked up from the // ambient AWS credential chain) and returns the subject token GCP expects. Supplier<String> webIdentityTokenSupplier = GcsFromAws::buildSubjectToken; CredentialsOverrider overrider = new CredentialsOverrider.Builder(CredentialsType.ASSUME_ROLE_WEB_IDENTITY) .withRole(AUDIENCE) .withWebIdentityTokenSupplier(webIdentityTokenSupplier) .build(); // Portable client: same API as the AWS side in Part 1, // only the provider string changes. BucketClient bucketClient = BucketClient.builder("gcp") .withBucket("my-archive-bucket") .withCredentialsOverrider(overrider) .build(); ListBlobsPageResponse page = bucketClient.listPage(ListBlobsPageRequest.builder().withMaxResults(10).build()); page.getBlobs().forEach(b -> System.out.println(b.getName())); // Signs a GetCallerIdentity request with the pod's AWS role, then shapes the // signed request into the URL-encoded JSON envelope that Google STS expects // as an AWS4 subject token. private static String buildSubjectToken() { // Google requires the audience to travel inside the signed headers, so it is // bound to the signature and the request cannot be replayed against any other // target. SignOptions options = SignOptions.builder() .withCustomHeader("x-goog-cloud-target-resource", AUDIENCE) .build(); StsUtilities stsUtil = StsUtilities.builder("aws").withRegion(REGION).build(); // Passing null means "just sign a GetCallerIdentity request, there is no // service payload to hash." The library fills in Action=GetCallerIdentity. SignedAuthRequest signed = stsUtil.newCloudNativeAuthSignedRequest(null, options); JsonObject envelope = .. // construct json object from signed request uri return URLEncoder.encode(envelope.toString(), StandardCharsets.UTF_8); } Conclusion Part 1 showed GCP calling AWS, and Part 2 completes the picture with AWS calling GCP. Both use the same idea: federation, no static keys, and only short-lived credentials. They differ only in how identity is proven. GCP to AWS presents a Google OAuth identity token, while AWS to GCP sends a signed request that GCP verifies with AWS. This is exactly where MultiCloudJ earns its place. All of these provider-specific differences, such as the bearer token here, the signed request and replay there, the STS token exchange, the service-account impersonation, and the token refresh, are abstracted away inside the library. You build one portable client, attach a credentials overrider, and call the API. Your application code never learns which cloud it is talking to or which way the call is going, so it stays clean, portable, and free of long-lived secrets in both directions.

Picture the scene: One of the services in your backend is a mature Django app that no one has the resources, time, or, frankly, the will to rewrite. The ORM, the admin panel, and the broader ecosystem all earn their keep. But you’re looking for the best way to describe your API, and FastAPI catches your eye. It looks like a great fit: native typing, pydantic-based validation, OpenAPI out of the box, and of course the support for async endpoints. That's the situation our team found itself in - we decided to use both frameworks and take from each what suited us best. Not everything went smoothly — this post is what we built, what broke afterward, and what we learned. The First Win So we wired it up, and it works. FastAPI runs as the ASGI application, and the existing Django app plugs into it. Python # asgi.py import os os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings") from django.core.asgi import get_asgi_application from fastapi import FastAPI app = FastAPI() django_app = get_asgi_application() app.mount("/legacy", django_app) Great! Now: Both Django and FastAPI endpoints live side by side, with no pressure to refactor everything in a single day — that was important for us.In the new parts of the app, Django steps back into a single role: communicating with the database through its models.Endpoints can be either sync or async. That was the win. But there was the other side also. Pitfall 1: Async Endpoints Started Running One at a Time When you reach out to external services, chances are you also want to enrich the request with something from your database, or save the result back to it (we did). Here's a tiny example: A single async handler that fetches data about Order from the database (we use Postgres) and forwards it to an external payment provider. Python from asgiref.sync import sync_to_async from fastapi import FastAPI app = FastAPI() @app.post("/orders/{order_id}/dispatch") async def dispatch_order(order_id: int) -> OrderDTO: order = await sync_to_async(get_order)(order_id) # fetch from DB await client.send_order(order) # call external service return order # code that uses a Django model def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) return OrderDTO(id=order.id, amount=order.amount) Inside an async function, you can’t call the Django ORM synchronously. The documented approach is sync_to_async, which moves the synchronous call to a separate thread so it doesn’t block the event loop. Now let's see what happens under concurrent load. Drop a three-second sleep into get_order: Python from django.db import connection def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) with connection.cursor() as cursor: cursor.execute("SELECT pg_sleep(3);") return OrderDTO(id=order.id, amount=order.amount) And fire three requests in parallel: Shell URL="http://localhost:8000/orders/1/dispatch" curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" >> 3.012s >> 6.024s >> 9.037s We expected ~3 seconds and got nine. The handlers ran one after another, not concurrently. And if you log the thread and database connection IDs from inside get_order, all three requests print the same values. Why? By default sync_to_async(get_order) runs with thread_sensitive=True, which means the function runs in the same thread as all other thread_sensitive functions. A standalone Django ASGI app does extra work here: it opens a fresh context per request, so requests run in parallel. The benchmark suggests that in our setup FastAPI doesn't: all three sync_to_async calls land on the same thread and line up one behind another. The event loop itself stays free, by the way: a purely async route keeps responding while the three /dispatch requests wait in that queue. But three async handlers with ORM calls queue up on the same thread, sharing the same connection. For a moment we hoped Order.objects.aget(...) or other Django async ORM helpers would save us here. They won't: for now under the hood they call the same sync_to_async. Can we just flip to sync_to_async(..., thread_sensitive=False)? Probably not - it is not a safe default. Django carries a lot of per-request state in thread-locals: the current DB connection, transaction.atomic(), etc. The Django docs say: "a lot of existing Django code assumes it all runs in the same thread." What to Do About It No silver bullet, but two approaches hold up: Split handlers by what they touch. Reserve async def for endpoints that genuinely don't touch the ORM — async-native HTTP calls, cache reads, etc. For ORM-bound endpoints, declare them as plain sync routes. FastAPI runs sync routes on its thread pool, so they actually run in parallel, and each thread gets its own Django connection. As long as these endpoints don't make many slow external calls, this can work.Move the work out of the handler entirely. If your project already runs with a message broker, the possible answer to "external API + DB write inside a handler" is to stop doing it inside a handler at all. Drop an event on the bus, let consumers handle the side effects, return immediately. The catch: this only makes sense when an event-driven flow already fits your system — because it is, of course, no small refactor. Pitfall 2: Tests That Can't See Their Own Data Now let's write a test for get_order — a sync endpoint that reads an order from the DB. The test runs with pytest-django: we create an order in the database and call the handler. Python # app.py import pytest from fastapi import FastAPI, HTTPException from fastapi.testclient import TestClient app = FastAPI() @app.get("/orders/{order_id}") def get_order(order_id: int) -> OrderDTO: try: order = Order.objects.get(id=order_id) except Order.DoesNotExist: raise HTTPException(status_code=404) return OrderDTO(id=order.id, amount=order.amount) @pytest.mark.django_db def test_get_order(): Order.objects.create(id=1) response = TestClient(app).get("/orders/1") assert response.status_code == 200 # and we'll have 404 You get 404 Not Found. The handler ran, looked at the database, and the order was nowhere to be found. Four facts conspire here: Pytest runs your test's data setup in one thread; when the FastAPI test client calls the endpoint, the handler runs in another.pytest-django wraps every test in an open transaction and rolls it back at the end. That's how the suite stays fast and isolated. The transaction lives on a single database connection.Django opens a database connection per thread.Postgres defaults to READ COMMITTED isolation: one connection cannot see another connection's uncommitted writes. So: the test body runs in the pytest thread. Its Order.objects.create(...) uses connection 1, inside pytest-django's open transaction. When TestClient hits the endpoint, FastAPI dispatches the handler to a worker thread from its thread pool, on another thread with its own connection 2. Connection 2 looks at the database and sees no order, because connection 1 hasn't committed, so connection 1's write is effectively invisible to everyone else. Again — What to Do? Test in layers. Unit-test the endpoint contract with the ORM mocked - those tests don't cross thread or connection boundaries, so the visibility problem simply can't appear. Test business logic and data access in their own tests, without going through TestClient. For cases when the full end-to-end test is still needed - the commonly suggested fix is @pytest.mark.django_db(transaction=True). This switches the test to a mode where writes actually commit, so other connections can see them. But it has its cost: pytest-django now does a database flush after every test, and the suite gets noticeably slower. On a large suite, for us "noticeably" meant minutes - too much on every run, so we use it only for exceptional cases. The Recap FastAPI brings obvious wins — OpenAPI docs, clean endpoint code, typing all the way through; Django gives you a greatly tested ORM and admin. Putting them in the same process gives us both — and a thread-and-connection model that doesn't behave the way we'd expect. Budget for the architecture work before you budget for the migration. Was it worth it? Yes — we got the clean, typed API we were after, and we kept Django's ORM instead of porting the whole data layer to another framework. Would we do it again? Not sure. The trade-offs of this integration may outweigh its benefits for us, so other combinations might be a better fit. If you’ve run into the same solution and found an approach with better trade-offs, please share; the comments are open. Reproduce it yourself. An example with a benchmark and failing tests is in https://github.com/evchibisova/fastapi-over-django-test.

I spent years building data pipelines, mostly in Snowflake, in a regulated banking environment. For most of that time, lineage was straightforward: data moves through a transformation, and you can trace exactly where every number came from. That changed the moment LLM functions started showing up inside those same pipelines. The pattern showed up the same way every time. A Cortex function would generate a narrative, a summary, a piece of text meant for a report someone downstream would rely on. The data going into the report was fully traceable. The text coming out of the LLM was not. I could tell you which table fed a number. I could not tell you which prompt, which model version, or which configuration produced a specific sentence. That gap kept showing up, and it bothered me enough that I eventually went and checked whether the tools I was using were ever going to close it on their own. The realization did not come from a single dramatic moment. It came from working backward. After a Cortex function ran and produced output that ended up in a report, I tried to reconstruct what had actually happened: which prompt had been active, what parameters had governed the run. The query history showed the function had executed. It showed the timestamp, the user, the warehouse. What it could not show me was what had been sent to the model or what version of the prompt template had produced the result. I was looking at evidence that something had happened, with no record of what that something actually was. They are not going to close it. I looked across the major data governance and lineage tools commonly used in this space: dbt, MLflow, Apache Atlas, Snowflake's own native tooling, Informatica. Every one of them is genuinely good at tracking structured data through deterministic transformations. Table versions, transformation logic, pipeline runs, all well covered. None of them, as far as I could find, natively captures what happens the moment an LLM enters the picture: which prompt template was used, what version of it, what parameters the model ran with, or how the output maps back to a specific section of a specific report. That is not a criticism of those tools. They were built before this problem existed in its current form. But it means that right now, if someone asks you to reconstruct exactly how an AI-generated paragraph in a regulated report came to exist, in most environments, you cannot. You have the output. You do not have the chain that produced it. The Question That Kept Coming Up The question that kept surfacing in compliance conversations was some version of: which version of this process produced this output? Not just which data, not just which model, but which version of the entire process, prompt included, was active at the time a specific report section was generated. That question is unanswerable with standard data governance tooling, because prompts are not treated as versioned process components the way SQL transformations are. A dbt model gets a version, a run ID, a test result. A prompt template gets saved somewhere, maybe, by someone, whenever they remember to. The governance gap is not subtle. It is the difference between a process that is version-controlled end to end and one that treats its most consequential step as an untracked artifact. Regulatory frameworks are beginning to reflect this expectation even if they do not yet spell out the technical solution. The EU AI Act, in Article 12, requires that high-risk AI systems allow for the automatic recording of events over the lifetime of the system. That language is more specific than most summaries suggest: it rules out manual log exports or after-the-fact human notes as a substitute. It requires automatic, system-level capture. That is exactly the kind of infrastructure that does not exist in most LLM reporting pipelines today. The Fix: Build It Into the Pipeline The fix, for me, was not to wait for a vendor to solve this. It was to treat prompt lineage as something that belongs inside the pipeline from day one, not something to bolt on after the fact. Concretely, that meant logging the prompt template and its version, the model and its configuration, and a hash of the output, automatically, every time the function ran, as part of the same process that writes the report, not as a separate step someone has to remember to do later. The architecture has six layers, each capturing a specific category of governed artifact. Source data provenance tracks which tables and rows fed the model. Transformation logic captures which pipeline version prepared the data. Prompt construction records exactly what was sent to the model, including template ID, version, variables, and rendered prompt hash. Model parameters log the specific model version, temperature, and inference settings. Output integrity creates a tamper-evident hash of the generated text. Report context maps the output to a specific filing section, including approval records. If You Can Only Start With One Thing If I could only implement one layer first, I would start with output hashing. The reason is practical: everything else in the lineage chain can potentially be reconstructed or approximated after the fact. You can check version control for the prompt template. You can look at model documentation for parameters. But once a generated output has been filed in a regulatory document and time has passed, there is no way to prove retroactively that what was filed matches what the model produced, unless you captured a hash at the moment of generation. Output hashing is the layer that makes the rest of the chain defensible. Without it, even a complete lineage record can be questioned, because you cannot prove the output it describes is the output that was actually filed. What to Do Starting Now A few things I would tell another data or IT leader looking at this same gap: Inventory every place an LLM touches something that ends up in a regulated or customer-facing document. You cannot fix what you have not mapped.Do not assume your existing data governance stack already covers this. Check specifically whether it captures prompt versions and model configuration, not just source data.Build the logging into the pipeline itself, not as a side process. If it is optional or manual, people will skip it under deadline pressure, and you will be back where you started.Start with output hashing if you have to prioritize. That single layer gives you tamper-evident proof of what was generated, which is the foundation everything else depends on.Treat this the same way you treat any other production logging you cannot afford to lose. Once a report goes out, the question is not whether someone will eventually ask how it was produced. It is when.

Not long ago, I broke a backtest without changing a single line of code. I moved the script to a different machine—same OS, supposedly the same Python version — and the equity curve suddenly told a completely different story. Nothing in the logic had changed. The environment was the only obvious difference. That was the day I stopped treating the runtime environment as an afterthought and started treating reproducibility as part of the experiment itself. What I learned is this: a backtest isn’t truly reproducible just because it ran once on your laptop. A result you can’t regenerate reliably isn’t a research finding — it’s a coincidence. And when that coincidence eventually meets real money, it can get expensive fast. What follows is a minimal but complete pipeline for containerizing and automatically testing a Python backtesting system. No over-engineering — just Docker, GitHub Actions, and a few habits that make your results trustworthy. The Real Goal We aren’t building a live trading platform. We’re building a workflow that ensures every change is verified in a controlled environment: Plain Text Code change → automated tests → versioned Docker image build The system must do four things: Produce deterministic results, within an acceptable numerical tolerance, from the same versioned inputs in the same containerized environment.Automatically test every code change before it can be merged.Halt the pipeline when a test fails, with no exceptions.Tag every image build so we can trace it back to the exact source-code version. If a workflow can’t do that, it’s just a script living on someone’s laptop. Project Structure Before containerizing the application, let’s organize the repository clearly: Python backtesting-system/ ├── app/ │ ├── engine.py │ └── main.py ├── tests/ │ └── test_engine.py ├── data/ │ └── sample.csv ├── requirements.txt ├── requirements-dev.txt ├── Dockerfile └── .github/ └── workflows/ └── ci.yml The app/ directory holds the strategy logic, while tests/ remains separate. The data/ directory contains a small, frozen sample dataset that never changes — our gold standard. There are no absolute paths or machine-specific assumptions. Everything that might vary, including the data path, starting capital, and fee rate, comes from environment variables or a configuration file. Containerizing With Docker The classic “works on my machine” problem usually indicates an environment mismatch. Docker reduces this drift by packaging the application and its runtime dependencies into a versioned image. Here is the Dockerfile: Dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app ./app COPY data ./data CMD ["python", "-m", "app.main"] A few decisions matter here. We pin the Python 3.12 image series and can use an image digest when stricter reproducibility is required. Dependencies are pinned to exact versions in requirements.txt, since version ranges can silently introduce changes. We also use slim to keep the image small. Crucially, we never copy local keys, cached results, or temporary files into the image. The container doesn’t guarantee that the logic is correct. It helps ensure that sound logic runs in a controlled and substantially more consistent environment. Adding Tests That Matter Automation without tests simply automates mistakes. Our tests don’t attempt to prove that a strategy is profitable. They prove that the program behaves consistently. We check for: Clear errors when input files are empty or missing.Deterministic results when the same data and seed are used.Correct fee calculations.Rejection of malformed rows rather than silent processing.Required fields in every output. Here is one example, including the necessary imports: Python import pytest from app.engine import run_backtest def test_backtest_is_reproducible(): first = run_backtest("data/sample.csv", seed=42) second = run_backtest("data/sample.csv", seed=42) assert first["trades"] == second["trades"] assert first["final_equity"] == pytest.approx( second["final_equity"], rel=1e-9 ) This test establishes a simple contract: given the same starting conditions, the system will not drift beyond an acceptable margin. The direct comparison of trades works here because we assume that every trade entry uses standardized types such as integers and strings. If the trade records contain floating-point prices, those values should be checked individually with an appropriate tolerance. Building the GitHub Actions Pipeline Now we automate the workflow. It runs on pushes, pull requests, and manual triggers through workflow_dispatch: YAML name: Backtest CI on: push: pull_request: workflow_dispatch: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install -r requirements-dev.txt - run: pytest - run: docker build -t backtest:${{ github.sha } . The steps are straightforward: check out the code, set up Python, install the development dependencies, run the tests, and build the Docker image. The requirements-dev.txt file includes both the production dependencies and a pinned version of pytest: Python -r requirements.txt pytest==8.3.4 If pytest detects a failure, the job stops immediately. The broken change never reaches the image-build step. The resulting image is tagged with the Git commit hash, creating a clear link between the source code and the image built during that workflow run. Managing Configuration and Secrets Environment-specific configuration and secrets should never be baked into the image. Environment variables can control data paths and run modes. If the system is later connected to a live data source, API credentials should be stored in GitHub Secrets or a cloud secrets manager—never in the source code or Dockerfile. Logs must not expose keys or sensitive headers. Even if today’s “production” environment is only a scheduled test run, development and production should use separate configuration sets. Treat every secret as sensitive, and keep environment-specific configuration outside the image. Lightweight Monitoring and a Path to Rollback Once the pipeline runs regularly, monitoring must go beyond asking whether the process is still alive. Useful questions include: Did the latest job complete, or did it hang?Has execution time increased dramatically?Is the input data intact?Were the output files generated, and are they non-empty?Which image version produced the latest results? If images are later pushed to a container registry, retaining the last few stable versions provides a straightforward rollback path. For scheduled backtest runs, we should also archive the data snapshot, parameters, and results. That allows us to return a month later and answer a very specific question: “What exactly did we test on July 24?” These practices aren’t unique to systems we build ourselves. Commercial grid-trading interfaces make automated execution accessible without revealing every part of their internal deployment pipelines. BYDFi is one example I encounter in my work. Because I work with the platform, this is a disclosed reference rather than an independent recommendation. The comparison is conceptual: understanding reproducibility, automated checks, and configuration management helps developers evaluate any automated tool more thoughtfully. The Experiment Isn’t Finished Until It’s Verified We started with a broken backtest and a frustrating realization. Now we have a different mindset. Docker reduces environment drift. Automated tests guard program behavior. GitHub Actions ensures every change passes through the same gate. Monitoring and versioning give us a clear path to detect problems and support rollback as the pipeline evolves. In a reliable backtesting system, reproducibility and verification are not tasks that come after the experiment. They are part of the experiment itself. The moment we treat them that way, our results stop being anecdotes and start becoming evidence. And when the decisions involved can carry real financial weight, evidence is the only thing worth building.