A detection rule and a hunting query may run against the same telemetry, but they want very different architectures.

The detection rule is known before the event arrives. It may need to maintain state across a stream, evaluate every event, and produce an answer within seconds or milliseconds.

The hunting query is created after the fact. It may scan months of history, join several data sources, and tolerate seconds or minutes of execution time without changing the outcome.

We routinely ask both workloads to run in the same place because security architecture has spent years treating data placement as the primary design decision. Decide where the telemetry lives, then put the analytics there.

That made sense when storage was the expensive part of the system. It makes less sense now.

Cheap object storage, high compression ratios, and indexes designed for large historical datasets have changed the economics of centralization. At the same time, federation has made it practical to query data that cannot or should not be moved. Both are useful developments.

Neither tells us where detection should execute.

The more useful design question is simpler:

Where should each workload run, given what it needs from the data and how quickly it needs an answer?

That produces a different architecture.

Storage has three credible answers

There are roughly three ways to handlesecurity telemetry today.

Traditional centralization sends telemetry to a SIEM or lake, normalizes it, indexes it, andretains it according to policy and budget. The advantage is control. You knowwhat you collected, how long you kept it, which schema it uses, and whether aquery covered the complete dataset.

Historically, the problem was cost. SIEMpricing made ingest and retention expensive enough that organizations routinelydiscarded useful telemetry simply because keeping it searchable cost too much.

Federationleaves telemetry in the systems that produced it and queries those systems whenneeded. There is no second copy and potentially much less infrastructure tooperate. It also provides access to sources that are difficult to export.

The compromise is that you inherit eachsource's retention period, availability, API behavior, schema, query semantics,and rate limits. The data may be physically distributed, but so are thedependencies required to answer a question.

Hyper-indexed object storage separates durable storage from the query engine. Telemetry can sit inrelatively inexpensive customer-owned object storage while indexes,compression, and query acceleration make large historical datasets practical tosearch.

This third model matters because itweakens one of the strongest historical arguments against centralization.

If retaining a large dataset no longerrequires paying SIEM-era prices for every gigabyte, "we cannot afford tocentralize this" becomes a narrower objection. Federation still hasimportant uses, but cost alone becomes a less convincing reason to make it thedefault architecture.

There is also something revealing abouthow the indexed-storage category is generally positioned. Its vendors tend todescribe themselves as the data layer beneath the SIEM rather than asreplacements for the detection system itself.

That distinction is useful. Storage anddetection are related, but they are different architectural problems.

Workloadplacement depends on three things

A security workload's natural executionpoint follows from a few fairly mundane properties.

First, do you know the query beforethe data arrives?

Detection logic usually exists inadvance. You know the fields, predicates, thresholds, relationships, and timewindows you intend to evaluate.

A hunting query is often created inresponse to new information. Until the analyst or agent forms the hypothesis,there is nothing useful to pre-position.

Second, does the workload runcontinuously?

Many detections maintain state over astream. A rule may track failed authentications per user, command sequences ona host, or activity within a rolling time window.

A hunt is usually transactional. Submit aquery, receive an answer, decide what to do next.

Third, what is the cost of latency?

If a detection arrives thirty secondslate, an intervention opportunity may already be gone.

If a retrospective hunt takes thirtyseconds longer, the result is still useful.

Those differences matter more to workloadplacement than the price of the storage tier underneath them.

Latency constraints vary

"Detection" is too broad acategory to place as a single workload.

At one end are streaming detections. Theymaintain state and evaluate continuously, typically with latency budgetsranging from sub-second to a few seconds. Sequence detection, per-entitythresholds, and behavioral state machines belong here.

Inline enforcement is the extreme case. Aprocess, packet, or authentication decision may need to return in tens ofmilliseconds. Once the action has been allowed, a later verdict has a differentoperational value.

Then there are detections that tolerateseconds or minutes. Cross-source correlations often fall into this category.Joining identity, endpoint, network, and application context may justify somedelay if the additional evidence materially improves the result.

Finally, there is retrospectivedetection: apply a newly discovered indicator or behavior to historical dataand find previous occurrences.

That last workload looks a lot likesearch, and modern centralized storage architectures are very good at it.Cheap, indexed historical storage is especially valuable here because itextends the period over which retroactive detection remains possible.

Streaming detection has differentrequirements.

Why search isthe wrong execution model for streaming detection

The common criticism of federateddetection is latency. A query fans out to several systems, each system respondsat a different speed, and the slowest dependency determines completion time.

That is true, but latency is only part ofthe problem.

Search and streaming detection havedifferent execution models.

A search starts with a request. Thesystem evaluates it against some dataset, returns a result, and stops.

A streaming detector is already runningwhen the event arrives. It may be holding state from thousands of earlier eventsand continuously updating that state as new ones appear.

Consider a simple rule: maintain afive-minute sliding window per account and trigger when a threshold is crossed.

You can reproduce something similar byrepeatedly querying a database, but polling introduces awkward semantics.Windows overlap or leave gaps. Events can arrive out of order. The samecondition may be rediscovered on consecutive polls. Shortening the pollinginterval increases cost without removing those boundary conditions.

Modern SIEMs have addressed this byadding stream processing to the ingest path. They evaluate rules as telemetryarrives rather than repeatedly querying stored data.

That is a sound implementation. It alsoexposes the remaining constraint: the telemetry still has to arrive.

For detections with a seconds-levelbudget, sending telemetry to a central processing layer may be entirelyacceptable. For enforcement-adjacent decisions, transport, buffering,serialization, and ingest can consume most of the available latency before therule starts evaluating.

The location of the data source thereforematters even when the central engine is very fast.

There is a second constraint that hasnothing to do with performance. Some telemetry cannot be exported at fullfidelity or at the required rate. SaaS security products may rate-limit APIs,delay access, charge for export, expose only summarized records, or provide nopractical streaming path at all.

A centralized detector can only evaluatewhat reaches it.

This leads to a useful property ofdetection logic: because much of it is known in advance, it can be distributedahead of time.

Instead of moving every event to a commonlocation before deciding whether it matters, the system can move the ruletoward the event source and evaluate there.

That does not eliminate centralizedanalytics. It changes what needs to cross the boundary.

Hunting wantsalmost the opposite architecture

Threat hunting starts from a differentpremise.

The question often does not exist untilafter the data has been collected. A new indicator appears. An investigationuncovers an unexpected relationship. An analyst notices behavior that was neverencoded as a detection.

There is nothing useful to deploy inadvance because the system does not yet know what it will be asked.

This is where centralized historicalstorage remains extremely attractive.

A normalized dataset gives you one querymodel instead of a collection of vendor-specific APIs. You control retention.Historical completeness is easier to reason about. The source that generated anevent cannot later remove the only copy. A fifteen-month hunt does not dependon whether fifteen individual products happened to keep fifteen months ofuseful history.

Indexed object storage improves thisfurther by reducing the economic penalty for long retention.

Federation still matters, but itsstrongest use cases are more specific:

• telemetry that cannot be exported at sufficient fidelity
• sources constrained by residency or sovereignty requirements
• datasets whose export cost exceeds their expected analytical value
• systems queried too rarely to justify continuous ingestion
• operational data that only exists inside a vendor-controlled interface

Those are substantial cases. Largeenterprises will probably always have some of them.

They do not imply that all historicalanalysis should be federated.

In practice, a mature environment islikely to use both: deterministic centralized history where collection ispractical, federated access for the remainder.

Agents makethe economics more interesting

Autonomous investigation changes thisbalance because software can generate far more queries than human analysts do.

A human investigation might involvedozens of queries. An agent can issue hundreds or thousands, branch intoseveral hypotheses simultaneously, abandon weak branches quickly, and repeatthat process across every investigation in the environment.

That creates very different loadcharacteristics.

In a federated architecture, eachinvestigation may produce repeated calls into third-party systems. API quotasthat were tolerable for human usage can become constraints on investigationthroughput. Query-driven egress increases. Vendor-specific concurrency limitsbecome part of the security system's control plane.

The workload can also become partiallyadversary-driven. If suspicious activity triggers investigation andinvestigation triggers large numbers of remote queries, activity generated byan attacker can indirectly influence query volume and cost.

Centralized data handles this patternmore predictably. Compute cost still rises with agent activity, sometimessubstantially, but the organization controls the query infrastructure and thedata it is querying. Capacity planning becomes an internal systems problemrather than a collection of external API contracts.

This does not make federation unsuitablefor agents. It makes selective federation more attractive than universalfederation.

The same architectural principle thathelps with detection starts to become useful for investigation as well: move computationtoward the source when you know enough about the computation to do so, and movedata when the question genuinely requires broad, retrospective access.

The layers do not needto exchange raw telemetry

Once detection is distributed, it is easyto picture an architecture in which every component still forwards all of itsunderlying events upward.

That defeats much of the point.

A local detection layer can send a muchsmaller set of information:

• detections and verdicts
• entity state and risk assertions
• supporting evidence
• detector health and coverage
• state transitions
• summaries required for investigation

The central layer can send instructions in the other direction:

• updated detection logic
• changed thresholds or policies
• investigation requests
• response actions
• collection directives

Raw telemetry can still move intocentralized storage where doing so is useful. It simply does not need to movethere as a prerequisite for every detection decision.

This is compatible with any of the threestorage models described earlier. An organization might centralize high-valuetelemetry into indexed object storage, federate into a few SaaS products thatcannot be exported, and run local detection at endpoints, identity providers,network enforcement points, cloud control planes, and regional data nodes.

The storage architecture and theexecution architecture become independent choices.

That is probably closer to how largesecurity systems will actually evolve.

Each modelfails differently

The differences become clearer whensomething goes wrong.

Federated search has a completenessproblem. A source may have rolled past its retention period, hit a rate limit,become temporarily unavailable, or changed its schema. The query can stillreturn a syntactically valid result.

"No matches" and "I couldonly search part of the relevant data" are very different answers, butmany federated systems do not make the distinction obvious enough.

Federation also leaves primary evidenceinside systems that may themselves be compromised. If the attacker controls theapplication, account, or security product that stores the only copy of itstelemetry, the evidence used to reconstruct the intrusion is exposed to thesame failure domain.

A central copy has value here even ifnobody ever runs a detection against it.

Schema is another recurring problem.Federation does not remove normalization work. It shifts more of that worktoward query time.

Open schemas such as OCSF helpconsiderably, particularly as source vendors adopt them natively. They do notremove differences in field availability, semantics, retention, orimplementation quality.

Centralization has its own failure modes.Collection pipelines break. Parsers mis-handle new formats. Export throttlingcreates gaps. Transformations discard fields that turn out to matter six monthslater.

The advantage is mostly operationalvisibility. If you own the collection path, you can instrument it, measure lag,detect dropped events, monitor parser failures, and preserve health history.

Indexed object storage adds another setof trade-offs. Good performance still depends on sensible partitioning,indexing, and data layout. Choices made during ingestion can be expensive torevisit. The storage may belong to the customer while the indexes or encodingremain proprietary, so claims of portability need to be examined carefully.

Cheap retention solves an importantproblem. It does not solve every analytics problem above it.

Architecturereviews should start with workloads

This changes the questions worth askingvendors and internal platform teams.

For each workload, determine itsacceptable latency and where execution occurs. Measure latency from thecreation of the event, not from the point at which the analytics enginereceives it.

Ask what happens when a source cannotprovide complete data. A system should be able to distinguish an empty resultfrom an incomplete result and expose the reason.

Understand what evidence is retainedoutside the source's own administrative boundary and whether that evidence issufficient to reconstruct what the security system knew at a particular pointin time.

Then look at the contracts betweenlayers. What schema carries detections, entities, evidence, health state, and commands?Who controls it? Can another implementation participate without reproducing aproprietary internal model?

Those questions reveal much more aboutthe architecture than whether the product describes itself as centralized,federated, or data-lake-native.

Where this leads

Cheap indexed storage has madecentralized historical analysis far more attractive than it was under theeconomics of traditional SIEMs. Federation remains necessary for data thatcannot reasonably be collected. Most large environments will use both.

Neither should determine where detectionruns.

Streaming detection benefits fromevaluating as close as practical to the point where the relevant data isproduced. The tighter the latency requirement, the stronger that pressurebecomes. Historical investigation has the opposite preference: broad, durable,normalized access to as much history as possible.

The result is a system in which logicmoves more freely than data.

That introduces another problem.Detection logic now runs across endpoints, networks, cloud environments, SaaSplatforms, regional collectors, and other execution nodes. Those nodes havedifferent capabilities, owners, software versions, and failure modes. Rules candrift. Coverage can disappear without anyone noticing. A distributed detectionarchitecture that cannot prove what was running where and when is difficult totrust.

So the hard part is no longer decidingwhere to put the telemetry.

It is operating a distributed detectionsystem without losing control of it.

_{In part two of this multi-part series,I'll cover strategies on managing distributed detection that mitigates"distributed chaos".}