Creating Multi-Modal Sensory Fusion

You will combine visual, auditory, and proximity data into a unified perception manager that determines the AI's current threat priority.

In this lesson, you will learn how to synthesize raw sensory inputs—sight, sound, and proximity—into a single "Threat Score" that allows your AI to make intelligent, prioritized decisions.

The Philosophy of Fusion

Up until now, we’ve treated visual line-of-sight, auditory triggers, and proximity sensors as isolated events. However, an AI that reacts to every footstep while ignoring a visible player is poorly tuned. Multi-Modal Sensory Fusion involves passing these disparate data points into a centralized PerceptionManager that weighs their importance before updating the AI’s Blackboard.

Implementing the Perception Manager

Think of your PerceptionManager as a filter. Each sensor sends data to this manager, which assigns a "Weight" to the input. For example, a visual sighting might carry a weight of 1.0, while a faint sound might only carry a weight of 0.3.

public class PerceptionManager : MonoBehaviour
{
    public Blackboard blackboard;
    private float currentThreatScore = 0f;

    public void ProcessSensorInput(float visualWeight, float audioWeight, float distanceWeight)
    {
        // Fusion logic: Combine inputs into a single perceived threat level
        currentThreatScore = (visualWeight * 1.5f) + (audioWeight * 0.8f) + (distanceWeight * 1.2f);
        
        // Update Blackboard so the Behavior Tree can react
        blackboard.SetValue("ThreatScore", currentThreatScore);
    }
}

Why Fusion Matters

By normalizing these inputs into a unified ThreatScore, your Behavior Tree nodes—specifically Condition Nodes—become much cleaner. Instead of checking IsPlayerVisible, IsSoundHeard, and IsPlayerClose separately, your tree can simply check: if (blackboard.GetFloat("ThreatScore") > 2.0f).

This approach creates a "gradient of awareness." If the AI hears a sound but doesn't see the player, the total score stays below the "Investigate" threshold. If the player then steps into the light, the fused score instantly jumps, triggering a high-priority "Combat" state. By fusing the data before it hits the logic layer, you prevent "decision flickering" where an AI constantly switches states because its sensors disagree.

Key Takeaway: Sensory Fusion abstracts raw, noisy data into a single, prioritized threat value, allowing your AI to make calm, informed decisions rather than reactive, twitchy ones.