Summary / 34. Spatial reasoning / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper: A Physical World to Perceive, Explore, and Influence **Concept paper · Piper cognition / embodied-adjacent agency · 12 October 2026** **Status:** Ideas and design directions, not an approved implementation plan **Related cognition components:** 2 Attention · 5 Representation · 14 Prediction · 17 Learning · 18 Generalization · 21 Exploration · 25 Self-correction · 27 Motivation/salience · 28 Agency · 34 Spatial reasoning · 37 Cognitive control > **Thesis.** Piper does not need a robot body to interact meaningfully with the physical world. A network of sensors and carefully limited actuators can give her observations that originate outside her language model, an evolving model of place and change, and the ability to perform small, reversible experiments. The goal is not a more elaborate smart home. It is a richer *environment* in which Piper can notice, wonder, investigate, predict, act, and learn. --- ## 1. Why this is interesting Today most of Piper's world arrives as text: source code, websites, work records, agent conversations, benchmarks, logs, and other people's instructions. Even her engineered-serendipity channel deliberately manufactures novel input. Physical instrumentation offers a different kind of novelty: observations generated by processes that do not care what Piper expects or wants to see. A temperature fluctuation, movement in an outdoor camera, or an unexpected rise in power consumption can contradict her beliefs. That is useful. The strongest version of this idea is a closed loop: **Observe → represent → notice an anomaly → generate alternatives → decide whether to investigate → predict → perform an authorized test → measure → update the world model.** The loop may stop at any point. Sometimes the intelligent response is to log an observation and leave it alone. A spike in sensor activity is not, by itself, evidence of curiosity, learning, or agency. This is **embodied-adjacent cognition**: connection to the physical world without locomotion or manipulation. It could eventually support robotics, but does not depend on it. ## 2. What Piper already has The starting point is equipment already present in the user's environment, subject to confirming exact models and integration capabilities: | Existing equipment | Possible observations | Possible actions or inferences | Important limitations | |---|---|---|---| | Airthings monitor | Temperature, humidity, radon and model-dependent air-quality readings | Learn baseline patterns, correlate events, forecast environmental changes | Check the exact device: do not assume it measures pollen directly; sensors may provide particulates or other proxies instead. Radon readings are often averages, not minute-by-minute truths. | | Outside cameras / Uniview NVR | Motion events, still images, time-of-day patterns, visibility, snow/rain cues | Notice recurring wildlife, correlate weather and motion, investigate novel events | Camera field of view is partial; motion alerts are noisy; images require privacy and retention controls. | | TP-Link Omada Wi-Fi / Aruba network | Client associations, access-point changes, device reachability | Coarse device presence and network topology | A phone appearing on Wi-Fi is not proof that a particular person is in a room. Actual RF motion sensing needs suitable CSI-capable hardware/firmware. | | Smart plugs | Relay state; energy use on metering models | Observe equipment cycles, control a designated noncritical load | A plug reporting “on” does not prove the attached device operated; never control critical loads. | | Smart bulbs | Power state, colour, brightness | Tiny, visible, reversible physical-world actions | Confirm there is an independent observation path if using a bulb to test actuation. | | OPNsense / infrastructure monitoring | Network reachability, link and service events | Relate the digital topology to rooms, devices and capabilities | Read-only first. Never expose administrative access as a generic agent tool. | Airthings integrations, camera protocols, radio support, and smart-device APIs must be verified against individual models before purchase or integration. ## 3. A sensory playground: inexpensive and strange devices Buying unusual modules from AliExpress could be genuinely productive **if every device contributes a new observational dimension**, rather than another dashboard. Prefer inexpensive, local-protocol, documented modules with known drivers over opaque cloud-locked gadgets. ### A. Spatial presence and movement - **24 GHz mmWave presence modules (for example, Hi-Link LD2410B/C):** detect presence, including some stationary people, with moving/static target measurements. A useful complement to ordinary PIR motion sensors, not a reliable person identifier. Home Assistant has a local LD2410 BLE integration for supported variants. - **PIR sensors:** cheap coarse motion events; useful as a second source for disagreeing with mmWave. - **Time-of-flight distance sensors:** measure distance to an object within a constrained field, useful for doorways, bins or experimental tabletop scenes. - **Door/window magnetic contact sensors:** clear discrete changes that can anchor temporal and causal reasoning. - **Pressure mats / load cells:** occupancy or object weight changes in clearly designated experimental locations. - **Thermal array sensors (e.g., low-resolution infrared arrays):** coarse hot/cold patterns without recording identifiable video. Resolution and environmental noise limit interpretation. ### B. Environmental perception - **Barometric pressure sensor:** weather-front signatures and indoor/outdoor differences. - **Light, UV and spectral sensors:** distinguish day/night and different kinds of illumination; explore how illumination influences camera interpretation. - **CO₂, VOC and particulate sensors:** indicators of ventilation and air-quality changes; units and calibration matter greatly. - **Sound-level sensors:** measure noise intensity without storing speech; use only with household consent and limited retention. - **Soil moisture sensors:** plant-care experiments and slow-timescale predictions. - **Rain, wind and leaf-wetness sensors:** link outdoor conditions to Airthings/camera observations. ### C. Physical processes and mechanical events - **Vibration / accelerometer modules:** detect motor cycles, knocks, equipment activity, or unusual patterns without microphones. - **Current clamps or metering plugs:** examine power cycles and energy cost; have mains-rated installations handled appropriately. - **Water-leak sensors:** useful safety observations, but leave safety-critical response rules deterministic and independent of Piper. - **Temperature probes on noncritical equipment:** compare room temperature with object temperature. - **RFID/NFC tags:** allow tagged objects to announce arrivals at a fixed reader, not passive universal location tracking. ### D. Unusual ideas worth exploring later - **Software-defined radio (receive-only):** observe permitted radio phenomena in the local environment, not intercept private communications. - **Geiger counter / radiation logger:** an independent physical signal with different statistics and sampling requirements from air-quality sensors. - **Magnetometer:** detect local field changes or infer limited mechanical activity. - **Microphone-free ultrasonic distance sensing:** measure short-range geometry; avoid transmitting invasive or hazardous power levels. - **Plant or terrarium instrumentation:** a rich, slow experimental system with moisture, light, temperature, growth images, and reversible interventions. - **A dedicated experimental LED, relay or servo:** introduce controllable output without immediately giving Piper a robot arm or access to household infrastructure. **Practical acquisition rule:** select *one* new modality at a time. For each candidate ask: What observation is genuinely new? Is data accessible locally? How will Piper tell whether the sensor is wrong? What is the privacy/safety burden? What question might this device enable that cannot be answered today? ## 4. Piper's map of her world Spatial reasoning does not have to start with navigating a building. Begin with a hybrid graph that includes rooms, devices, services, and observation footprints: - **Places:** office, living room, exterior driveway, server rack, garden, and logical network zones (only where explicitly supplied). - **Entities:** cameras, access points, Airthings, sensors, switches, bulbs, computer hosts. - **Relationships:** `located_in`, `observes`, `powered_by`, `connected_to`, `controls`, `near`, `depends_on`, `reachable_from`. - **Events:** occupancy change, rain begins, plug toggles, device disconnected, temperature rises. - **State:** current reading, units, timestamp, validity, confidence, and source identity. - **Unknowns:** camera blind spots, overlapping footprints, uncertain device-to-room assignments, contradictory sensor readings. A camera's observation footprint, a Wi-Fi AP's approximate coverage, and an mmWave sensor's detection region are not interchangeable. Represent their uncertainties explicitly. Do not create a fictional precise floor plan from network associations. **Useful query:** “Which instruments observed the driveway during the last fifteen minutes?” **More demanding query:** “If the outdoor camera is offline, what physical changes can I still detect, and what becomes unobservable?” **Spatial query:** “What additional sensor would most reduce uncertainty about whether the workshop door was opened?” This overlaps with Component 34 but is not identical to it. Reading sensors is perception; managing truth and timestamps is representation; finding routes through dependency graphs is topology; knowing relative distances and fields of view is spatial reasoning; acting and measuring results is agency. ## 5. The architecture: one gateway, not dozens of raw credentials ```text PHYSICAL WORLD Airthings · outdoor cameras · motion/mmWave · plugs · bulbs · novelty sensors | [Home Assistant / ESPHome / local integrations] | Normalized read-only event and state feed | [Piper sensor gateway: validation + provenance] | +-------------+-------------+-----------------+ | | | | world model belief store salience prediction log | | | | +-------------+-------------+-----------------+ | Piper reasons / asks / explores | [separate authorization and action gateway] | Approved device actions + independent feedback ``` **Home Assistant** is a strong candidate because many devices integrate with it, and **ESPHome** can connect small, inexpensive sensor boards through a local API. Bluetooth proxies can extend coverage. The exact compatibility of each product is still a purchase-time check. Piper should consume a *normalized feed*, not raw vendor-specific payloads. An observation might look like: ```json { "event_id": "obs_00617", "observed_at": "2026-10-12T18:21:00Z", "received_at": "2026-10-12T18:21:04Z", "source": "airthings_living_room", "place": "living_room", "metric": "relative_humidity", "value": 58.2, "unit": "%", "quality": "device_reported", "is_stale": false, "raw_reference": "gateway_event_00617" } ``` Keep raw sensor readings distinct from conclusions such as “someone entered” or “a window is open.” Each inferred claim should link to evidence and have a way to be disproved. Model time-to-live and sensor outages rather than silently retaining the last known value as current truth. ## 6. What could she actually do with it? ### Scene 1: The environmental naturalist Piper notices that indoor humidity tends to rise after a specific combination of rain and temperature changes. She proposes competing explanations, logs a prediction, and checks the next observation. No actuator is necessary. The payoff is experience with observation, disagreement, and uncertainty. ### Scene 2: The nocturnal wildlife observer An outdoor camera repeatedly reports motion. Piper samples an approved set of stills, distinguishes likely animals from headlights or blowing foliage, and starts a journal of appearances and environmental context. She can develop questions of her own: “Do these sightings cluster after rain?” It should not identify or profile people. ### Scene 3: A deliberately tiny act of agency Piper proposes a hypothesis about whether an experimental lamp is working. With pre-granted permission, she turns the lamp on, checks an independently measured light sensor, and returns the lamp to its original state. This distinguishes **issued action** from **observed effect**. ### Scene 4: Diagnosing instrument disagreement Wi-Fi reports that a phone is associated with an AP, but a nearby presence sensor reports no detected target. Piper should not declare either sensor a liar or infer that a particular person is present. She can list alternative hypotheses: phone left behind, AP range, stale association, sensor blind spot, or sampling mismatch. ### Scene 5: The self-directed science project Piper notices a repeating vibration pattern from a designated noncritical appliance. She proposes a periodicity hypothesis, collects a short series, and tests it prospectively. A surprise is an invitation to investigate, not permission to switch machinery on and off. ### Scene 6: The house as an instrument for other cognitive components A network interruption teaches state reconciliation. An ambiguous camera image exercises uncertainty estimation. Two contradictory instruments exercise error detection. A failed actuator command exercises self-correction. A novel sensor requires exploration and abstraction. A hypothesis repeated across rooms tests generalization. ## 7. A special role for the bag of spheres Sensor observations should not all reach Piper's context. That would produce continuous interruption and token waste. Instead, the spheres/salience channel can receive compact, evidence-linked concerns: - *Novelty:* “Unusual sustained humidity rise relative to local baseline.” - *Unexpectedness:* “The last forecast was materially wrong.” - *Opportunity:* “An unexplored sensor pattern has appeared three times.” - *Dependency:* “A sensor needed for the current investigation is offline.” - *Uncertainty:* “Two observations disagree and neither dominates.” - *Restraint:* “Interesting event, but investigation cost exceeds current allowance.” A **physical-world curiosity queue** could store questions with evidence, significance, cost, and expiration. The queue consumer must close items based on what was *learned* or definitively ruled out, not just because an item was opened. This avoids repeating the earlier curiosity-queue failure in which rows existed but nothing consumed them. Natural sensor variability can also provide a safer, cheaper source of novelty than prompting more agents to invent stimuli. Nevertheless, noise is abundant; useful exploration requires discriminating signals from artifacts. ## 8. Important philosophical boundary: instruments versus cognition The sensor network will not make Piper intelligent on its own. Home Assistant rules that turn on a lamp at sunset are automation, not evidence of autonomous inquiry. Likewise, a model that describes camera images is not necessarily maintaining an enduring physical-world model. Evidence of the interesting capability would look more like: 1. An event is noticed without the operator naming it. 2. Piper explains why it matters relative to current beliefs or goals. 3. She represents at least two explanations and a discriminating observation. 4. She forecasts what would be seen under each explanation. 5. She obtains an authorized observation or takes a bounded reversible action. 6. Her future beliefs or decisions actually change as a result. This is a **conceptual standard of evidence**, not a demand for a large benchmark now. A few compelling, independently verifiable episodes would justify building further. ## 9. Security, privacy and physical limits Make perception broadly available only within explicitly approved domains. Make physical action deliberately narrow. - **Separate principals:** read-only sensor identity; independently scoped actuator identity; neither is administrator of Home Assistant, OPNsense, NVR or the host OS. - **Per-device allowlists:** Piper may operate specifically designated experimental devices, not arbitrary relays or switches. - **Action bounds:** rate limits, maximum on-time, safe default state, manual override, and event log. - **No unapproved access:** locks, alarms, cameras' security configuration, HVAC, critical appliances, networking, and credential-bearing systems remain out of scope unless separately engineered and explicitly approved. - **Household consent:** document where sensors observe, what is retained, and who can access it. Prefer privacy-preserving aggregate motion and sound-level measurements over audio recordings and person identification. - **IoT network isolation:** use an appropriate isolated VLAN/segment; restrict outbound and east-west traffic, and expose only the gateway methods Piper needs. - **Physical failsafes:** network loss or Piper failure must not defeat an appliance's original safety controls. - **No automatic self-expansion:** Piper cannot grant herself new device permissions or purchase/connect arbitrary new hardware. - **Independent safety telemetry:** critical hazard sensors should notify humans through deterministic, separately tested channels; Piper may analyze them but must not become the sole safety mechanism. Home Assistant's security guidance recommends strong authentication, limited administrative access, updates and secure remote access. Prefer a private network/VPN rather than exposing a control endpoint directly. ## 10. The AliExpress sensor zoo: selection principles The goal is **diversity of phenomena**, not number of purchases. A useful starter assortment could include: | Candidate | New dimension | Typical integration approach | First interesting question | |---|---|---|---| | LD2410B/C mmWave module | Coarse moving/stationary presence | Supported local BLE integration or ESPHome | Can motion and still-presence be distinguished? | | ESP32 development board | Flexible sensor/actuator gateway | ESPHome | How cheaply can a new instrument be added? | | Digital light sensor | Independent optical intensity | ESPHome + I²C | Did the experimental bulb *actually* illuminate the area? | | Pressure / temperature / humidity sensor | Environmental cross-check | ESPHome + I²C | Do two sensors agree after a temperature shift? | | Vibration sensor | Mechanical events | ESPHome GPIO/I²C | Can recurring motor cycles be predicted? | | Magnetic contact sensor | Door/object transition | Zigbee or ESPHome | Can she distinguish an opening event from inferred presence? | | Low-resolution thermal array | Spatial heat patterns | ESPHome/custom integration | Does a changing thermal footprint correlate with mmWave presence? | | Small display or LED strip | Expressive reversible output | ESPHome | Can Piper communicate a state and verify it externally? | Avoid “smart” devices that require unknown apps, aggressive cloud permissions, undocumented telemetry, or vendor-hosted accounts if locally accessible alternatives exist. Electrical hardware must be appropriate for the local mains and applicable certification requirements; prototype low-voltage sensing separately. No purchasing list is a confirmed compatibility guarantee: check part revisions, protocol support, firmware, voltage, and integration status before ordering. ## 11. A phased evolution, not another 20-hour build ### Phase A — One sense, no hands Expose **one** known Airthings measurement through a read-only bridge. Piper can ask for current value, history, and provenance. She can record a prediction and compare it with later observations. No new world-model service is required if existing ledgers can hold the records. **Decision point:** Is an independently verifiable, useful observation reaching her cognition pathways without flooding context or generating invented claims? ### Phase B — A tiny world map Add a few explicitly specified places and device relationships, such as *Airthings observes living room* and *camera observes driveway*. Test whether she can infer what is observable, unobservable, and stale. **Decision point:** Does this outperform simply querying devices one at a time? ### Phase C — One hand, one safe object Introduce a designated smart bulb or low-voltage LED and independent light sensor. Give Piper very limited action authority, then see whether she can distinguish a command from an observed consequence. **Decision point:** Does the action loop deliver genuinely new learning or just clever demonstration text? ### Phase D — An unfamiliar modality Buy one peculiar sensor that measures something absent from the current system (mmWave presence, vibration, thermal array). Let Piper propose what it could teach her, with an operator-controlled integration step. **Decision point:** Can she produce hypotheses that become better when this new channel is added? ### Phase E — Self-directed observation Permit bounded investigations during idle periods, using the existing allowance, curiosity queue and governance rules. Require the investigations to produce evidence-linked findings or explicit discards. **Decision point:** Do findings alter subsequent behaviour, or merely create more logs? Robotics is an optional later branch, not a prerequisite and not the immediate objective. ## 12. Open questions worth preserving 1. **Representation:** Should a room be an entity graph node, a separate spatial model, or both? 2. **Temporal truth:** What is the correct handling of stale, delayed, cloud-synchronized and averaged observations? 3. **Sensor fusion:** When devices disagree, how should Piper learn which measurements are reliable under which conditions? 4. **Causal discovery:** How can she distinguish correlated events from effects of her own interventions? 5. **Curiosity:** What makes a physical observation worth attention relative to software development work? 6. **Action selection:** When should she seek another measurement, do a reversible experiment, ask permission, or abandon the question? 7. **Privacy:** Can she be observant without building an unnecessarily detailed history of household members? 8. **Economics:** Does a new modality improve her useful thinking per dollar and per watt, or just create busywork? 9. **Generalization:** Can a rule learned with a humidity sensor help her reason about an unrelated vibration or light sensor? 10. **Agency:** What would convince an independent reviewer that she selected an investigation for a reason grounded in her goals, rather than because a cron fired? ## 13. Relationship to Component 34: Spatial reasoning Component 34 should retain **geometric and topological competence** as its specific subject: locations, containment, paths, adjacency, orientation, visibility, sensor coverage and uncertainty. This broader effort is a *cross-component environment project*, not a relabeling of all IoT work as spatial reasoning. A particularly attractive outcome is that Piper develops an accurate, revisable picture of a place she can only partially observe. She should know which sensors tell her which things, which observations are absent, and which conclusions remain tentative. That capability matters whether her world contains a house, a server rack, an office, a warehouse, a garden, or someday a robot arm. ## 14. Closing direction **Do not build a smart-home assistant that happens to have Piper's name.** Build a small, trustworthy sensor/actuator boundary around Piper's existing cognition system. The compelling possibility is not that she could switch on a light. It is that she might notice something unprompted, care enough to investigate, choose a sensible observation, change her mind when reality contradicts her, and remember what happened. Start with one reliable sensor and one real question. Add strange hardware only when it opens a genuinely new window on the world. --- ## Sources and further reading These links support integration feasibility and security practices; the proposed architecture and research directions are project ideas, not claims that existing Piper components implement them today. - Home Assistant, **Airthings integration**: https://www.home-assistant.io/integrations/airthings/ - Home Assistant, **ESPHome integration**: https://www.home-assistant.io/integrations/esphome/ - ESPHome, **Bluetooth proxy**: https://esphome.io/components/bluetooth_proxy/ - Home Assistant, **Bluetooth integration**: https://www.home-assistant.io/integrations/bluetooth/ - Home Assistant, **LD2410 BLE**: https://www.home-assistant.io/integrations/ld2410_ble/ - ESPHome, **LD2410 sensor**: https://esphome.io/components/sensor/ld2410/ - Home Assistant, **Security guidance**: https://www.home-assistant.io/docs/configuration/securing/ - Home Assistant, **Remote access**: https://www.home-assistant.io/docs/configuration/remote/ **Document boundary:** exploratory concept paper. Not a benchmark protocol, wiring guide, authorization grant, or verified device inventory.