# Integration Pattern: Federated Device Inventory Ingestion — Custom API-to-CMDB Pipeline ## Pattern Name **Federated Device Inventory Ingestion — Custom API-to-CMDB Pipeline** _Also known as: Multi-Source CMDB Ingestion via Scripted API Integration with IRE Reconciliation_ --- ## Intent Ingest device inventory and configuration data from multiple on-premises device management platforms into the ServiceNow CMDB via scheduled, MID Server-relayed API extractions — transforming vendor-specific payloads into normalized CI records and reconciling attribute authority across sources using ServiceNow's Identification and Reconciliation Engine (IRE). --- ## Motivation Enterprise device management is rarely owned by a single platform. Organizations operate multiple tools — endpoint management, patch management, mobile device management, network discovery — each maintaining its own inventory with overlapping but inconsistent records. No single source is complete or fully trusted. This pattern aggregates inventory data from all authoritative sources into the CMDB, resolves duplicate CI records across sources, and enforces attribute-level trust rules so that the CMDB reflects the best available picture of the configuration estate — without any single source overwriting another's authoritative attributes. --- ## Applicability Use this pattern when: - Multiple on-premises device management platforms each hold partial or overlapping CI inventory data - Source system APIs are not publicly reachable and require an on-prem relay to bridge the network boundary - Native ServiceNow connectors (IntegrationHub spokes, Graph connectors) are unavailable or not yet mature enough for the source systems in scope - The CMDB must reconcile competing attribute values from multiple sources rather than accepting any single source as fully authoritative - Each source system should be tracked as a distinct data contributor with its own trust weight per CI attribute --- ## Participants |Participant|Role|Technology (Example Implementation)| |---|---|---| |Source Systems|On-premises device management consoles providing inventory via API|IBM BigFix, Microsoft SCCM/MECM, Jamf, Tanium| |MID Server|On-prem relay; bridges network boundary between ServiceNow cloud instance and internal APIs|ServiceNow MID Server| |ECC Queue|Internal ServiceNow message bus; brokers instructions and responses between instance and MID Server|ServiceNow ECC Queue| |Scheduled Job|Triggers extraction on a recurring schedule|ServiceNow Scheduled Script Execution| |Extraction Script|Constructs and dispatches API calls through MID Server|ServiceNow server-side script using `RESTMessageV2`| |Staging Layer|Receives raw API response payload for pre-transformation holding|ServiceNow Import Set tables| |Transformation Layer|Maps vendor-specific fields to CMDB CI class schema|ServiceNow Transform Maps (with scripted field mappings)| |Identity & Reconciliation|Matches incoming records to existing CIs; manages attribute authority across sources|ServiceNow Identification and Reconciliation Engine (IRE)| |CMDB|Authoritative configuration item repository|ServiceNow CMDB (cmdb_ci_computer, cmdb_ci_server, etc.)| --- ## Structure ``` ServiceNow Instance ┌──────────────────────────────────────────┐ │ │ │ Scheduled Script Execution │ │ (RESTMessageV2 — route via MID Server) │ │ │ │ │ │ ECC Queue (outbound) │ └─────────┼────────────────────────────────┘ │ ▼ MID Server (on-prem, inside network perimeter) │ │ API call (REST / SOAP) │ per source system schedule ▼ ┌───────────────────────────────┐ │ Device Management Consoles │ │ (BigFix, SCCM, Jamf, etc.) │ └───────────────┬───────────────┘ │ API response payload ▼ MID Server │ │ ECC Queue (inbound) ▼ ServiceNow Instance ┌──────────────────────────────────────────┐ │ │ │ Import Set Table │ │ (raw staging, per source system) │ │ │ │ │ ▼ │ │ Transform Map │ │ (field mapping + scripted transforms) │ │ │ │ │ ▼ │ │ IRE — Identification & Reconciliation │ │ (CI matching + attribute authority) │ │ │ │ │ ▼ │ │ CMDB CI Tables │ │ (cmdb_ci_computer, cmdb_ci_server, ...) │ │ │ └──────────────────────────────────────────┘ ``` --- ## Flow ### 1. Schedule Trigger - A **Scheduled Script Execution** fires on a configured recurring interval (e.g., every 4–24 hours depending on source system volatility) - Separate schedules maintained per source system to allow independent tuning of frequency and failure handling ### 2. API Extraction via MID Server - Scheduled script instantiates `RESTMessageV2`, configured to route the outbound request through a designated **MID Server** - ServiceNow writes an outbound instruction to the **ECC Queue** - MID Server polls the ECC Queue, picks up the instruction, and executes the API call against the on-premises device management console from inside the network perimeter - API response (JSON or XML) returned to MID Server, written back to the ECC Queue as an inbound message - ServiceNow instance reads the inbound ECC Queue entry and processes the response payload ### 3. Import Set Landing - Response payload written to a **source-specific Import Set table** — one table per source system to isolate schemas and simplify transform mapping - Raw vendor payload preserved at this layer; no transformation applied - Each Import Set row tagged with source system identifier and load timestamp for traceability ### 4. Transform Map Execution - **Transform Map** fires against the Import Set table, executing field-level mappings from vendor-specific attribute names to CMDB CI class fields - **Scripted transform entries** handle: - Data type coercion (string → boolean, vendor codes → normalized values) - Null / missing value handling - Vendor-specific quirks (e.g., non-standard hostname formats, platform-specific serial number patterns) - Choice list normalization (e.g., OS name variants → canonical CMDB values) ### 5. IRE — Identification - On each transformed record, the **IRE Identification Engine** attempts to match the incoming record to an existing CI using configured **identifier entries** — evaluated in priority order: - Primary: serial number, asset tag - Secondary: hostname, MAC address, IP address - Match result determines whether the operation is a **create** (new CI), **update** (existing CI), or **duplicate** (merge candidate) ### 6. IRE — Reconciliation - For update operations, **Reconciliation Rules** determine which source system's value is accepted for each CI attribute - Each integration registered as a distinct **Discovery Source** with a configured trust precedence - Higher-trust sources overwrite lower-trust values; lower-trust sources update only attributes not already populated by a higher-trust source - Prevents any single source from silently overwriting authoritative data owned by another ### 7. CMDB Population - Reconciled CI record written to the appropriate CMDB class table (`cmdb_ci_computer`, `cmdb_ci_server`, `cmdb_ci_hardware`, etc.) - CI relationships (e.g., device → location, device → owner) populated where source data supports it - Audit trail maintained via CMDB's built-in sys_audit and source tracking fields --- ## Tradeoffs ### Advantages - **Network boundary traversal** — MID Server enables ServiceNow cloud instances to reach on-premises APIs without exposing those systems to the public internet - **Multi-source reconciliation** — IRE prevents the CMDB from becoming a last-write-wins data store; attribute authority is explicit and governed - **Source isolation** — Per-source Import Set tables and Transform Maps allow each integration to be developed, tested, and failed independently - **Auditability** — Import Set records and IRE reconciliation logs provide a traceable history of what each source contributed and what was accepted - **Incremental buildout** — New source systems can be onboarded by adding a new scheduled script, Import Set table, and Transform Map without modifying existing integrations ### Limitations - **Custom development overhead** — Without IntegrationHub spokes, every source system requires bespoke scripting; there is no configuration-driven onboarding path - **Maintenance burden** — Source API changes (schema, auth, pagination behavior) require manual script updates with no vendor-managed compatibility - **ECC Queue latency** — MID Server communication is asynchronous via queue polling; not suitable for near-real-time inventory requirements - **IRE sensitivity** — Poorly configured identifier entries produce false matches (CI merges) or missed matches (duplicate CIs); identifier quality is a persistent operational challenge - **MID Server as single point of failure** — If the MID Server is unavailable, all integrations routed through it fail silently unless monitored explicitly - **Schema sprawl** — One Import Set table per source system accumulates over time; governance required to prevent abandoned integrations lingering in the instance - **No native error surfacing** — Scripted integrations require explicit error handling and alerting; failures do not surface in a centralized integration monitoring UI the way IntegrationHub spokes do --- ## Key ServiceNow Components |Component|Purpose in this Pattern| |---|---| |`RESTMessageV2`|Server-side API for constructing and dispatching HTTP requests, with MID Server routing support| |ECC Queue|Asynchronous message bus between ServiceNow instance and MID Server; input (inbound) and output (outbound) queues| |MID Server|Java-based agent deployed on-prem; executes API calls on behalf of the ServiceNow instance within the network perimeter| |Import Set|Staging table framework for receiving and holding raw inbound data prior to transformation| |Transform Map|Field mapping configuration (with optional scripted entries) that converts Import Set rows into target table records| |IRE|Identification and Reconciliation Engine; matches incoming records to existing CIs and governs attribute authority| |Discovery Source|Registered label for each contributing data source; used by IRE reconciliation rules to assign trust precedence per attribute| --- ## Related Patterns - **Batch API-to-Warehouse ELT** — Similar scheduled pull pattern targeting a data warehouse rather than an operational CMDB; no identity resolution layer - **ServiceNow Discovery** — Native alternative for network-reachable devices; uses MID Server probes/sensors rather than third-party API calls; complementary rather than competing - **IntegrationHub Spoke Pattern** — Successor pattern; replaces custom `RESTMessageV2` scripts with configuration-driven spoke actions and flow designer orchestration - **Graph Connector CMDB Ingestion** — Further evolution; vendor-managed connectors with native IRE integration, replacing custom Transform Maps --- ## Implementation Notes - Register each source system as a distinct **Discovery Source** in ServiceNow before configuring reconciliation rules — this is required for IRE to apply per-source attribute trust - Define **identifier entry priority** explicitly per CI class; the order matters — a misconfigured identifier chain is the most common cause of duplicate CIs at scale - Build **error handling into every scheduled script** — catch HTTP errors, log to a dedicated integration log table, and trigger alerting; the ECC Queue will not surface failures automatically - Monitor **MID Server heartbeat** as a health check; a MID Server that stops checking in will silently halt all integrations routed through it - Maintain a **source system registry** document (or CMDB data source CI) tracking which systems are integrated, their schedules, their Discovery Source names, and their IRE trust precedence — this becomes critical tribal knowledge without documentation - Periodically audit **Import Set table row counts and ages** to detect stalled integrations or runaway data accumulation - Consider a **full reload vs. delta strategy** per source: high-churn sources (endpoint agents reporting frequently) benefit from delta/incremental loads; slower-changing sources (network inventory) can tolerate full reloads