Documentation > OpenStack Cloud > V&V > Integration Test Plan

Documentation Drift Detection Test Procedures

Reference: Requirement VERIF-04, SC-018, CF-SK-022, CF-CH-012
Status: Specification
NASA SE Phase: Phase D (Integration and Test)

---

Purpose

This document defines test procedures for verifying that the doc-verifier skill correctly detects documentation drift when drift is intentionally introduced. These are "verify the verifier" tests — they confirm that the drift detection mechanism works before it is trusted to protect the accuracy of the operations documentation.

Documentation drift is the gap between what documentation says and what the running system does. The doc-verifier skill detects four categories of drift using NASA's TAID verification methods (SP-6105 SS 5.3).

---

Drift Categories

Category	Definition	Primary Detection Method
Configuration drift	Documented setting X=A, running system has X=B	Analysis — read config files, compare to docs
Endpoint drift	Documented URL/port differs from actual endpoint	Test — probe actual endpoints, compare to docs
Procedure drift	Documented steps no longer produce documented results	Test — execute commands, compare output
Reference drift	External documentation link no longer valid	Inspection — resolve links, verify targets

---

Test Scenarios

DRIFT-CONFIG-001: Keystone Token Lifetime Mismatch

Category: Configuration drift
Detection method: Analysis

Setup: Documentation states token_expiration = 3600. Running system has token_expiration = 3600. Match.

Intentional drift: Modify documentation to state token_expiration = 7200 while system remains at 3600.

Expected detection:

[CRITICAL] OPS-KEYSTONE-002: Configuration drift detected
  Setting: keystone.conf [fernet_tokens] token_expiration
  Documented: 7200
  Actual:     3600

DRIFT-CONFIG-002: Nova Compute Driver Mismatch

Category: Configuration drift
Detection method: Analysis

Setup: Documentation states compute_driver = libvirt.LibvirtDriver. System matches.

Intentional drift: Modify documentation to state compute_driver = fake.FakeDriver.

Expected detection:

[CRITICAL] OPS-NOVA-001: Configuration drift detected
  Setting: nova.conf [DEFAULT] compute_driver
  Documented: fake.FakeDriver
  Actual:     libvirt.LibvirtDriver

DRIFT-ENDPOINT-001: Service Catalog Endpoint URL Change

Category: Endpoint drift
Detection method: Test

Setup: Documentation references http://10.0.0.10:5000/v3. System matches.

Intentional drift: Modify documentation to reference port 5001.

Expected detection:

[CRITICAL] OPS-KEYSTONE-001: Endpoint drift detected
  Documented endpoint: http://10.0.0.10:5001/v3
  Service catalog (actual): http://10.0.0.10:5000/v3
  HTTP probe of documented endpoint: Connection refused

DRIFT-ENDPOINT-002: Horizon Dashboard Port Mismatch

Category: Endpoint drift
Detection method: Test

Intentional drift: Documentation references port 8080 instead of actual port 80.

Expected detection:

[CRITICAL] OPS-HORIZON-001: Endpoint drift detected
  Documented port: 8080
  Actual port: 80

DRIFT-PROCEDURE-001: Fernet Key Rotation Procedure Produces Different Result

Category: Procedure drift
Detection method: Test

Intentional drift: Modify documented expected output to reference a deprecated message format.

Expected detection:

[WARNING] RB-KEYSTONE-002 Step 3: Procedure drift detected
  Command: openstack credential migrate
  Documented expected output: "Rotating Fernet keys... (deprecated v2 format)"
  Actual output: "Migrating credentials to a new key..."

DRIFT-PROCEDURE-002: Nova Instance Creation with Wrong Flavor Name

Category: Procedure drift
Detection method: Test

Intentional drift: Documentation references non-existent flavor m1.tiny-deprecated.

Expected detection:

[CRITICAL] RB-NOVA-003 Step 1: Procedure drift detected
  Command references flavor: m1.tiny-deprecated
  Available flavors: m1.nano, m1.small, m1.medium, m1.large, m1.xlarge

DRIFT-REFERENCE-001: OpenStack Docs URL Version Mismatch

Category: Reference drift
Detection method: Inspection

Intentional drift: Reference points to Queens release documentation instead of the deployed 2024.1.

Expected detection:

[WARNING] docs/sysadmin-guide/references.md: Reference drift detected
  Deployed version: 2024.1
  URL resolves: YES (archive exists for Queens)
  Version match: NO (Queens != 2024.1)

DRIFT-REFERENCE-002: SP-6105 Section Number Cross-Reference Error

Category: Reference drift
Detection method: Inspection

Intentional drift: Citation changed from SS 5.3 (Product Verification) to SS 5.4 (Product Validation).

Expected detection:

[CRITICAL] docs/vv/safety-critical-tests.md: Reference drift detected
  Cited: SP-6105 SS 5.4 (described as "Product Verification")
  Actual content of SS 5.4: Product Validation (not Verification)
  Product Verification is SP-6105 SS 5.3

---

Detection Verification Procedure

1. Establish Baseline: Run doc-verifier against unmodified document. Expected: 0 drift items.
2. Introduce Intentional Drift: Apply exactly the change specified in the scenario.
3. Run Detection: doc-verifier verify [target-document]
4. Validate Detection Report: Confirm artifact path, expected value, actual value, severity, and location.
5. Restore to Baseline: git checkout [target-document]. Re-verify: 0 drift items.

---

Integration with Doc-Sync Communication Loop

Automatic Trigger Path

SURGEON detects system change → sends drift alert to doc-sync loop → doc-verifier runs targeted verification → findings reported to FLIGHT → EXEC updates documentation → re-verified before doc-verified flag re-set.

Drift Resolution Paths

Drift type	Automatic resolution	Manual resolution
Documentation ahead of system	EXEC applies config change	Operator evaluates whether change is intended
System ahead of documentation	EXEC-docs updates documentation	Operator verifies change was intentional
Reference link invalid	EXEC-docs updates link	Operator verifies replacement source quality
Version mismatch	EXEC-docs updates version reference	Operator confirms deployed version