NSAuditor AI EE 0.16.4: a false-clean post-mortem — when an audited cloud surfaced zero of eight CRITICALs

NSAuditor AI EE 0.16.4 (with Community Edition 0.1.95 and agent-skill 0.1.63) shipped on 2026-05-30 to close a defect of the most dangerous class an audit tool can carry: a false clean. The MCP scan_cloud tool — the “audit my AWS account” capability operators drive from Claude Desktop or Claude Code — ran a full cloud audit, the auditors found real CRITICAL exposures, and the tool surfaced zero findings to the user. This is an engineering post-mortem of that bug, the root cause at the surfacing boundary, the fix, and the adversarial review that hardened it.

The incident

An operator installed the build, asked Claude Desktop to “audit my AWS account,” and got back a clean-looking report: the account was audited, roughly 20 auditor plugins ran, and 0 findings were returned. The account in question was anything but clean. It carried eight CRITICAL exposures, all of which the auditors had detected:

• two shadow-admin IAM users with full wildcard (*) permissions
• a public S3 bucket with a missing public-access block
• an internet-open security group on SSH (22), Postgres (5432), and Redis (6379)
• a public, no-auth Lambda Function URL (AuthType: NONE)
• a DynamoDB table with no point-in-time recovery and no deletion protection

The verdict said “secure” over a live, exploitable estate. That is the worst outcome an audit tool can produce.

Root cause: a summarizer at the wrong altitude

The findings were never actually lost in the scan — they were lost at the surfacing layer. handleScanCloud built its caller-visible summary from result_concluder.mjs, an internal component written for network port/service scans. That concluder reads only result.data (the host/port shape) and has no concept of cloud compliance findings, which live in result.findings. It silently dropped them. The raw scan results held all eight CRITICALs the entire time; they simply never crossed the boundary into the answer the operator saw.

This is a distinct bug class from an earlier fold (CE 0.1.93) that covered “a cloud that was not audited reported as clean.” Here the cloud was audited — the coverage accounting was correct — but a summarizer at the wrong altitude discarded the substance. It is a reminder that a correct scan and a correct report of that scan are two different guarantees.

The fix

CE 0.1.95 introduces a dedicated utils/cloud_finding_summary.mjs and rewires handleScanCloud to build a per-cloud findings summary — severity counts plus the full CRITICAL/HIGH list — directly from result.findings, then renders the report from that. The structural invariant is now explicit and tested: a cloud that has findings can never report zero while reporting audited. The fix also drops the misleading “host is up” conclusion (a network-scan artifact that has no meaning for a cloud audit) and scopes “audit my AWS account” to AWS only, rather than fanning out across all three providers.

Notably, this is entirely a Community Edition engine fix. EE 0.16.4 is a paired version bump with no Enterprise-side code change, no new plugin (count stays 28), no dependency change, and all six compliance coverage matrices unchanged.

Adversarial review and the four folds

The change went through the project’s audit-cloud-plugin-false-negatives lens — a red-team review whose single obsession is what a scanner silently misses. Verdict: SHIP-WITH-FOLDS (0 critical / 0 high / 2 medium / 2 low). The headline guarantee held, and four edge cases were fixed in the same session:

1. Resource labels. Each finding now carries its resource label (bucket, user, security-group, key, table, function). The review caught that the S3 CRITICAL was losing its bucket name — a CRITICAL you can’t name isn’t actionable.
2. Truncation ordering. Findings are sorted critical-first before any display cap, so a CRITICAL can never be evicted behind lower-severity items.
3. Clean fallbacks. A raw JSON.stringify dump for indeterminate findings was replaced with a clean descriptor.
4. No silent drops. A result whose provider can’t be resolved is bucketed under unknown and rendered, not discarded — defense-in-depth against future identifier drift.

Verification

Beyond live test-infra acceptance, the fix was confirmed against the published global build and in production: re-running “audit my AWS account” in Claude Desktop now returns all 20 plugins and surfaces 8 CRITICAL · 38 HIGH · 41 MEDIUM · 46 LOW · 25 PASS, each CRITICAL named with its resource.

One operator note from the published re-smoke: at the default CLOUD_PLUGIN_TIMEOUT_MS=25000, the heaviest S3 auditor timed out on a near-ceiling estate and only 7 CRITICALs surfaced — the surfacing layer behaved correctly (it surfaced every plugin that ran and would flag a timeout as a coverage gap, never a clean pass). Setting CLOUD_PLUGIN_TIMEOUT_MS=45000 (still under the MCP 60-second tool-call budget) restored full coverage.

The 0.16.x MCP cloud-audit arc

0.16.4 is the capstone of a same-day, CE-led arc that made conversational cloud auditing real: 0.16.1 added NSA_ENV_FILE (switch accounts by pointing the MCP server at a per-environment .env); 0.16.2 introduced the scan_cloud tool itself, with no throwaway network host; 0.16.3 made the cloud auditors run concurrently so a full ~20-plugin account audit fits Claude Desktop’s ~60-second budget; 0.16.4 ensures those findings actually reach the answer.

Positioning

NSAuditor AI EE is a local-only, zero-data-exfiltration (ZDE) cloud and network security scanner. A single scan maps findings to six compliance frameworks — SOC 2, HIPAA, NIST CSF 2.0, PCI DSS v4.0.1, ISO 27001:2022, and CIS Controls v8 — with RFC 3161 trusted timestamps, and it runs as an MCP server so an AI assistant can drive the audit conversationally. No data leaves the operator’s machine. The lesson this release encodes is durable: a false clean is the only audit defect that actively makes you less safe than running nothing at all, and the surfacing boundary deserves the same scrutiny as the scanner itself.

Learn more about NSAuditor AI Enterprise →