# shadowaitools

> Shadow AI detection from the logs you already have: parse a DNS, proxy or firewall export locally, look up each unique domain against the AI Tools Blocklist, and get an inventory of the AI tools in use with category, AI type and vendor training verdicts.

Latest version **1.0.1** (published 2026-09-23) · MIT license · 0 weekly downloads

## Install

```sh
npm install shadowaitools
pnpm add shadowaitools
yarn add shadowaitools
bun add shadowaitools
```

Provides the command `shadowaitools`.

## Health

**Score 65/100 (B)** — status: active.

Positive: has types; no vulnerabilities; recently updated; high maintenance score; high quality score.

Warnings: low downloads; no esm support.

## Facts

| | |
|---|---|
| Version | 1.0.1 |
| Published | 2026-09-23 |
| First published | 2026-09-20 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | CommonJS |
| Node | >=14 |
| Dependencies | 0 |
| Unpacked size | 50.3 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Alpha Quantum |
| Maintainers | websitecategorization |
| Keywords | shadow ai, shadow ai detection, shadow ai tools, shadow ai discovery, ai tools inventory, dns log analysis, proxy log analysis, firewall log, ai governance, ai acceptable use, data loss prevention, ai blocklist, generative ai usage, ai risk management |

## Links

- npm: https://www.npmjs.com/package/shadowaitools
- Repository: https://github.com/explainableaixai/shadowaitools
- Homepage: https://www.shadowaitools.com
- Issues: https://www.shadowaitools.com/contact.php
- npm.io page: https://npm.io/package/shadowaitools

## Alternatives

- [async-exit-hook](https://npm.io/package/async-exit-hook.md) — 3.7M weekly downloads
- [evnty](https://npm.io/package/evnty.md) — 7.2K weekly downloads
- [eleventy-plugin-asciidoc](https://npm.io/package/eleventy-plugin-asciidoc.md) — 3.5K weekly downloads
- [@jswork/next-get2get](https://npm.io/package/@jswork/next-get2get.md) — 945 weekly downloads
- [@dashersw/axon](https://npm.io/package/@dashersw/axon.md) — 934 weekly downloads

## Recent versions

- 1.0.1 (latest) — 2026-09-23
- 1.0.0 — 2026-09-20

## README

# shadowaitools

Shadow AI detection from the logs you already have. `shadowaitools` reads a DNS, proxy or firewall export on your own machine, reduces every line to a registrable domain, looks each unique domain up once against the [unauthorized AI tool detection](https://www.aitoolsblocklist.com), and returns an inventory of the AI tools in use on your network: the tool, its functional category and subcategories, whether it is an AI-native service or an ordinary product with AI features bolted on, how often it was reached, who reached it, and what the vendor's terms say about training on your data.

The raw log never leaves the machine. Only unique domains are sent, one lookup per domain, and a JSON cache lets a nightly run re-use yesterday's answers. The hosted [shadow AI detection](https://www.shadowaitools.com) service at shadowaitools.com runs the same inventory in the browser and adds the per-user breakdown, dated vendor training verdicts, the sanctioned versus unsanctioned split and a PDF evidence pack you can hand to management.

Zero runtime dependencies. Node.js 14 and newer. TypeScript definitions ship with the package.

---

## Installation

```bash
npm install shadowaitools
```

Or run the command line tool without installing:

```bash
npx shadowaitools scan umbrella-export.csv --key YOUR_API_KEY
```

The API key is an AI Tools Blocklist key from the account area at aitoolsblocklist.com. It can be passed with `--key`, or set once as `SHADOWAITOOLS_API_KEY` (the `ATB_API_KEY` name is also read).

## Quick start

Point it at any export and read the table:

```bash
export SHADOWAITOOLS_API_KEY=your_key
shadowaitools scan dns-export.csv --csv inventory.csv --json inventory.json
```

```
Shadow AI inventory (csv export, 400 lines, 45 unique domains, 45 lookups)
AI tools found: 34   users involved: 10   train on your data by default: 16   terms silent: 15

domain                      hits   ai type     category                    trains on data   sanctioned  users
--------------------------  -----  ----------  --------------------------  ---------------  ----------  ----------------------
grammarly.com               12     ai_native   Productivity & Collab       opt_out_default  no          laptop-marketing-01 +3
character.ai                10     ai_native   Text & Language             yes              no          laptop-marketing-02 +2
openai.com                  10     ai_native   Code & Development          opt_out_default  yes         laptop-eng-07 +4
otter.ai                    7      ai_native   Audio, Voice & Music        yes              no          laptop-sales-09 +2
suno.com                    7      ai_native   Audio, Voice & Music        yes              no          desktop-support-03 +2
deepfakemaker.io            6      ai_native   Video                       unstated         no          laptop-marketing-02 +1
runwayml.com                6      ai_native   Image & Visual              yes              no          desktop-support-03 +1
```

From code:

```js
const { scan, toCsv } = require('shadowaitools');

(async () => {
  const inventory = await scan('zscaler-web.csv', {
    apiKey: process.env.SHADOWAITOOLS_API_KEY,
    sanctioned: ['openai.com', 'github.com'],
    cacheFile: './lookups.json',
  });

  console.log(inventory.summary);
  // { lines: 400, records: 400, unique_domains: 45, lookups: 45, ai_tools_found: 34,
  //   sanctioned: 2, unsanctioned: 32, users_involved: 10, training_default_yes: 16,
  //   training_no: 3, unstated: 15, quota_remaining: 9999940 }

  for (const tool of inventory.tools) {
    if (!tool.sanctioned && tool.trains_on_data !== 'no') {
      console.log(`${tool.domain}: ${tool.hits} hits, ${tool.users.length} users, training: ${tool.trains_on_data}`);
    }
  }
  require('fs').writeFileSync('inventory.csv', toCsv(inventory));
})();
```

## What the export can look like

The parser detects the format from the first line. It handles the exports that DNS filters, secure web gateways and firewalls produce today, and it falls back to pulling the first hostname off each line when nothing else matches.

| Format | Detected by | Hostname source | User or client source |
|---|---|---|---|
| `csv` | a header row with a recognised column name | column named `domain`, `query`, `QueryName`, `hostname`, `host`, `url`, `dest`, `destination`, `fqdn`, `site`, `sni`, `question` | column named `Identities`, `user`, `Source User`, `client`, `client_ip`, `device_name`, `DeviceName`, `src`, `cip`, `email` and similar |
| `key-value` | `hostname=` / `dstname=` / `url=` tokens | `hostname`, `domain`, `url`, `dstname`, `dsthost`, `query`, `fqdn`, `sni` | `user`, `usr`, `srcname`, `devname`, `srcip`, `src`, `client` |
| `dnsmasq` | `query[A]` lines | the queried name | the `from` address |
| `squid` | native `access.log` layout | the `CONNECT host:443` target | the client IP, or the authenticated user when present |
| `windows-dns` | Windows DNS Server debug log packets | the encoded question name `(3)www(6)openai(3)com(0)` | the client address |
| `generic` | anything else | first hostname on the line | first private IPv4 address on the line |

Delimiters can be comma, tab, semicolon or pipe. Column names are matched case-insensitively, so a Cisco Umbrella export with `Identities` and `Domain`, a Cloudflare Gateway export with `QueryName` and `DeviceName`, a Palo Alto URL log with `Source User` and `URL`, or a Zscaler web log with `user` and `url` all work without configuration. When the hostname column holds a full URL, the host is extracted from it.

Hostnames in reserved zones (`.local`, `.internal`, `.lan`, `.arpa`, `.corp`, `.home`, `.test`, `.example`) are skipped before any lookup, so internal names never leave the machine.

## Technical overview

### One lookup endpoint

Every unique registrable domain in the export is checked once against:

```
GET https://www.aitoolsblocklist.com/api/check?domain=<domain>
X-API-Key: <key>
```

The response for a known AI tool:

```json
{
  "domain": "chatgpt.com",
  "blocked": true,
  "primary_category": "Marketing, Sales & SEO",
  "ai_type": "ai_native",
  "categories": [
    { "category": "Text & Language", "subcategory": "General assistants & chatbots" }
  ],
  "trains_on_data": "opt_out_default",
  "opt_out_available": "yes",
  "enterprise_no_training": "yes",
  "api_no_training": "yes",
  "terms_checked": "2026-09-17",
  "quota_remaining": 9999986
}
```

And for a domain that is not an AI tool:

```json
{ "domain": "example.com", "blocked": false, "categories": [], "quota_remaining": 9999985 }
```

`chat.openai.com`, `api.openai.com` and `platform.openai.com` collapse to `openai.com` before the request, so an export with 40,000 lines and 900 distinct hostnames usually needs a few hundred lookups.

### Fields on every tool

| Field | Meaning |
|---|---|
| `domain` | the registrable domain the tool lives on |
| `hosts` | every hostname under it seen in the export |
| `hits` | number of log lines that reached it |
| `users` | `[{ name, hits }]` per user, device or client IP, when the export carries one |
| `sanctioned` | true when the domain is in your `sanctioned` list |
| `primary_category` | one of 18 functional categories, for example Text & Language, Code & Development, Image & Visual, Audio, Voice & Music, Video, Agents & Automation |
| `categories` | every category and subcategory the tool belongs to |
| `ai_type` | `ai_native` for a service whose product is AI, `ai_enabled` for an ordinary product with AI features |
| `trains_on_data` | `yes`, `no`, `opt_out_default` (trains unless you opt out) or `unstated` |
| `opt_out_available` | whether the vendor offers a way out of training |
| `enterprise_no_training` | whether the enterprise tier is excluded from training |
| `api_no_training` | whether API traffic is excluded from training |
| `terms_checked` | the date the vendor terms were last read |

### Summary counters

`summary.lines` and `summary.records` show how much of the file was usable. `summary.unique_domains` is the number of lookups a first run needs, `summary.lookups` the number actually made after the cache. `training_default_yes` counts tools whose verdict is `yes` or `opt_out_default`, which is the set that trains on staff input unless somebody has changed a setting. `unstated` counts vendors whose terms say nothing, which is the larger group across the AI tool population: 85.5% of tools are silent on training according to the Policy Silence Index on aitoolsblocklist.com.

### Methods

| Method | Returns | Purpose |
|---|---|---|
| `parseLog(pathOrText, { maxLines })` | `{ format, lines, records }` | Parse without any network call |
| `extractDomains(records)` | `[{ domain, hosts, hits, users }]` | Group by registrable domain, sorted by hits |
| `registrableDomain(host)` | `string` | `docs.anthropic.com` to `anthropic.com`, `x.co.uk` keeps three labels |
| `scan(pathOrText, options)` | `Inventory` | Parse, look up, assemble |
| `toCsv(inventory)` | `string` | One row per AI tool |
| `toTable(inventory)` | `string` | Fixed-width table for terminals and tickets |
| `new Client(apiKey, options).lookup(domain)` | API object | Direct access to the lookup endpoint |

`scan` options: `apiKey`, `concurrency` (default 1), `pauseMs` between lookups, `maxLines`, `cacheFile`, `sanctioned` (array of domains), `onProgress(done, total)`, and the client options `baseUrl`, `timeout`, `maxRetries`.

### Errors

| HTTP | Error class | When |
|---|---|---|
| 401 | `AuthenticationError` | no key, or a key that matches no account |
| 403 | `QuotaError` | account not active, or the monthly lookup quota is used up |
| 429 | `RateLimitError` | too many requests; the client retries twice with a pause before throwing |
| 503 | `ShadowAIToolsError` | lookup service busy; retried the same way |

Every error carries `status` and the parsed response `body`.

## Worked examples

### Nightly cron with a lookup cache

A cache file turns a daily run into a cheap delta: only domains that were not seen yesterday cost a lookup.

```js
// nightly.js: run from cron after the DNS filter export lands
const { scan, toCsv } = require('shadowaitools');
const fs = require('fs');

(async () => {
  const date = new Date().toISOString().slice(0, 10);
  const inventory = await scan(`/var/exports/dns-${date}.csv`, {
    apiKey: process.env.SHADOWAITOOLS_API_KEY,
    cacheFile: '/var/lib/shadowaitools/lookups.json',
    sanctioned: fs.readFileSync('/etc/shadowaitools/sanctioned.txt', 'utf8').split('\n'),
    pauseMs: 50,
  });
  fs.writeFileSync(`/var/reports/shadow-ai-${date}.csv`, toCsv(inventory));

  const newUnsanctioned = inventory.tools.filter((t) => !t.sanctioned && t.ai_type === 'ai_native');
  if (newUnsanctioned.length) {
    console.log(`${newUnsanctioned.length} unsanctioned AI-native tools reached today:`);
    newUnsanctioned.forEach((t) => console.log(` ${t.domain} (${t.hits} hits, ${t.users.length} users)`));
    process.exitCode = 2; // let the scheduler flag the run
  }
})();
```

### Feeding a SIEM or ticketing system

The inventory is plain JSON, so the tool list drops straight into a webhook, a SIEM ingest endpoint or a ticket body.

```js
const { scan } = require('shadowaitools');
const https = require('https');

(async () => {
  const inv = await scan('proxy-week-37.log', { apiKey: process.env.SHADOWAITOOLS_API_KEY });
  const events = inv.tools.map((t) => ({
    type: 'shadow_ai_tool',
    domain: t.domain,
    category: t.primary_category,
    ai_type: t.ai_type,
    hits: t.hits,
    users: t.users.map((u) => u.name),
    trains_on_data: t.trains_on_data,
    terms_checked: t.terms_checked,
    severity: t.trains_on_data === 'yes' || t.trains_on_data === 'opt_out_default' ? 'high' : 'medium',
  }));
  const body = JSON.stringify({ source: 'shadowaitools', generated: inv.generated, events });
  const req = https.request(process.env.SIEM_WEBHOOK, { method: 'POST', headers: { 'Content-Type': 'application/json' } });
  req.end(body);
})();
```

### Parsing only, no lookups

Useful for checking that an export is readable before spending a single lookup, or for counting how many lookups a first run will need.

```bash
shadowaitools domains firewall.log | head
# key-value export, 120000 lines, 812 unique domains
#    9123  microsoft.com
#    4410  google.com
#    2201  openai.com
```

```js
const { parseLog, extractDomains } = require('shadowaitools');
const parsed = parseLog('/tmp/pihole.log');
const groups = extractDomains(parsed.records);
console.log(parsed.format, parsed.lines, groups.length);
```

### Sanctioned split against an approved list

```js
const approved = ['openai.com', 'github.com', 'grammarly.com', 'deepl.com'];
const inv = await scan('gateway.csv', { apiKey, sanctioned: approved });
console.log(`sanctioned ${inv.summary.sanctioned}, unsanctioned ${inv.summary.unsanctioned}`);
```

## Why start from the logs

Shadow AI is a visibility problem before it is a policy problem. Staff adopt chatbots, code assistants, transcription services and file converters months before procurement hears about them, and a survey never catches the tools people would rather not mention. The DNS filter, proxy or firewall already records every hostname the network reached, so the honest inventory is sitting in an export nobody has read.

The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) asks organisations to map the AI systems in use before they can measure or manage the risk, and mapping starts with a list. [Shadow IT](https://en.wikipedia.org/wiki/Shadow_IT) has been a governance topic for two decades; what changed with AI tools is that the thing leaving the network is the content itself, pasted into a prompt. The [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) lists sensitive information disclosure among its leading risks, and the first control is knowing which services receive that information. Because the raw material is a DNS log, the format notes in [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035) still describe what a resolver writes down.

A domain-level inventory is the cheapest reliable evidence: no agent on endpoints, no TLS inspection, no change to the network. It also answers the question that follows immediately, which is whether the vendor trains on what was pasted. That is why every tool row carries the training verdict and the date it was checked.

## Where the hosted audit goes further

`shadowaitools` gives you the inventory. The hosted [shadow AI audit from a DNS export](https://www.shadowaitools.com) at shadowaitools.com takes the same file and adds what a report for management or a client needs: per-user, per-device or per-IP breakdown, dated vendor training verdicts on every tool, the sanctioned versus unsanctioned split against your approved list, sector policy verdicts (Block, Controls or Allow) from the AI Policy Profiles, a CSV export and a PDF evidence pack. The free preview names a fifth of the tools found; full reports are one-time purchases without a subscription, and the subscription plans on aitoolsblocklist.com include one to ten audits a month.

## Related packages and data from Alpha Quantum

- [`aiblocklist`](https://www.npmjs.com/package/aiblocklist) and [`aitoolsblocklist`](https://www.npmjs.com/package/aitoolsblocklist): Node.js clients for the AI Tools Blocklist lookup and feed APIs, the database this tool resolves against: 20,000+ [categorized AI-tool domains](https://www.aitoolsblocklist.com) in 18 functional categories, refreshed daily, shipped as EDL, PAC, hosts and DNS feeds.
- [`aiagentallowlist`](https://www.npmjs.com/package/aiagentallowlist): the [AI agent allow list](https://www.aiagentallowlist.com) covers the other direction of AI governance, what your own browsing agents may open, with verified page-type URLs across 40 million+ domains and up to 28 page types each.
- [`webfilteringdatabase`](https://www.npmjs.com/package/webfilteringdatabase) and [`websitecategorization`](https://www.npmjs.com/package/websitecategorization): the [web filtering database](https://www.webfilteringdatabase.com) (120M+ domains, 59 categories) and the [website categorization API](https://www.websitecategorizationapi.com) (700+ IAB content categories) behind the same classification infrastructure.
- [`phishingdetectionapi`](https://www.npmjs.com/package/phishingdetectionapi): the [phishing detection API](https://www.phishingdetectionapi.com), 390,000+ DNS-verified active phishing domains for the same resolvers and gateways.
- [`cipawebfiltering`](https://www.npmjs.com/package/cipawebfiltering): the [CIPA web filtering](https://www.cipawebfiltering.com) client for schools and libraries.
- Python: [`shadowaitools` on PyPI](https://pypi.org/project/shadowaitools/), same parser, same inventory, same cache format.
- Source: [github.com/explainableaixai/shadowaitools](https://github.com/explainableaixai/shadowaitools), mirrored at [gitlab.com/url-classifications/shadowaitools](https://gitlab.com/url-classifications/shadowaitools).

## Frequently asked questions

**What is shadow AI?**
Shadow AI is the use of AI tools inside an organisation without the knowledge or approval of IT, security or compliance: chatbots, code assistants, image and voice generators, meeting transcribers and document tools that staff sign up for on their own. It is the AI-era form of shadow IT, with the added problem that the data pasted into a prompt may be used to train the vendor's models.

**How do I find out which AI tools my employees are using?**
Export a week of logs from the DNS filter, proxy or firewall you already run and scan it with `shadowaitools scan export.csv`. Every hostname is matched against a database of 20,000+ AI tool domains, so the result reflects real traffic from your own network rather than survey answers. The hosted audit at [shadowaitools.com](https://www.shadowaitools.com) does the same in the browser and produces a PDF.

**Does the log file leave my machine?**
No. The file is parsed locally. Only unique registrable domains are sent to the lookup API, one request per domain, and hostnames in reserved zones such as `.local` or `.internal` are never sent at all. The hosted audit reads an upload once and discards it.

**Which log formats does shadowaitools accept?**
Exports from Cisco Umbrella, Zscaler, Palo Alto, Fortinet, Cloudflare Gateway, DNSFilter, NextDNS, Pi-hole, SonicWall, Squid and Windows DNS Server debug logs are detected automatically, as is any CSV with a header row, key=value syslog lines, and a plain list with one hostname or URL per line.

**How many lookups does a scan use?**
One per unique registrable domain, after subdomains are collapsed. A 400-line export with 45 distinct domains uses 45 lookups; a second run with `--cache` uses none for domains already seen. The `domains` command shows the count before any lookup is made.

**What does `trains_on_data: opt_out_default` mean?**
The vendor trains on customer input unless the customer changes a setting or files a request. It is grouped with `yes` in `summary.training_default_yes`, because on a network where nobody has opted out it behaves like `yes`. `unstated` means the terms say nothing either way as of `terms_checked`.

**Can I mark the tools we have approved?**
Yes. Pass `sanctioned: ['openai.com', 'github.com']` or `--sanctioned openai.com,github.com`; every tool gets a `sanctioned` flag and the summary splits the counts. The hosted audit keeps an approved list per account.

**How is this different from the AI category in a web filter?**
A filter category answers whether a domain is AI. This inventory answers which tool it is, what kind, who used it, how often, and whether the vendor trains on the input, with a dated verdict. The database behind it is rebuilt daily so new tools are caught close to launch.

**Who builds shadowaitools?**
Alpha Quantum, the company behind the [list of AI websites to block](https://www.aitoolsblocklist.com), the [AI agent allow list](https://www.aiagentallowlist.com), the [website categorization API](https://www.websitecategorizationapi.com) and the [web filtering database](https://www.webfilteringdatabase.com).

## Links

- Hosted shadow AI audit: [https://www.shadowaitools.com](https://www.shadowaitools.com)
- AI tools blocklist, the lookup database: [https://www.aitoolsblocklist.com](https://www.aitoolsblocklist.com)
- AI agent allow list: [https://www.aiagentallowlist.com](https://www.aiagentallowlist.com)
- NIST AI Risk Management Framework: [https://www.nist.gov/itl/ai-risk-management-framework](https://www.nist.gov/itl/ai-risk-management-framework)
- OWASP Top 10 for LLM Applications: [https://owasp.org/www-project-top-10-for-large-language-model-applications/](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- Shadow IT: [https://en.wikipedia.org/wiki/Shadow_IT](https://en.wikipedia.org/wiki/Shadow_IT)
- RFC 1035, Domain Names: [https://www.rfc-editor.org/rfc/rfc1035](https://www.rfc-editor.org/rfc/rfc1035)

## License

MIT

---
_Source: https://npm.io/package/shadowaitools · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._