Metadata-Version: 2.4
Name: leaddemon
Version: 0.2.0
Summary: Run LeadDemon lead discovery, website validation, and CRM gap analysis workflows.
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4>=4.12.3
Requires-Dist: azure-storage-blob>=12.20.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: psycopg[binary]>=3.2.0
Requires-Dist: python-dotenv>=1.2.0
Requires-Dist: python-multipart>=0.0.18
Requires-Dist: rich>=14.0.0
Requires-Dist: typer>=0.16.0
Requires-Dist: uvicorn[standard]>=0.34.0
Provides-Extra: dev
Requires-Dist: coverage>=7.14.0; extra == "dev"
Requires-Dist: pytest>=8.3.0; extra == "dev"

# LeadDemon

LeadDemon is a configurable lead-discovery and CRM-gap pipeline. Users define
the company profile they want to find, select the geography to search, choose
free or approved paid sources, validate company websites, backfill public
contact data, compare against a CRM baseline, and download clean import-ready
files.

## What It Does

1. Captures a target-company description, search terms, desired service/product
   signals, and exclusion terms.
2. Builds a market universe from selected countries, states/provinces, and
   configured market-depth tiers.
3. Runs free DuckDuckGo discovery and optional paid provider searches after cost
   approval.
4. Deduplicates provider results into a prospect universe.
5. Reviews company websites for configured target evidence and public contact
   data.
6. Optionally compares prospects to HubSpot or an uploaded CRM export.
7. Packages a lean customer download set: the analysis workbook, likely
   net-new leads, import-ready companies, and the selected CRM import CSV.
   Diagnostic review and audit data remain in the workbook and internal run
   records.

## Architecture

The FastAPI product API lives under
`src/leaddemon/web_app.py` and routes in
`src/leaddemon/api/`. The pipeline engine lives in
`src/leaddemon/pipeline.py`, with provider discovery,
website extraction, scoring, CRM gap analysis, artifact storage, worker, and
observability modules split across the package.

The Python package and command-line entrypoints are LeadDemon:

```bash
leaddemon --help
leaddemon-web --help
leaddemon-worker --help
```

Coding agents use the same product API as the web app. Interactive CLI login
uses a short-lived browser authorization code; the API stores only hashed
device/session credentials. HubSpot OAuth and Stripe subscription changes stay
in the human web experience.

```mermaid
flowchart LR
    Human["Human operator"] --> Web["Next.js web app"]
    Agent --> Guide["/agents setup guide"]
    Guide --> Wheel["Versioned CLI wheel"]
    Wheel --> CLI
    Agent["Coding agent"] --> CLI["LeadDemon CLI"]
    CLI --> Proxy["/api/backend proxy"]
    Web --> Proxy
    Proxy --> API["FastAPI product API"]
    API --> Auth["Clerk + scoped CLI sessions"]
    API --> Pipeline["Discovery and CRM-gap pipeline"]
    API --> Store["Product DB + artifact storage"]
    CLI --> Files["Selected CSV / XLSX prospect rows"]
    Agent --> FacebookResearch["Optional Facebook prospecting skill"]
    FacebookResearch --> Evidence["Permitted business research + qualification"]
    Evidence --> Files
    Files --> Dedupe["Dedupe-only queued job"]
    API --> Dedupe
    Dedupe --> HubSpot["Fresh connected-HubSpot company pull"]
    HubSpot --> Match["Conservative company matching"]
    Match --> Store
    Human --> OAuth["HubSpot / Stripe authorization"]
    OAuth --> API
```

## Agent CLI

Install the current CLI without repository access:

```bash
pipx install https://leaddemon.io/downloads/leaddemon-0.2.0-py3-none-any.whl
leaddemon login
leaddemon whoami
leaddemon workspace status
```

The login command opens LeadDemon in a browser, where the account owner approves
the CLI session. Credentials are stored in an owner-readable file under
`~/.config/leaddemon/credentials.json`; access tokens expire hourly and refresh
sessions rotate for up to 30 days. Sessions can be revoked from the account
page.

Install version-matched instructions for a coding agent in the current project:

```bash
leaddemon agent init
```

Install the separate Facebook business-research skill when needed:

```bash
leaddemon agent init --skill facebook-prospecting
```

The skill uses the agent's available research tools to qualify businesses from
Facebook evidence and optionally hand off a prospect file to fresh-HubSpot dedupe.
It is not a hosted Facebook crawler, does not bypass source-access restrictions,
and performs no owner research or outreach. See the
[skill source](src/leaddemon/skills/facebook-prospecting/SKILL.md) for scope and
evidence requirements.

The public [Agent Setup guide](https://leaddemon.io/agents) and its
[plain-text version](https://leaddemon.io/agents.txt) cover installation without
repository access. See [`docs/agent-cli.md`](docs/agent-cli.md) for the in-repo
reference on run configuration, JSON output, paid-source approval, downloads,
and human handoffs.

Check an existing prospect file against a fresh HubSpot pull without running
discovery or writing to the CRM:

```bash
leaddemon dedupe prospects.xlsx --sheet Prospects --dry-run --json
leaddemon dedupe prospects.xlsx --sheet Prospects --output ./dedupe-results --json
```

The CLI reads only selected tables. `POST /api/dedupe` persists those inputs in
artifact storage and queues a tenant-scoped job using the existing worker. The
job retrieves current non-archived HubSpot companies, compares them in
`src/leaddemon/prospect_dedupe.py`, and publishes `net_new.csv` plus
`dedupe_review.xlsx`. Status and downloads use the existing run endpoints. No
new always-on service or database migration is required.

## Local Setup

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

Run the API locally:

```bash
leaddemon-web --reload
```

Run a local queued worker:

```bash
LEAD_PIPELINE_RUNNER_MODE=queue leaddemon-web --reload
leaddemon-worker --worker-id local-worker --poll-interval-seconds 5
```

Run critical tests:

```bash
./scripts/run_critical_app_tests.sh
```

Run the opt-in local Stripe billing test against the Winklytics Stripe sandbox:

```bash
cd apps/web
LEADDEMON_LIVE_STRIPE_E2E=1 npm run test:e2e:stripe
```

This starts the local Python API, starts the local Next.js app, starts a Stripe
CLI webhook listener, creates a test subscription through Stripe Checkout with
Stripe's test card, opens the Customer Portal, cancels the test subscription via
Stripe's test API, and waits for the local webhook to mark billing canceled.

## Generic Pipeline Commands

Plan paid Google usage without making provider calls:

```bash
leaddemon prospects \
  --config config/prospecting_all_states.json \
  --google-plan-only
```

Run a free discovery refresh:

```bash
leaddemon prospects \
  --config config/prospecting_all_states.json \
  --skip-google \
  --skip-yelp \
  --use-discovery \
  --output data/output/prospects_review.csv
```

Run HubSpot gap analysis from a prospect CSV and CRM export:

```bash
leaddemon hubspot-gap \
  --prospects data/output/prospects_review.csv \
  --hubspot-export "/path/to/crm-export.xlsx" \
  --profile config/lead_gap_profiles/generic.json \
  --output-dir data/output/hubspot_gap \
  --stem leaddemon_gap \
  --vet-offerings \
  --enrich-contacts
```

## Configuration

Generic active configs live in:

- `config/prospecting_all_states.json`
- `config/prospecting_starter_states.json`
- `config/prospecting_core.json`
- `config/lead_gap_profiles/generic.json`

These configs intentionally ship without industry-specific search terms,
service terms, negative signals, or excluded company lists. Runtime target
settings come from the web app or CLI inputs for each run.

## Data Safety

- Paid provider runs require explicit estimate acknowledgement.
- Direct CRM upload/push should stay disabled unless a user explicitly confirms
  after reviewing output files.
- Secrets belong in environment variables, not source control.
- Production should use managed Postgres and blob/object storage rather than
  local SQLite or local filesystem artifacts.
