Pro-Coder commited on
Commit
4c8ed45
·
verified ·
1 Parent(s): 1efaafd

Upload 44 files

Browse files
Files changed (4) hide show
  1. DEPLOY.md +19 -29
  2. README.md +18 -9
  3. app.py +23 -23
  4. src/llm_client.py +207 -133
DEPLOY.md CHANGED
@@ -59,35 +59,25 @@ The Space will automatically build from `requirements.txt` and launch
59
 
60
  ## Enabling the LLM (recommended before sharing this project)
61
 
62
- By default the Space runs in **retrieval-only fallback mode** — it still
63
- works, but answers are extractive rather than LLM-generated. To turn on
64
- real LLM responses:
65
-
66
- 1. Create an access token at https://huggingface.co/settings/tokens
67
- (a "Read" token is sufficient for Inference API calls).
68
- 2. In your Space, go to **Settings → Variables and secrets → New secret**.
69
- - Name: `HF_TOKEN`
70
- - Value: your token
71
- 3. (Optional) Add another secret/variable `LLM_MODEL_ID` if you want a
72
- different hosted model than the default `Qwen/Qwen2.5-7B-Instruct`
73
- (any chat-capable model available via HF Inference Providers works).
74
- 4. Restart the Space (**Settings → Factory reboot**, or just wait — it
75
- picks up new secrets on the next restart).
76
-
77
- **Note on the API endpoint:** `src/llm_client.py` explicitly passes
78
- `provider="auto"` to `InferenceClient`, and `requirements.txt` pins a
79
- recent `huggingface_hub` version. Both matter:
80
- - Older client versions / omitting the provider can silently route calls
81
- through the now-deprecated `api-inference.huggingface.co` domain, which
82
- fails with a DNS resolution error rather than a clear auth error.
83
- - Hardcoding a specific provider (e.g. `"hf-inference"`) can fail with
84
- *"Model not supported by provider ..."* for models that are actually
85
- hosted via a different backend (Together, Fireworks, Novita, SambaNova,
86
- etc.) under HF's Inference Providers system. `provider="auto"` lets HF's
87
- router pick whichever backend actually serves each candidate model.
88
-
89
- If you ever see errors in the "Test LLM connection" diagnostics, they'll
90
- show you exactly which of the above it is per candidate model.
91
 
92
  ## Re-training / updating the models
93
 
 
59
 
60
  ## Enabling the LLM (recommended before sharing this project)
61
 
62
+ By default the Space runs in **retrieval-only fallback mode** — it still works, but answers are extractive
63
+ rather than LLM-generated. To turn on real LLM responses:
64
+
65
+ 1. Create a **free** API key at https://console.groq.com (Groq has a free plan and an OpenAI-compatible API).
66
+ 2. In your Space: **Settings → Variables and secrets → New secret**, name `GROQ_API_KEY`, value = your key.
67
+ 3. (Optional) add `LLM_MODEL` to choose another Groq model if the default (`llama-3.1-8b-instant`) is ever retired.
68
+ 4. Restart the Space, open **AI Assistant → LLM connection diagnostics → Test LLM connection**.
69
+
70
+ **Why not just use the Hugging Face token?** The app still supports `HF_TOKEN` (via HF's router), but two real
71
+ problems showed up in practice:
72
+ - `402 Payment Required` — Hugging Face Inference Providers bills against account credits, and free accounts
73
+ get almost none (a public demo exhausts it quickly). No code change can fix that.
74
+ - `Provider 'featherless-ai' not supported` — an old `InferenceClient` pinned in `requirements.txt` didn't know
75
+ providers that HF's router had started using. The client now speaks plain OpenAI-compatible HTTP instead of
76
+ the SDK, so there is no provider list to go stale.
77
+
78
+ Any other OpenAI-compatible service (OpenRouter, Together, Cerebras…) works with `LLM_API_KEY` + `LLM_BASE_URL`
79
+ + `LLM_MODEL`. A 402/401 stops further attempts on that backend immediately; other errors (rate limit, unknown
80
+ model, server error, timeout) move on to the next candidate.
 
 
 
 
 
 
 
 
 
 
81
 
82
  ## Re-training / updating the models
83
 
README.md CHANGED
@@ -64,20 +64,29 @@ Open the printed local URL (usually `http://127.0.0.1:7860`).
64
 
65
  ## Enabling full LLM responses
66
 
67
- The app works out of the box in **retrieval-only fallback mode** (no
68
- external API calls). To enable real LLM-generated answers:
 
69
 
70
- 1. Create a Hugging Face access token: https://huggingface.co/settings/tokens
71
- 2. Set it as an environment variable / Space secret named `HF_TOKEN`.
72
- 3. (Optional) Set `LLM_MODEL_ID` to override the default model
73
- (`Qwen/Qwen2.5-7B-Instruct`) with any chat-capable model available via
74
- HF Inference Providers.
 
 
 
 
 
 
75
 
76
  ```bash
77
- export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx
78
  python app.py
79
  ```
80
 
 
 
81
  ## Deploying to Hugging Face Spaces
82
 
83
  See [`DEPLOY.md`](./DEPLOY.md) for full step-by-step instructions.
@@ -98,7 +107,7 @@ smart-warehouse-ai/
98
  │ ├── retriever.py # TF-IDF retriever
99
  │ ├── intent_model.py # intent classifier (train/predict)
100
  │ ├── anomaly_model.py # Isolation Forest anomaly detector
101
- │ ├── llm_client.py # HF Inference API client + fallback
102
  │ └── inventory_db.py # NL -> structured query helpers
103
  ├── models/ # trained model artifacts (.joblib)
104
  ├── data/ # generated datasets + evaluation JSON
 
64
 
65
  ## Enabling full LLM responses
66
 
67
+ The app works out of the box in **retrieval-only fallback mode** (no external API calls). For real
68
+ LLM-generated answers it needs an API key for any **OpenAI-compatible** chat endpoint. The simplest free
69
+ option is Groq:
70
 
71
+ 1. Create a free API key at https://console.groq.com
72
+ 2. Set it as a Space secret (or env var) named `GROQ_API_KEY`. The default model is `llama-3.1-8b-instant`;
73
+ override with `LLM_MODEL` if Groq retires or renames it.
74
+
75
+ Other options (checked in this order; the first that answers wins):
76
+
77
+ | Secrets | Backend |
78
+ |---|---|
79
+ | `LLM_API_KEY` + `LLM_BASE_URL` + `LLM_MODEL` | any OpenAI-compatible API (OpenRouter, Together, Cerebras, …) |
80
+ | `GROQ_API_KEY` | Groq (free plan) |
81
+ | `HF_TOKEN` | Hugging Face Inference Providers router — **needs credits**; free accounts get almost none, which shows up as `402 Payment Required` |
82
 
83
  ```bash
84
+ export GROQ_API_KEY=gsk_xxxxxxxxxxxx
85
  python app.py
86
  ```
87
 
88
+ The **Test LLM connection** button (AI Assistant tab) shows exactly what each backend returned.
89
+
90
  ## Deploying to Hugging Face Spaces
91
 
92
  See [`DEPLOY.md`](./DEPLOY.md) for full step-by-step instructions.
 
107
  │ ├── retriever.py # TF-IDF retriever
108
  │ ├── intent_model.py # intent classifier (train/predict)
109
  │ ├── anomaly_model.py # Isolation Forest anomaly detector
110
+ │ ├── llm_client.py # OpenAI-compatible LLM client (Groq / custom / HF router) + fallback
111
  │ └── inventory_db.py # NL -> structured query helpers
112
  ├── models/ # trained model artifacts (.joblib)
113
  ├── data/ # generated datasets + evaluation JSON
app.py CHANGED
@@ -5,7 +5,7 @@ A warehouse/intralogistics AI copilot demo, built for a Hugging Face
5
  Space. Combines:
6
 
7
  1. An LLM-powered assistant (RAG: TF-IDF retrieval + hosted LLM via the
8
- HF Inference API) for natural-language warehouse operations Q&A.
9
  2. A TF-IDF + Logistic Regression intent classifier that routes queries
10
  into 8 operational categories.
11
  3. A lightweight NL -> structured query layer over synthetic inventory /
@@ -36,7 +36,7 @@ from src.real_vision import (
36
  from src.data_generation import generate_inventory_db, generate_orders_db
37
  from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
38
  from src.inventory_db import query_inventory, query_orders
39
- from src.llm_client import answer_query, test_connection, _get_hf_token, TOKEN_ENV_VAR_CANDIDATES
40
  from src.retriever import KBRetriever
41
 
42
  ROOT = os.path.dirname(os.path.abspath(__file__))
@@ -112,13 +112,13 @@ retrieval_eval = _load_json("retrieval_eval.json")
112
  latency_eval = _load_json("latency_eval.json")
113
  vision_eval = _load_json("vision_eval.json")
114
 
115
- HF_TOKEN_SET = bool(_get_hf_token())
116
 
117
  # --------------------------------------------------------------------------
118
  # ZeroGPU compatibility shim
119
  # --------------------------------------------------------------------------
120
  # This app is CPU-only by design (scikit-learn locally, LLM calls go to the
121
- # remote HF Inference API). Some Spaces accounts, however, only offer the
122
  # free "ZeroGPU" hardware tier, which requires at least one function
123
  # decorated with `@spaces.GPU` to be present or the platform's startup
124
  # check fails with "No @spaces.GPU function detected". This is a harmless,
@@ -171,15 +171,14 @@ def test_llm_fn():
171
  f"✅ **LLM connection working.** Model: `{response.model_id}` · "
172
  f"{response.latency_s * 1000:.0f} ms\n\nSample answer: {response.answer}"
173
  )
174
- err_lines = "\n".join(f"- `{e}`" for e in response.debug_errors) or "(no error detail captured)"
175
  return (
176
  "❌ **LLM connection failed** — running in retrieval-only fallback mode.\n\n"
177
- f"**Errors from each candidate model tried:**\n{err_lines}\n\n"
178
- "**Common causes:** missing/invalid `HF_TOKEN` secret, the token's account "
179
- "lacking Inference API access, or the candidate models being temporarily "
180
- "unavailable on HF's free serverless tier. See `src/llm_client.py` to add "
181
- "or reorder candidate models, or set the `LLM_MODEL_ID` secret to force a "
182
- "specific one."
183
  )
184
 
185
 
@@ -459,7 +458,7 @@ being realistic enough to demonstrate the underlying ML techniques properly.
459
  |---|---|---|---|
460
  | **Intent classifier** | Decide what kind of request a query is (maintenance, safety, inventory, order status, navigation, picking, system status, general FAQ) | TF-IDF + Logistic Regression | Free text → intent label + confidence |
461
  | **KB retriever** | Find the most relevant knowledge-base passage(s) for a query, to ground the LLM's answer | TF-IDF + cosine similarity | Free text → top-k ranked passages |
462
- | **LLM assistant** | Generate a natural-language answer grounded in the retrieved context | Hosted instruct LLM (HF Inference API), multi-model fallback chain | Query + context → grounded answer (or an extractive fallback if the LLM is unavailable) |
463
  | **Inventory/order query** | Turn a question into a filtered table lookup | Regex slot extraction (SKU / order ID / zone) + intent routing | Free text → filtered inventory or orders table |
464
  | **Anomaly detector** | Flag abnormal equipment sensor readings before they cause a stoppage | Isolation Forest (unsupervised), StandardScaler | 4 sensor features → anomaly / normal + anomaly score |
465
 
@@ -485,11 +484,12 @@ being realistic enough to demonstrate the underlying ML techniques properly.
485
  ## Tech stack
486
 
487
  - **UI / deployment:** [Gradio](https://gradio.app) on Hugging Face Spaces
488
- - **LLM:** Hosted instruct model via the Hugging Face **Inference API**
489
- (`huggingface_hub.InferenceClient`), with a multi-model fallback chain and
490
- auto provider routing. Configurable via the `LLM_MODEL_ID` env var. Falls
491
- back gracefully to a retrieval-only answer if no API token is configured
492
- or every candidate model fails, so the public demo never just breaks.
 
493
  - **Retrieval:** TF-IDF + cosine similarity over a small hand-written
494
  warehouse-operations knowledge base (simple, fast, fully local RAG).
495
  - **Intent classification:** TF-IDF + Logistic Regression (scikit-learn) --
@@ -517,7 +517,7 @@ being realistic enough to demonstrate the underlying ML techniques properly.
517
  │ top-k passages
518
  ▼
519
  ┌───────────────────────┐
520
- │ Hosted LLM │ (HF Inference API)
521
  │ (or extractive │
522
  │ fallback if offline) │
523
  └──────────┬────────────┘
@@ -763,12 +763,12 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
763
  gr.Markdown("<h1 id='title-banner'>🏭 Smart Warehouse AI Assistant</h1>")
764
  subtitle_md = gr.Markdown(I18N["subtitle"]["en"])
765
  lang_radio = gr.Radio(["English", "日本語"], value="English", label="🌐 Language / 言語", interactive=True)
766
- if not HF_TOKEN_SET:
767
  gr.Markdown(
768
- "> ⚠️ **No HF token secret detected.** The AI Assistant tab will run in "
769
- "**retrieval-only fallback mode** (still functional, just not LLM-generated "
770
- f"prose). Add a secret named `HF_TOKEN` (or one of: {', '.join(TOKEN_ENV_VAR_CANDIDATES[1:])}) "
771
- "in *Space settings → Variables and secrets* to enable full LLM responses."
772
  )
773
 
774
  with gr.Tab("💬 AI Assistant"):
 
5
  Space. Combines:
6
 
7
  1. An LLM-powered assistant (RAG: TF-IDF retrieval + hosted LLM via the
8
+ an OpenAI-compatible API) for natural-language warehouse operations Q&A.
9
  2. A TF-IDF + Logistic Regression intent classifier that routes queries
10
  into 8 operational categories.
11
  3. A lightweight NL -> structured query layer over synthetic inventory /
 
36
  from src.data_generation import generate_inventory_db, generate_orders_db
37
  from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
38
  from src.inventory_db import query_inventory, query_orders
39
+ from src.llm_client import answer_query, test_connection, llm_configured
40
  from src.retriever import KBRetriever
41
 
42
  ROOT = os.path.dirname(os.path.abspath(__file__))
 
112
  latency_eval = _load_json("latency_eval.json")
113
  vision_eval = _load_json("vision_eval.json")
114
 
115
+ LLM_CONFIGURED = llm_configured()
116
 
117
  # --------------------------------------------------------------------------
118
  # ZeroGPU compatibility shim
119
  # --------------------------------------------------------------------------
120
  # This app is CPU-only by design (scikit-learn locally, LLM calls go to the
121
+ # remote LLM API). Some Spaces accounts, however, only offer the
122
  # free "ZeroGPU" hardware tier, which requires at least one function
123
  # decorated with `@spaces.GPU` to be present or the platform's startup
124
  # check fails with "No @spaces.GPU function detected". This is a harmless,
 
171
  f"✅ **LLM connection working.** Model: `{response.model_id}` · "
172
  f"{response.latency_s * 1000:.0f} ms\n\nSample answer: {response.answer}"
173
  )
174
+ err_lines = "\n".join(f"- {e}" for e in response.debug_errors) or "(no error detail captured)"
175
  return (
176
  "❌ **LLM connection failed** — running in retrieval-only fallback mode.\n\n"
177
+ f"**What each configured backend reported:**\n{err_lines}\n\n"
178
+ "**Most reliable fix:** add a `GROQ_API_KEY` Space secret (free key at console.groq.com), then "
179
+ "restart the Space. Hugging Face's router needs inference credits, and free accounts get almost none "
180
+ "(that's the `402 Payment Required` error). Any OpenAI-compatible API also works via "
181
+ "`LLM_API_KEY` + `LLM_BASE_URL` + `LLM_MODEL`."
 
182
  )
183
 
184
 
 
458
  |---|---|---|---|
459
  | **Intent classifier** | Decide what kind of request a query is (maintenance, safety, inventory, order status, navigation, picking, system status, general FAQ) | TF-IDF + Logistic Regression | Free text → intent label + confidence |
460
  | **KB retriever** | Find the most relevant knowledge-base passage(s) for a query, to ground the LLM's answer | TF-IDF + cosine similarity | Free text → top-k ranked passages |
461
+ | **LLM assistant** | Generate a natural-language answer grounded in the retrieved context | Hosted instruct LLM via an OpenAI-compatible API (Groq / custom / HF router), with fallback | Query + context → grounded answer (or an extractive fallback if the LLM is unavailable) |
462
  | **Inventory/order query** | Turn a question into a filtered table lookup | Regex slot extraction (SKU / order ID / zone) + intent routing | Free text → filtered inventory or orders table |
463
  | **Anomaly detector** | Flag abnormal equipment sensor readings before they cause a stoppage | Isolation Forest (unsupervised), StandardScaler | 4 sensor features → anomaly / normal + anomaly score |
464
 
 
484
  ## Tech stack
485
 
486
  - **UI / deployment:** [Gradio](https://gradio.app) on Hugging Face Spaces
487
+ - **LLM:** any hosted model behind an **OpenAI-compatible chat API** — Groq's
488
+ free tier (`GROQ_API_KEY`), a custom endpoint (`LLM_API_KEY` / `LLM_BASE_URL` /
489
+ `LLM_MODEL`), or Hugging Face's inference router. Plain HTTP rather than a
490
+ provider-specific SDK, so there's no provider list to go stale. It stops after
491
+ one attempt on a 402/401 (every model would fail the same way) and falls back
492
+ to a retrieval-only answer, so the public demo never just breaks.
493
  - **Retrieval:** TF-IDF + cosine similarity over a small hand-written
494
  warehouse-operations knowledge base (simple, fast, fully local RAG).
495
  - **Intent classification:** TF-IDF + Logistic Regression (scikit-learn) --
 
517
  │ top-k passages
518
  ▼
519
  ┌───────────────────────┐
520
+ │ Hosted LLM │ (OpenAI-compatible API)
521
  │ (or extractive │
522
  │ fallback if offline) │
523
  └──────────┬────────────┘
 
763
  gr.Markdown("<h1 id='title-banner'>🏭 Smart Warehouse AI Assistant</h1>")
764
  subtitle_md = gr.Markdown(I18N["subtitle"]["en"])
765
  lang_radio = gr.Radio(["English", "日本語"], value="English", label="🌐 Language / 言語", interactive=True)
766
+ if not LLM_CONFIGURED:
767
  gr.Markdown(
768
+ "> ⚠️ **No LLM configured.** The AI Assistant tab runs in **retrieval-only fallback mode** "
769
+ "(still functional, just not LLM-generated prose). To enable full answers, add a Space secret "
770
+ "`GROQ_API_KEY` (free key at console.groq.com) — or `LLM_API_KEY` + `LLM_BASE_URL` + `LLM_MODEL` "
771
+ "for any OpenAI-compatible API, or an `HF_TOKEN` with Hugging Face inference credits."
772
  )
773
 
774
  with gr.Tab("💬 AI Assistant"):
src/llm_client.py CHANGED
@@ -1,72 +1,76 @@
1
  """
2
  llm_client.py
3
  -------------
4
- Wraps a hosted LLM (via huggingface_hub's InferenceClient, using HF's
5
- serverless Inference Providers) to answer warehouse-operations questions,
6
- grounded with context retrieved from the local knowledge base (simple RAG).
7
-
8
- Design notes
9
- ------------
10
- * Reads the HF token from the `HF_TOKEN` environment variable, which should
11
- be added as a Space "secret" when deployed (Settings -> Variables and
12
- secrets). The public demo also works without a token: it falls back to a
13
- deterministic, still-useful extractive answer built from the retrieved
14
- knowledge-base passages, so the Space never shows a broken demo.
15
- * Serverless model availability on the free HF Inference API changes over
16
- time (models get gated, deprecated, or moved between providers), so
17
- rather than hard-depending on a single model id, we try a short list of
18
- candidates in order and use the first one that responds successfully.
19
- `LLM_MODEL_ID` (env var) is tried first if set, ahead of the built-in list.
20
- * On failure, the *actual* exception message (not just its type) is
21
- surfaced back to the UI, so a broken deployment is debuggable from the
22
- Space itself instead of requiring log access.
 
 
 
 
 
 
 
 
 
 
 
23
  """
24
 
25
  import os
 
26
  import time
27
  from dataclasses import dataclass, field
28
- from typing import List, Optional
 
 
29
 
30
  from src.retriever import KBRetriever, RetrievedDoc
31
 
32
- # Small, widely-available instruct models known to work well on HF's free
33
- # serverless Inference API. Tried in order; first success wins. If
34
- # LLM_MODEL_ID is set as an env var, it is tried first, ahead of this list.
 
 
 
 
 
35
  MODEL_CANDIDATES = [
36
  "Qwen/Qwen2.5-7B-Instruct",
37
  "meta-llama/Llama-3.1-8B-Instruct",
38
  "meta-llama/Llama-3.2-3B-Instruct",
39
  "mistralai/Mistral-7B-Instruct-v0.3",
40
- "microsoft/Phi-3.5-mini-instruct",
41
- "HuggingFaceH4/zephyr-7b-beta",
42
  ]
43
-
44
  _env_model = os.environ.get("LLM_MODEL_ID")
45
  if _env_model:
46
  MODEL_CANDIDATES = [_env_model] + [m for m in MODEL_CANDIDATES if m != _env_model]
47
-
48
  DEFAULT_MODEL_ID = MODEL_CANDIDATES[0]
49
 
50
- # The token is normally expected as `HF_TOKEN`, but we also accept a few
51
- # common alternate secret names in case the Space was set up with a
52
- # different name. First one found wins. Add your own name here if needed.
53
  TOKEN_ENV_VAR_CANDIDATES = [
54
- "HF_TOKEN",
55
- "Smart_Warehouse",
56
- "HUGGINGFACE_TOKEN",
57
- "HUGGINGFACEHUB_API_TOKEN",
58
- "HF_API_TOKEN",
59
- "HUGGING_FACE_HUB_TOKEN",
60
- "HF_ACCESS_TOKEN",
61
  ]
62
 
63
-
64
- def _get_hf_token() -> Optional[str]:
65
- for var in TOKEN_ENV_VAR_CANDIDATES:
66
- val = os.environ.get(var)
67
- if val:
68
- return val
69
- return None
70
 
71
  SYSTEM_PROMPT = (
72
  "You are the Smart Warehouse AI Assistant, a helpful operations copilot "
@@ -74,10 +78,130 @@ SYSTEM_PROMPT = (
74
  "sortation, and a WMS). Answer concisely and practically, in the tone of "
75
  "an experienced warehouse operations engineer. Use the provided CONTEXT "
76
  "when relevant, and say so plainly if the question is outside the "
77
- "context. Prefer short paragraphs or bullet points over long prose."
 
78
  )
79
 
80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  @dataclass
82
  class AssistantResponse:
83
  answer: str
@@ -85,35 +209,23 @@ class AssistantResponse:
85
  sources: List[RetrievedDoc]
86
  latency_s: float
87
  model_id: str
88
- debug_errors: List[str] = field(default_factory=list) # non-empty only when used_llm is False due to failures
89
 
90
 
91
  def _extractive_fallback(query: str, sources: List[RetrievedDoc]) -> str:
92
- """Deterministic answer used when no HF token / API call fails, so the
93
- Space always returns something useful instead of an error."""
94
  if not sources:
95
- return (
96
- "I don't have grounded context for that yet. Try asking about "
97
- "inventory, order status, equipment maintenance, AGV routing, "
98
- "picking strategy, safety incidents, or general warehouse "
99
- "automation concepts."
100
- )
101
- lead = sources[0]
102
  bullets = "\n".join(f"- **{s.title}**: {s.text}" for s in sources)
103
- return (
104
- f"(Showing retrieved knowledge instead of an LLM-generated answer -- "
105
- f"see the diagnostics below.)\n\n"
106
- f"Based on **{lead.title}**, here's the relevant information:\n\n{bullets}"
107
- )
108
-
109
-
110
- MAX_HISTORY_TURNS = 4 # prior (user, assistant) pairs carried into the LLM call
111
 
112
 
113
  def _clean_history(history: Optional[List[dict]]) -> List[dict]:
114
- """Normalize Gradio's messages-format history into a short, clean list safe
115
- to feed back to the LLM: strips our own metadata footer from assistant
116
- turns and keeps only the last MAX_HISTORY_TURNS pairs (bounded prompt)."""
117
  if not history:
118
  return []
119
  cleaned = []
@@ -135,82 +247,44 @@ def answer_query(
135
  max_tokens: int = 350,
136
  history: Optional[List[dict]] = None,
137
  ) -> AssistantResponse:
138
- """`history`: prior turns (Gradio messages format) enabling multi-turn
139
- follow-ups like "what about Zone D instead?". Omit for single-turn."""
140
  start = time.time()
141
  sources = retriever.retrieve(query, k=k)
142
  context_block = "\n\n".join(f"[{s.title}]\n{s.text}" for s in sources)
143
- prior_turns = _clean_history(history)
144
-
145
- hf_token = _get_hf_token()
146
-
147
- if not hf_token:
148
- answer = _extractive_fallback(query, sources)
149
- return AssistantResponse(
150
- answer=answer,
151
- used_llm=False,
152
- sources=sources,
153
- latency_s=time.time() - start,
154
- model_id="extractive-fallback",
155
- debug_errors=[
156
- "No HF token secret found. Checked env vars: "
157
- + ", ".join(TOKEN_ENV_VAR_CANDIDATES)
158
- + ". Set one of these as a Space secret (Settings -> Variables and secrets)."
159
- ],
160
- )
161
-
162
- try:
163
- from huggingface_hub import InferenceClient
164
- except ImportError as e:
165
- answer = _extractive_fallback(query, sources)
166
- return AssistantResponse(
167
- answer=answer, used_llm=False, sources=sources,
168
- latency_s=time.time() - start, model_id="extractive-fallback",
169
- debug_errors=[f"huggingface_hub not importable: {e}"],
170
- )
171
 
172
  messages = [{"role": "system", "content": SYSTEM_PROMPT}]
173
- messages.extend(prior_turns)
174
  messages.append({"role": "user", "content": f"CONTEXT:\n{context_block}\n\nQUESTION: {query}"})
175
 
176
- errors = []
177
- for candidate in MODEL_CANDIDATES:
178
- try:
179
- # provider="auto" (the huggingface_hub default) lets HF's Inference
180
- # Providers router pick whichever backend (hf-inference, Together,
181
- # Fireworks, Novita, SambaNova, etc.) actually serves this specific
182
- # model -- hardcoding a single provider caused "model not supported
183
- # by provider X" errors for models hosted elsewhere.
184
- client = InferenceClient(model=candidate, token=hf_token, provider="auto")
185
- completion = client.chat_completion(messages=messages, max_tokens=max_tokens, temperature=0.3)
186
- text = completion.choices[0].message.content
187
- if text and text.strip():
188
- return AssistantResponse(
189
- answer=text,
190
- used_llm=True,
191
- sources=sources,
192
- latency_s=time.time() - start,
193
- model_id=candidate,
194
- )
195
- errors.append(f"{candidate}: empty response")
196
- except Exception as e: # noqa: BLE001 -- try the next candidate model
197
- errors.append(f"{candidate}: {type(e).__name__}: {e}")
198
-
199
- # All candidates failed -- fall back, but surface the real errors so the
200
- # deployment is debuggable directly from the UI.
201
- answer = _extractive_fallback(query, sources)
202
- return AssistantResponse(
203
- answer=answer,
204
- used_llm=False,
205
- sources=sources,
206
- latency_s=time.time() - start,
207
- model_id="extractive-fallback",
208
- debug_errors=errors,
209
- )
210
 
211
 
212
  def test_connection(retriever: Optional[KBRetriever] = None) -> AssistantResponse:
213
- """Runs a single canned query through the full pipeline -- used by the
214
- 'Test LLM connection' diagnostics button in the app."""
215
  retriever = retriever or KBRetriever()
216
  return answer_query("What is a WMS?", retriever, k=1, max_tokens=60)
 
1
  """
2
  llm_client.py
3
  -------------
4
+ Answers warehouse-operations questions with a hosted LLM, grounded in context
5
+ retrieved from the local knowledge base (simple RAG), with multi-turn memory.
6
+
7
+ Why this talks plain HTTP instead of `huggingface_hub.InferenceClient`
8
+ ---------------------------------------------------------------------
9
+ An earlier version used InferenceClient(provider="auto"). That broke twice in
10
+ the wild: (1) Hugging Face's router started assigning models to providers
11
+ ('featherless-ai', 'deepinfra') that the pinned client version didn't know,
12
+ raising "Provider ... not supported"; (2) the account had no Inference
13
+ Providers credits, giving HTTP 402. Everything modern (Groq, OpenRouter,
14
+ Together, Cerebras, HF's own router, ...) exposes the same OpenAI-compatible
15
+ `/chat/completions` protocol, so this module speaks that directly. There is
16
+ no provider enum to go stale, and any such endpoint can be used by setting
17
+ env vars / Space secrets.
18
+
19
+ Backends, tried in this order (first success wins)
20
+ --------------------------------------------------
21
+ 1. Custom OpenAI-compatible endpoint:
22
+ LLM_API_KEY + LLM_BASE_URL + LLM_MODEL
23
+ or just `GROQ_API_KEY` (Groq has a free plan; base URL and a default model
24
+ are filled in; override the model with LLM_MODEL).
25
+ 2. Hugging Face Inference Providers router (https://router.huggingface.co/v1),
26
+ using the HF token. NOTE: free HF accounts get very little monthly credit;
27
+ when it runs out the router answers 402 and this backend is skipped.
28
+ 3. Extractive fallback: the retrieved knowledge-base passages are shown
29
+ directly, clearly labelled -- the demo never breaks.
30
+
31
+ Failure handling: a 402/401/403 means *every* model on that backend will fail
32
+ the same way, so the backend is abandoned after one attempt instead of
33
+ hammering six models. Error text never includes API keys.
34
  """
35
 
36
  import os
37
+ import re
38
  import time
39
  from dataclasses import dataclass, field
40
+ from typing import List, Optional, Tuple
41
+
42
+ import requests
43
 
44
  from src.retriever import KBRetriever, RetrievedDoc
45
 
46
+ # ---------------------------------------------------------------------------
47
+ # Configuration
48
+ # ---------------------------------------------------------------------------
49
+ GROQ_BASE_URL = "https://api.groq.com/openai/v1"
50
+ GROQ_DEFAULT_MODEL = "llama-3.1-8b-instant"
51
+ HF_ROUTER_BASE_URL = os.environ.get("HF_ROUTER_BASE_URL", "https://router.huggingface.co/v1")
52
+
53
+ # Models tried on the Hugging Face router, in order (LLM_MODEL_ID goes first if set).
54
  MODEL_CANDIDATES = [
55
  "Qwen/Qwen2.5-7B-Instruct",
56
  "meta-llama/Llama-3.1-8B-Instruct",
57
  "meta-llama/Llama-3.2-3B-Instruct",
58
  "mistralai/Mistral-7B-Instruct-v0.3",
 
 
59
  ]
 
60
  _env_model = os.environ.get("LLM_MODEL_ID")
61
  if _env_model:
62
  MODEL_CANDIDATES = [_env_model] + [m for m in MODEL_CANDIDATES if m != _env_model]
 
63
  DEFAULT_MODEL_ID = MODEL_CANDIDATES[0]
64
 
65
+ # The HF token is normally `HF_TOKEN`; a few other names are accepted too.
 
 
66
  TOKEN_ENV_VAR_CANDIDATES = [
67
+ "HF_TOKEN", "Smart_Warehouse", "HUGGINGFACE_TOKEN", "HUGGINGFACEHUB_API_TOKEN",
68
+ "HF_API_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HF_ACCESS_TOKEN",
 
 
 
 
 
69
  ]
70
 
71
+ REQUEST_TIMEOUT_S = 40
72
+ TOTAL_BUDGET_S = 75 # stop trying more candidates after this long (UI shouldn't hang)
73
+ MAX_HISTORY_TURNS = 4 # prior (user, assistant) pairs carried into the call
 
 
 
 
74
 
75
  SYSTEM_PROMPT = (
76
  "You are the Smart Warehouse AI Assistant, a helpful operations copilot "
 
78
  "sortation, and a WMS). Answer concisely and practically, in the tone of "
79
  "an experienced warehouse operations engineer. Use the provided CONTEXT "
80
  "when relevant, and say so plainly if the question is outside the "
81
+ "context. Prefer short paragraphs or bullet points over long prose. "
82
+ "Reply in the same language the user wrote in."
83
  )
84
 
85
 
86
+ def _get_hf_token() -> Optional[str]:
87
+ for var in TOKEN_ENV_VAR_CANDIDATES:
88
+ val = os.environ.get(var)
89
+ if val:
90
+ return val
91
+ return None
92
+
93
+
94
+ @dataclass
95
+ class Backend:
96
+ label: str
97
+ base_url: str
98
+ api_key: str
99
+ models: List[str]
100
+
101
+
102
+ def get_backends() -> List[Backend]:
103
+ """Backends configured via env vars / Space secrets, in priority order."""
104
+ backends: List[Backend] = []
105
+ key, url, model = os.environ.get("LLM_API_KEY"), os.environ.get("LLM_BASE_URL"), os.environ.get("LLM_MODEL")
106
+ if key and url:
107
+ host = re.sub(r"^https?://", "", url).split("/")[0]
108
+ backends.append(Backend(f"custom ({host})", url.rstrip("/"), key, [model] if model else []))
109
+ groq_key = os.environ.get("GROQ_API_KEY")
110
+ if groq_key:
111
+ backends.append(Backend("Groq", GROQ_BASE_URL, groq_key, [model or GROQ_DEFAULT_MODEL]))
112
+ hf = _get_hf_token()
113
+ if hf:
114
+ backends.append(Backend("Hugging Face router", HF_ROUTER_BASE_URL.rstrip("/"), hf, list(MODEL_CANDIDATES)))
115
+ return backends
116
+
117
+
118
+ def llm_configured() -> bool:
119
+ return bool(get_backends())
120
+
121
+
122
+ # ---------------------------------------------------------------------------
123
+ # Wire protocol (OpenAI-compatible /chat/completions)
124
+ # ---------------------------------------------------------------------------
125
+ class LLMError(Exception):
126
+ """kind: credits | auth | rate | not_found | server | network | parse | empty"""
127
+ def __init__(self, kind: str, message: str, status: Optional[int] = None):
128
+ super().__init__(message)
129
+ self.kind, self.status = kind, status
130
+
131
+
132
+ # Failures after which trying more models on the SAME backend is pointless.
133
+ BACKEND_FATAL = {"credits", "auth"}
134
+
135
+
136
+ def _scrub(text: str, secret: str) -> str:
137
+ text = text.replace(secret, "***") if secret else text
138
+ return re.sub(r"(hf_|gsk_|sk-)[A-Za-z0-9_\-]{8,}", r"\1***", text)
139
+
140
+
141
+ def _error_message(resp: requests.Response) -> str:
142
+ try:
143
+ data = resp.json()
144
+ err = data.get("error", data)
145
+ if isinstance(err, dict):
146
+ return str(err.get("message") or err)[:240]
147
+ return str(err)[:240]
148
+ except Exception: # noqa: BLE001
149
+ return (resp.text or "")[:240].strip()
150
+
151
+
152
+ def _chat(backend: Backend, model: str, messages: list, max_tokens: int, temperature: float = 0.3) -> str:
153
+ try:
154
+ resp = requests.post(
155
+ f"{backend.base_url}/chat/completions",
156
+ headers={"Authorization": f"Bearer {backend.api_key}", "Content-Type": "application/json"},
157
+ json={"model": model, "messages": messages, "max_tokens": max_tokens, "temperature": temperature},
158
+ timeout=REQUEST_TIMEOUT_S,
159
+ )
160
+ except requests.exceptions.Timeout:
161
+ raise LLMError("network", f"timed out after {REQUEST_TIMEOUT_S}s")
162
+ except requests.exceptions.RequestException as e:
163
+ raise LLMError("network", _scrub(f"{type(e).__name__}: {e}", backend.api_key)[:200])
164
+
165
+ s = resp.status_code
166
+ if s == 200:
167
+ try:
168
+ content = resp.json()["choices"][0]["message"]["content"]
169
+ except Exception: # noqa: BLE001
170
+ raise LLMError("parse", "response was not a valid chat completion", s)
171
+ if not content or not str(content).strip():
172
+ raise LLMError("empty", "model returned an empty answer", s)
173
+ return str(content).strip()
174
+
175
+ detail = _scrub(_error_message(resp), backend.api_key)
176
+ if s == 402:
177
+ raise LLMError("credits", detail or "payment required", s)
178
+ if s in (401, 403):
179
+ raise LLMError("auth", detail or "unauthorized", s)
180
+ if s == 429:
181
+ raise LLMError("rate", detail or "rate limited", s)
182
+ if s in (400, 404, 410, 422):
183
+ raise LLMError("not_found", detail or "model/request not accepted", s)
184
+ raise LLMError("server", f"HTTP {s}: {detail}", s)
185
+
186
+
187
+ _HINTS = {
188
+ "credits": ("402 Payment Required — this account has no inference credits "
189
+ "(Hugging Face free accounts get almost none). Use a free-tier provider instead: "
190
+ "set a `GROQ_API_KEY` secret (free key at console.groq.com), or add credits."),
191
+ "auth": "key rejected (401/403) — check the secret's value and permissions.",
192
+ "rate": "rate limited (429) — the free tier's per-minute/day limit was hit; wait and retry.",
193
+ }
194
+
195
+
196
+ def _explain(backend: Backend, model: str, e: LLMError) -> str:
197
+ base = f"{backend.label} · {model or '(no model set)'}: {e.kind}"
198
+ hint = _HINTS.get(e.kind)
199
+ return f"{base} — {hint}" if hint else f"{base} — {e}"
200
+
201
+
202
+ # ---------------------------------------------------------------------------
203
+ # Public API
204
+ # ---------------------------------------------------------------------------
205
  @dataclass
206
  class AssistantResponse:
207
  answer: str
 
209
  sources: List[RetrievedDoc]
210
  latency_s: float
211
  model_id: str
212
+ debug_errors: List[str] = field(default_factory=list) # filled when used_llm is False
213
 
214
 
215
  def _extractive_fallback(query: str, sources: List[RetrievedDoc]) -> str:
216
+ """Deterministic answer used when no LLM is configured / reachable."""
 
217
  if not sources:
218
+ return ("I don't have grounded context for that yet. Try asking about inventory, order status, "
219
+ "equipment maintenance, AGV routing, picking strategy, safety incidents, or general "
220
+ "warehouse automation concepts.")
 
 
 
 
221
  bullets = "\n".join(f"- **{s.title}**: {s.text}" for s in sources)
222
+ return (f"(Showing retrieved knowledge instead of an LLM-generated answer -- see the diagnostics below.)\n\n"
223
+ f"Based on **{sources[0].title}**, here's the relevant information:\n\n{bullets}")
 
 
 
 
 
 
224
 
225
 
226
  def _clean_history(history: Optional[List[dict]]) -> List[dict]:
227
+ """Normalize Gradio's messages-format history: strip our own metadata footer from
228
+ assistant turns and keep only the last MAX_HISTORY_TURNS pairs (bounded prompt)."""
 
229
  if not history:
230
  return []
231
  cleaned = []
 
247
  max_tokens: int = 350,
248
  history: Optional[List[dict]] = None,
249
  ) -> AssistantResponse:
250
+ """`history`: prior turns (Gradio messages format) so follow-ups like
251
+ "what about Zone D instead?" work. Omit for single-turn."""
252
  start = time.time()
253
  sources = retriever.retrieve(query, k=k)
254
  context_block = "\n\n".join(f"[{s.title}]\n{s.text}" for s in sources)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
255
 
256
  messages = [{"role": "system", "content": SYSTEM_PROMPT}]
257
+ messages.extend(_clean_history(history))
258
  messages.append({"role": "user", "content": f"CONTEXT:\n{context_block}\n\nQUESTION: {query}"})
259
 
260
+ backends = get_backends()
261
+ errors: List[str] = []
262
+ if not backends:
263
+ errors.append("No LLM configured. Set a `GROQ_API_KEY` secret (free at console.groq.com), "
264
+ "or `LLM_API_KEY` + `LLM_BASE_URL` + `LLM_MODEL` for any OpenAI-compatible API, "
265
+ "or an `HF_TOKEN` (needs Hugging Face inference credits).")
266
+
267
+ for backend in backends:
268
+ if not backend.models:
269
+ errors.append(f"{backend.label}: LLM_MODEL is not set.")
270
+ continue
271
+ for model in backend.models:
272
+ if time.time() - start > TOTAL_BUDGET_S:
273
+ errors.append(f"Stopped after {TOTAL_BUDGET_S}s without a successful answer.")
274
+ break
275
+ try:
276
+ text = _chat(backend, model, messages, max_tokens)
277
+ return AssistantResponse(text, True, sources, time.time() - start, f"{backend.label}: {model}")
278
+ except LLMError as e:
279
+ errors.append(_explain(backend, model, e))
280
+ if e.kind in BACKEND_FATAL:
281
+ break # every other model on this backend would fail identically
282
+
283
+ return AssistantResponse(_extractive_fallback(query, sources), False, sources,
284
+ time.time() - start, "extractive-fallback", debug_errors=errors)
 
 
 
 
 
 
 
 
 
285
 
286
 
287
  def test_connection(retriever: Optional[KBRetriever] = None) -> AssistantResponse:
288
+ """One canned query through the full pipeline -- used by the diagnostics button."""
 
289
  retriever = retriever or KBRetriever()
290
  return answer_query("What is a WMS?", retriever, k=1, max_tokens=60)