robin/Dramlog-Prod
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Replace Nebius with Pixtral AI for bottle scanning

Browse Source
This commit is contained in:
robin
2025-12-19 21:53:18 +01:00
parent 06891a3619
commit 25b1378794
10 changed files with 184 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files

125
.aiideas
View File

@@ -1,46 +1,123 @@
3. Timeline & "Flight Recorder" (Reihenfolge-Logik)
Hier ist der Code, um Pixtral Large (das europäische Flaggschiff-Modell von Mistral) direkt in deine Next.js App zu integrieren.
Ziel: Die Geschichte des Abends rekonstruieren. Analyse des Konsumverhaltens.
Feature: Die Timeline-Ansicht
Damit kannst du einen direkten "A/B-Test" gegen Gemini 3 Flash fahren.
1. Vorbereitung
Statt einer einfachen Liste, eine vertikale Zeitstrahl-Ansicht.
Du brauchst das Mistral SDK und einen API Key von console.mistral.ai.
Bash
14:00 Uhr: Start der Session "Whisky Herbst".
npm install @mistralai/mistralai
14:15 Uhr: Glenfiddich 12 (Mild, Start).
Füge deinen Key in die .env ein (nicht NEXT_PUBLIC_!): MISTRAL_API_KEY=dein_key_hier
2. Der Code (Server Action)
15:30 Uhr: Laphroaig Cask Strength (Der Gaumen-Killer).
Erstelle eine neue Datei, z.B. app/actions/scan-mistral.ts.
TypeScript
16:00 Uhr: Auchentoshan (Schmeckt nach nichts mehr, weil Laphroaig davor war).
'use server'
Analyse & Warnungen (Smart Features):
import { Mistral } from '@mistralai/mistralai';
Der "Palette-Checker":
const client = new Mistral({ apiKey: process.env.MISTRAL_API_KEY });
Wenn der User einen extrem rauchigen Whisky (80ppm) loggt und 10 Minuten später einen milden Lowlander eintragen will.
export async function scanWithPixtral(base64Image: string, mimeType: string) {
// Pixtral braucht das Bild als Data-URL
const dataUrl = `data:${mimeType};base64,${base64Image}`;
Warnung (lustig): "Achtung! Du hast gerade eine Torfbombe getrunken. Warte lieber noch 10 Min oder trink Wasser, sonst schmeckst du den nächsten nicht!"
try {
const chatResponse = await client.chat.complete({
model: 'pixtral-large-latest', // Das beste Modell (Stand Dez 2025)
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: `Du bist ein Whisky-Experte und OCR-Spezialist.
Analysiere dieses Etikett präzise.
ABV-Kurve:
Antworte AUSSCHLIESSLICH mit gültigem JSON (kein Markdown, kein Text davor/danach):
{
"distillery": "Name der Brennerei (z.B. Lagavulin)",
"name": "Exakter Name/Edition (z.B. 16 Year Old)",
"vintage": "Jahrgang oder null",
"age": "Alter oder null (z.B. 16)",
"abv": "Alkoholgehalt (z.B. 43%)",
"search_query": "site:whiskybase.com [Brennerei] [Name] [Alter]"
}`
},
{
type: 'image_url',
imageUrl: dataUrl
}
]
}
],
responseFormat: { type: 'json_object' }, // Erzwingt JSON (wichtig!)
temperature: 0.1 // Niedrig = präziser, weniger Halluzinationen
});
Ein Liniendiagramm am Ende der Session: Wie hat sich der Alkoholgehalt entwickelt?
const rawContent = chatResponse.choices?.[0].message.content;
Ideal: Langsamer Anstieg.
if (!rawContent) throw new Error("Keine Antwort von Pixtral");
Gefährlich: Zick-Zack.
// JSON parsen
return JSON.parse(rawContent as string);
Time-Stamping:
} catch (error) {
console.error("Pixtral Error:", error);
return null; // Fallback auslösen
}
}
Nutze nicht nur created_at (Upload Zeit), sondern speichere explizit tasted_at.
3. Der "A/B-Switcher" (So nutzt du beides)
Warum? Wenn du 3 Stunden offline warst und dann online gehst, haben alle 5 Whiskys das gleiche created_at (Upload-Zeitpunkt). Du brauchst den Zeitpunkt, an dem der Button gedrückt wurde (lokale Handy-Zeit).
In deiner Haupt-Logik (app/actions/scan.ts) kannst du jetzt einfach umschalten oder Pixtral als Fallback nutzen, wenn Gemini zickt (oder andersrum).
TypeScript
Zusammenfassung für die Session-Logik:
'use server'
import { scanWithGemini } from './scan-gemini'; // Deine bestehende Funktion
import { scanWithPixtral } from './scan-mistral';
Das Datenmodell muss wissen:
export async function scanBottle(formData: FormData) {
// ... Bild zu Base64 konvertieren ...
const base64 = "...";
const mime = "image/jpeg";
session_start (Zeitstempel)
// STRATEGIE A: Der "Qualitäts-Check"
// Wir nutzen standardmäßig Gemini, aber Pixtral als EU-Option
session_end (Zeitstempel)
let result;
const useEuModel = process.env.USE_EU_MODEL === 'true'; // Schalter in .env
Innerhalb der Session: Relative Zeit ("Dram Nr. 3, +45min nach Start").
if (useEuModel) {
console.log("🇪🇺 Nutze Pixtral (Mistral AI)...");
result = await scanWithPixtral(base64, mime);
} else {
console.log("🇺🇸 Nutze Gemini 3 Flash...");
result = await scanWithGemini(base64, mime);
}
// Wenn das erste Modell versagt (null zurückgibt), versuche das andere
if (!result) {
console.log("⚠️ Erster Versuch fehlgeschlagen, starte Fallback...");
result = useEuModel
? await scanWithGemini(base64, mime)
: await scanWithPixtral(base64, mime);
}
// ... weiter mit Supabase Caching & Brave Search ...
return result;
}
Pixtral vs. Gemini 3 Flash (Dein Check)
Achte beim Testen auf diese Feinheiten:
Helle Schrift auf dunklem Grund: Hier ist Gemini oft aggressiver und liest besser. Pixtral ist manchmal vorsichtiger.
Schreibschrift (Signatory Vintage Abfüllungen): Pixtral Large ist hier erstaunlich gut, fast besser als Gemini, da es Handschrift extrem gut kann.
JSON-Struktur: Dank responseFormat: { type: 'json_object' } sollten beide Modelle sehr sauberen Code liefern.
Wenn Pixtral Large für dich ähnlich gut funktioniert wie Gemini 3 Flash, hast du den großen Vorteil: Daten bleiben in Europa (Server in Frankreich/EU). Das ist ein starkes Marketing-Argument ("We love Whisky & Privacy").

2
next-env.d.ts vendored
View File

@@ -1,6 +1,6 @@
/// <reference types="next" />
/// <reference types="next/image-types/global" />
import "./.next/dev/types/routes.d.ts";
import "./.next/types/routes.d.ts";
// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.

1
package.json
View File

@@ -13,6 +13,7 @@
},
"dependencies": {
"@google/generative-ai": "^0.24.1",
"@mistralai/mistralai": "^1.11.0",
"@supabase/ssr": "^0.5.2",
"@supabase/supabase-js": "^2.47.10",
"@tanstack/react-query": "^5.62.7",

20
pnpm-lock.yaml generated
View File

@@ -11,6 +11,9 @@ importers:
'@google/generative-ai':
specifier: ^0.24.1
version: 0.24.1
'@mistralai/mistralai':
specifier: ^1.11.0
version: 1.11.0
'@supabase/ssr':
specifier: ^0.5.2
version: 0.5.2(@supabase/supabase-js@2.88.0)
@@ -599,6 +602,9 @@ packages:
'@jridgewell/trace-mapping@0.3.31':
resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==}
'@mistralai/mistralai@1.11.0':
resolution: {integrity: sha512-6/BVj2mcaggYbpMzNSxtqtM2Tv/Jb5845XFd2CMYFO+O5VBkX70iLjtkBBTI4JFhh1l9vTCIMYXBVOjLoBVHGQ==}
'@napi-rs/wasm-runtime@0.2.12':
resolution: {integrity: sha512-ZVWUcfwY4E/yPitQJl481FjFo3K22D6qF0DuFH6Y/nbnE11GY5uguDxZMGXPQ8WQ0128MXQD7TnfHyK4oWoIJQ==}
@@ -2825,6 +2831,11 @@ packages:
resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==}
engines: {node: '>=10'}
zod-to-json-schema@3.25.0:
resolution: {integrity: sha512-HvWtU2UG41LALjajJrML6uQejQhNJx+JBO9IflpSja4R03iNWfKXrj6W2h7ljuLyc1nKS+9yDyL/9tD1U/yBnQ==}
peerDependencies:
zod: ^3.25 || ^4
zod-validation-error@4.0.2:
resolution: {integrity: sha512-Q6/nZLe6jxuU80qb/4uJ4t5v2VEZ44lzQjPDhYJNztRQ4wyWc6VF3D3Kb/fAuPetZQnhS3hnajCf9CsWesghLQ==}
engines: {node: '>=18.0.0'}
@@ -3242,6 +3253,11 @@ snapshots:
'@jridgewell/resolve-uri': 3.1.2
'@jridgewell/sourcemap-codec': 1.5.5
'@mistralai/mistralai@1.11.0':
dependencies:
zod: 3.25.76
zod-to-json-schema: 3.25.0(zod@3.25.76)
'@napi-rs/wasm-runtime@0.2.12':
dependencies:
'@emnapi/core': 1.7.1
@@ -5732,6 +5748,10 @@ snapshots:
yocto-queue@0.1.0: {}
zod-to-json-schema@3.25.0(zod@3.25.76):
dependencies:
zod: 3.25.76
zod-validation-error@4.0.2(zod@3.25.76):
dependencies:
zod: 3.25.76

12
src/components/CameraCapture.tsx
View File

@@ -65,7 +65,7 @@ export default function CameraCapture({ onImageCaptured, onAnalysisComplete, onS
const [isDiscovering, setIsDiscovering] = useState(false);
const [originalFile, setOriginalFile] = useState<File | null>(null);
const [isAdmin, setIsAdmin] = useState(false);
const [aiProvider, setAiProvider] = useState<'gemini' | 'nebius'>('gemini');
const [aiProvider, setAiProvider] = useState<'gemini' | 'pixtral'>('gemini');
React.useEffect(() => {
const checkAdmin = async () => {
@@ -200,7 +200,7 @@ export default function CameraCapture({ onImageCaptured, onAnalysisComplete, onS
setError(null);
try {
const { data: { user } } = await supabase.auth.getUser();
const { data: { user } = {} } = await supabase.auth.getUser();
if (!user) {
throw new Error(t('camera.authRequired'));
}
@@ -242,7 +242,7 @@ export default function CameraCapture({ onImageCaptured, onAnalysisComplete, onS
setError(null);
try {
const { data: { user } } = await supabase.auth.getUser();
const { data: { user } = {} } = await supabase.auth.getUser();
if (!user) {
throw new Error(t('camera.authRequired'));
}
@@ -364,10 +364,10 @@ export default function CameraCapture({ onImageCaptured, onAnalysisComplete, onS
Gemini
</button>
<button
onClick={() => setAiProvider('nebius')}
className={`px-3 py-1 text-[10px] font-black uppercase tracking-widest rounded-lg transition-all ${aiProvider === 'nebius' ? 'bg-white dark:bg-zinc-700 text-amber-600 shadow-sm' : 'text-zinc-400'}`}
onClick={() => setAiProvider('pixtral')}
className={`px-3 py-1 text-[10px] font-black uppercase tracking-widest rounded-lg transition-all ${aiProvider === 'pixtral' ? 'bg-white dark:bg-zinc-700 text-amber-600 shadow-sm' : 'text-zinc-400'}`}
>
Nebius
Pixtral 🇪🇺
</button>
</div>
)}

4
src/components/UploadQueue.tsx
View File

@@ -3,7 +3,7 @@
import React, { useEffect, useState, useCallback } from 'react';
import { useLiveQuery } from 'dexie-react-hooks';
import { db, PendingScan, PendingTasting } from '@/lib/db';
import { analyzeBottle } from '@/services/analyze-bottle';
import { magicScan } from '@/services/magic-scan';
import { saveBottle } from '@/services/save-bottle';
import { saveTasting } from '@/services/save-tasting';
import { createClient } from '@/lib/supabase/client';
@@ -41,7 +41,7 @@ export default function UploadQueue() {
const itemId = `scan-${item.id}`;
setCurrentProgress({ id: itemId, status: 'Analysiere Scan...' });
try {
const analysis = await analyzeBottle(item.imageBase64, undefined, item.locale);
const analysis = await magicScan(item.imageBase64, item.provider, item.locale);
if (analysis.success && analysis.data) {
const bottleData = analysis.data;
setCurrentProgress({ id: itemId, status: 'Speichere Flasche...' });

2
src/lib/db.ts
View File

@@ -5,7 +5,7 @@ export interface PendingScan {
temp_id: string; // Used to link tasting notes before sync
imageBase64: string;
timestamp: number;
provider?: 'gemini' | 'nebius';
provider?: 'gemini' | 'pixtral';
locale?: string;
}

89
src/services/analyze-bottle-nebius.ts → src/services/analyze-bottle-pixtral.ts
View File

@@ -1,22 +1,20 @@
'use server';
import { getNebiusClient } from '@/lib/ai-client';
import { SYSTEM_INSTRUCTION as GEMINI_SYSTEM_INSTRUCTION } from '@/lib/gemini';
import { Mistral } from '@mistralai/mistralai';
import { BottleMetadataSchema, AnalysisResponse, BottleMetadata } from '@/types/whisky';
import { createClient } from '@/lib/supabase/server';
import { createHash } from 'crypto';
import { trackApiUsage } from './track-api-usage';
import { checkCreditBalance, deductCredits } from './credit-service';
export async function analyzeBottleNebius(base64Image: string, tags?: string[], locale: string = 'de'): Promise<AnalysisResponse & { search_string?: string }> {
if (!process.env.NEBIUS_API_KEY) {
return { success: false, error: 'NEBIUS_API_KEY is not configured.' };
export async function analyzeBottlePixtral(base64Image: string, tags?: string[], locale: string = 'de'): Promise<AnalysisResponse & { search_string?: string }> {
if (!process.env.MISTRAL_API_KEY) {
return { success: false, error: 'MISTRAL_API_KEY is not configured.' };
}
let supabase;
try {
supabase = await createClient();
console.log('[analyzeBottleNebius] Initialized Supabase client');
const { data: { session } } = await supabase.auth.getSession();
if (!session || !session.user) {
return { success: false, error: 'Nicht autorisiert oder Session abgelaufen.' };
@@ -48,63 +46,70 @@ export async function analyzeBottleNebius(base64Image: string, tags?: string[],
};
}
const instruction = GEMINI_SYSTEM_INSTRUCTION
.replace('{AVAILABLE_TAGS}', tags ? tags.join(', ') : 'No tags available')
.replace('{LANGUAGE}', locale === 'en' ? 'English' : 'German')
+ "\nAdditionally, generate a 'search_string' field for Whiskybase in this format: 'site:whiskybase.com [Distillery] [Name] [Vintage]'. Include this field in the JSON object.";
console.log(`[analyzeBottleNebius] Instruction prepared for AI: ${instruction.substring(0, 100)}...`);
const client = new Mistral({ apiKey: process.env.MISTRAL_API_KEY });
const dataUrl = `data:image/jpeg;base64,${base64Data}`;
const prompt = `Du bist ein Whisky-Experte und OCR-Spezialist.
Analysiere dieses Etikett präzise.
Sprache für Beschreibungen: ${locale === 'en' ? 'Englisch' : 'Deutsch'}.
Verfügbare Tags zur Einordnung: ${tags ? tags.join(', ') : 'Keine Tags verfügbar'}.
const client = getNebiusClient();
const response = await client.chat.completions.create({
model: "Qwen/Qwen2.5-VL-72B-Instruct",
Antworte AUSSCHLIESSLICH mit gültigem JSON (kein Markdown, kein Text davor/danach):
{
"distillery": "Name der Brennerei (z.B. Lagavulin)",
"name": "Exakter Name/Edition (z.B. 16 Year Old)",
"vintage": "Jahrgang als Zahl oder null",
"age": "Alter als Zahl oder null (z.B. 16)",
"abv": "Alkoholgehalt als Zahl ohne % (z.B. 43)",
"category": "Kategorie (z.B. Single Malt Scotch Whisky)",
"search_string": "site:whiskybase.com [Brennerei] [Name] [Alter]"
}`;
const chatResponse = await client.chat.complete({
model: 'pixtral-large-latest',
messages: [
{
role: "system",
content: instruction
},
{
role: "user",
role: 'user',
content: [
{
type: "text",
text: "Extract whisky metadata from this image."
},
{
type: "image_url",
image_url: {
url: `data:image/jpeg;base64,${base64Data}`
}
}
{ type: 'text', text: prompt },
{ type: 'image_url', imageUrl: dataUrl }
]
}
],
response_format: { type: "json_object" }
responseFormat: { type: 'json_object' },
temperature: 0.1
});
const content = response.choices[0].message.content;
if (!content) throw new Error('Empty response from Nebius AI');
const rawContent = chatResponse.choices?.[0].message.content;
if (!rawContent) throw new Error("Keine Antwort von Pixtral");
// Extract JSON content in case the model wraps it in markdown blocks
const jsonContent = content.match(/\{[\s\S]*\}/)?.[0] || content;
const jsonData = JSON.parse(jsonContent);
const jsonData = JSON.parse(rawContent as string);
// Extract search_string before validation if it's not in schema
// Extract search_string before validation
const searchString = jsonData.search_string;
delete jsonData.search_string;
// Ensure abv is a number if it came as a string
if (typeof jsonData.abv === 'string') {
jsonData.abv = parseFloat(jsonData.abv.replace('%', '').trim());
}
// Ensure age/vintage are numbers
if (jsonData.age) jsonData.age = parseInt(jsonData.age);
if (jsonData.vintage) jsonData.vintage = parseInt(jsonData.vintage);
const validatedData = BottleMetadataSchema.parse(jsonData);
// Track usage
await trackApiUsage({
userId: userId,
apiType: 'gemini_ai', // Keep tracking as gemini_ai for budget or separate later
endpoint: 'nebius/qwen2.5-vl',
apiType: 'gemini_ai', // Keep as generic 'gemini_ai' for now or update schema later
endpoint: 'mistral/pixtral-large',
success: true
});
// Deduct credits
await deductCredits(userId, 'gemini_ai', 'Nebius AI analysis');
await deductCredits(userId, 'gemini_ai', 'Pixtral AI analysis');
// Store in Cache
await supabase
@@ -118,7 +123,7 @@ export async function analyzeBottleNebius(base64Image: string, tags?: string[],
};
} catch (error) {
console.error('Nebius Analysis Error:', error);
console.error('Pixtral Analysis Error:', error);
// Track failed API call
try {
@@ -128,7 +133,7 @@ export async function analyzeBottleNebius(base64Image: string, tags?: string[],
await trackApiUsage({
userId: session.user.id,
apiType: 'gemini_ai',
endpoint: 'nebius/qwen2.5-vl',
endpoint: 'mistral/pixtral-large',
success: false,
errorMessage: error instanceof Error ? error.message : 'Unknown error'
});
@@ -140,7 +145,7 @@ export async function analyzeBottleNebius(base64Image: string, tags?: string[],
return {
success: false,
error: error instanceof Error ? error.message : 'Nebius AI analysis failed.',
error: error instanceof Error ? error.message : 'Pixtral AI analysis failed.',
};
}
}

8
src/services/magic-scan.ts
View File

@@ -1,14 +1,14 @@
'use server';
import { analyzeBottle } from './analyze-bottle';
import { analyzeBottleNebius } from './analyze-bottle-nebius';
import { analyzeBottlePixtral } from './analyze-bottle-pixtral';
import { searchBraveForWhiskybase } from './brave-search';
import { getAllSystemTags } from './tags';
import { supabase } from '@/lib/supabase';
import { supabaseAdmin } from '@/lib/supabase-admin';
import { AnalysisResponse, BottleMetadata } from '@/types/whisky';
export async function magicScan(base64Image: string, provider: 'gemini' | 'nebius' = 'gemini', locale: string = 'de'): Promise<AnalysisResponse & { wb_id?: string }> {
export async function magicScan(base64Image: string, provider: 'gemini' | 'pixtral' = 'gemini', locale: string = 'de'): Promise<AnalysisResponse & { wb_id?: string }> {
try {
console.log('[magicScan] Starting scan process...');
if (!supabase) {
@@ -22,8 +22,8 @@ export async function magicScan(base64Image: string, provider: 'gemini' | 'nebiu
// 1. AI Analysis
let aiResponse: any;
if (provider === 'nebius') {
aiResponse = await analyzeBottleNebius(base64Image, tagNames, locale);
if (provider === 'pixtral') {
aiResponse = await analyzeBottlePixtral(base64Image, tagNames, locale);
} else {
aiResponse = await analyzeBottle(base64Image, tagNames, locale);
}

2
tsconfig.tsbuildinfo
View File

File diff suppressed because one or more lines are too long