robin/Dramlog-Prod
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement AI custom tag proposals

Browse Source 
- AI now suggests dominant notes not in the system list (Part 3: Custom Suggestions)
- Updated TagSelector to show 'Neu anlegen?' buttons for AI-proposed custom tags
- Added suggested_custom_tags to bottles table and metadata schema
- Updated TastingNoteForm to handle both system and custom AI suggestions
This commit is contained in:
robin
2025-12-19 13:20:13 +01:00
parent b2a1d292da
commit 74916aec73
12 changed files with 216 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
.aiideas Normal file
View File

@@ -0,0 +1,69 @@
1. Feature: Tasting Notes Auto-Fill (Die "Tag-Matching" Strategie)
Du hast völlig recht: Wenn Gemini einfach wild Text generiert (z.B. "Grüner Granny Smith Apfel"), und deine Datenbank nur "Apfel" kennt, hast du Chaos.
Die Lösung: "Constrained Generation" (Gezwungene Auswahl)
Du fütterst Gemini nicht nur mit dem Bild, sondern auch mit deiner Master-Liste an Tags im Prompt.
Der Workflow:
Input: Bild vom Label + Deine Liste der System Tags (als JSON-String).
Frontend:
Die App empfängt die IDs.
In der UI werden diese Tags aktiviert/vorausgewählt angezeigt (z.B. farbig hinterlegt).
Der User sieht: "Vorschlag: Rauch, Vanille".
Wichtig: Der User kann sie abwählen (wenn er es nicht schmeckt) oder andere aus der Liste hinzufügen.
Das ist der "Sweet Spot". Wir kombinieren die harte Fakten-Extraktion (Metadata) mit der "halluzinierten" aber kontrollierten Sensorik (Tags).
Hier ist dein "Master Prompt", der beides erledigt.
Das Konzept der "Constrained Generation"
Wichtig: Damit Gemini nicht irgendwelche Wörter erfindet, müssen wir ihm deine Tag-Liste im Prompt mitgeben. Ich habe im Prompt einen Platzhalter {AVAILABLE_TAGS_JSON} eingefügt. Diesen musst du in deinem Code (Next.js API Route oder Edge Function) mit deiner echten Tag-Liste ersetzen, bevor du den String an Gemini schickst.
Der Prompt (Copy & Paste)
You are a master sommelier and strict database clerk.
Your task is to analyze the whisky bottle image provided.
PART 1: METADATA EXTRACTION
Extract precise metadata from the visible label text.
- If the image is NOT a whisky bottle or if you are very unsure, set "is_whisky" to false and provide a low "confidence" score.
- If a value is not visible, use null.
- Infer the 'Category' (e.g., Islay Single Malt, Bourbon, Rye) based on the Distillery if possible.
- Search specifically for a "Whiskybase ID" or "WB ID" on the label (often handwritten or small print).
- Search for "Bottle Codes" (Laser codes often on the glass).
PART 2: SENSORY ANALYSIS (AUTO-FILL)
Based on the identified bottle (using your internal knowledge about this specific release/distillery), select the most appropriate flavor tags.
CONSTRAINT: You must ONLY select tags from the following provided list. Do NOT invent new tags.
If you recognize the whisky, try to select 3-6 tags that best describe its character.
AVAILABLE TAGS LIST:
{AVAILABLE_TAGS_JSON}
PART 3: OUTPUT
Output strictly raw JSON matching the following schema (no markdown, no code blocks):
{
"name": string | null,
"distillery": string | null,
"category": string | null,
"abv": number | null,
"age": number | null,
"vintage": string | null,
"bottleCode": string | null,
"whiskybaseId": string | null,
"distilled_at": string | null,
"bottled_at": string | null,
"batch_info": string | null,
"is_whisky": boolean,
"confidence": number,
"suggested_tags": string[]
}

2
add_custom_tags_to_bottles.sql Normal file
View File

@@ -0,0 +1,2 @@
-- Add suggested_custom_tags to bottles table
ALTER TABLE bottles ADD COLUMN IF NOT EXISTS suggested_custom_tags text[];

2
add_suggested_tags_to_bottles.sql Normal file
View File

@@ -0,0 +1,2 @@
-- Add suggested_tags to bottles table
ALTER TABLE bottles ADD COLUMN IF NOT EXISTS suggested_tags text[];

64
src/components/TagSelector.tsx
View File

@@ -10,14 +10,17 @@ interface TagSelectorProps {
selectedTagIds: string[];
onToggleTag: (tagId: string) => void;
label?: string;
suggestedTagNames?: string[];
suggestedCustomTagNames?: string[];
}
export default function TagSelector({ category, selectedTagIds, onToggleTag, label }: TagSelectorProps) {
export default function TagSelector({ category, selectedTagIds, onToggleTag, label, suggestedTagNames, suggestedCustomTagNames }: TagSelectorProps) {
const { t } = useI18n();
const [tags, setTags] = useState<Tag[]>([]);
const [search, setSearch] = useState('');
const [isLoading, setIsLoading] = useState(true);
const [isCreating, setIsCreating] = useState(false);
const [creatingSuggestion, setCreatingSuggestion] = useState<string | null>(null);
useEffect(() => {
const fetchTags = async () => {
@@ -128,11 +131,68 @@ export default function TagSelector({ category, selectedTagIds, onToggleTag, lab
)}
</div>
{/* AI Suggestions */}
{!search && suggestedTagNames && suggestedTagNames.length > 0 && (
<div className="space-y-2">
<div className="flex items-center gap-1.5 text-[9px] font-black uppercase tracking-widest text-amber-500">
<Sparkles size={10} /> {t('camera.wbMatchFound') ? 'KI Vorschläge' : 'AI Suggestions'}
</div>
<div className="flex flex-wrap gap-1.5">
{tags
.filter(t => !selectedTagIds.includes(t.id) && suggestedTagNames.some((s: string) => s.toLowerCase() === t.name.toLowerCase()))
.map(tag => (
<button
key={tag.id}
type="button"
onClick={() => onToggleTag(tag.id)}
className="px-2.5 py-1 rounded-lg bg-amber-50 dark:bg-amber-900/20 text-amber-600 dark:text-amber-400 text-[10px] font-bold uppercase tracking-tight hover:bg-amber-100 dark:hover:bg-amber-900/30 transition-colors border border-amber-200 dark:border-amber-800/50 flex items-center gap-1.5"
>
<Sparkles size={10} />
{tag.is_system_default ? t(`aroma.${tag.name}`) : tag.name}
</button>
))}
</div>
</div>
)}
{/* AI Custom Suggestions */}
{!search && suggestedCustomTagNames && suggestedCustomTagNames.length > 0 && (
<div className="space-y-2">
<div className="flex items-center gap-1.5 text-[9px] font-black uppercase tracking-widest text-zinc-400">
Dominante Note anlegen?
</div>
<div className="flex flex-wrap gap-1.5">
{suggestedCustomTagNames
.filter(name => !tags.some(t => t.name.toLowerCase() === name.toLowerCase()))
.map(name => (
<button
key={name}
type="button"
disabled={creatingSuggestion === name}
onClick={async () => {
setCreatingSuggestion(name);
const result = await createCustomTag(name, category);
if (result.success && result.tag) {
setTags(prev => [...prev, result.tag!]);
onToggleTag(result.tag!.id);
}
setCreatingSuggestion(null);
}}
className="px-2.5 py-1 rounded-lg bg-zinc-50 dark:bg-zinc-800/50 text-zinc-500 dark:text-zinc-400 text-[10px] font-bold uppercase tracking-tight hover:bg-amber-600 hover:text-white transition-all border border-dashed border-zinc-200 dark:border-zinc-700/50 flex items-center gap-1.5 disabled:opacity-50"
>
{creatingSuggestion === name ? <Loader2 size={10} className="animate-spin" /> : <Plus size={10} />}
{name}
</button>
))}
</div>
</div>
)}
{/* Suggestions Chips (limit to 6 random or most common) */}
{!search && tags.length > 0 && (
<div className="flex flex-wrap gap-1.5 pt-1">
{tags
.filter(t => !selectedTagIds.includes(t.id))
.filter(t => !selectedTagIds.includes(t.id) && (!suggestedTagNames || !suggestedTagNames.some((s: string) => s.toLowerCase() === t.name.toLowerCase())))
.slice(0, 8)
.map(tag => (
<button

22
src/components/TastingNoteForm.tsx
View File

@@ -33,6 +33,7 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
const [noseTagIds, setNoseTagIds] = useState<string[]>([]);
const [palateTagIds, setPalateTagIds] = useState<string[]>([]);
const [finishTagIds, setFinishTagIds] = useState<string[]>([]);
const [suggestedTags, setSuggestedTags] = useState<string[]>([]);
const { activeSession } = useSession();
const effectiveSessionId = sessionId || activeSession?.id;
@@ -43,6 +44,17 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
const { data: buddiesData } = await supabase.from('buddies').select('id, name').order('name');
setBuddies(buddiesData || []);
// Fetch Bottle Suggestions
const { data: bottleData } = await supabase
.from('bottles')
.select('suggested_tags')
.eq('id', bottleId)
.maybeSingle();
if (bottleData?.suggested_tags) {
setSuggestedTags(bottleData.suggested_tags);
}
// If Session ID, fetch session participants and pre-select them
if (effectiveSessionId) {
const { data: participants } = await supabase
@@ -58,7 +70,7 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
}
};
fetchData();
}, [effectiveSessionId]);
}, [effectiveSessionId, bottleId]);
const toggleBuddy = (id: string) => {
setSelectedBuddyIds(prev =>
@@ -185,6 +197,8 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
selectedTagIds={noseTagIds}
onToggleTag={toggleNoseTag}
label={t('tasting.nose')}
suggestedTagNames={suggestedTags}
suggestedCustomTagNames={suggestedCustomTags}
/>
<div className="space-y-2">
<label className="text-[10px] font-black text-zinc-400 uppercase tracking-widest block opacity-50">Zusätzliche Notizen</label>
@@ -204,6 +218,8 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
selectedTagIds={palateTagIds}
onToggleTag={togglePalateTag}
label={t('tasting.palate')}
suggestedTagNames={suggestedTags}
suggestedCustomTagNames={suggestedCustomTags}
/>
<div className="space-y-2">
<label className="text-[10px] font-black text-zinc-400 uppercase tracking-widest block opacity-50">Zusätzliche Notizen</label>
@@ -223,6 +239,8 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
selectedTagIds={finishTagIds}
onToggleTag={toggleFinishTag}
label={t('tasting.finish')}
suggestedTagNames={suggestedTags}
suggestedCustomTagNames={suggestedCustomTags}
/>
<TagSelector
@@ -230,6 +248,8 @@ export default function TastingNoteForm({ bottleId, sessionId }: TastingNoteForm
selectedTagIds={finishTagIds} // Using finish state for texture for now, or separate if needed
onToggleTag={toggleFinishTag}
label="Textur & Mundgefühl"
suggestedTagNames={suggestedTags}
suggestedCustomTagNames={suggestedCustomTags}
/>
<div className="space-y-2">

30
src/lib/gemini.ts
View File

@@ -12,12 +12,24 @@ export const geminiModel = genAI.getGenerativeModel({
});
export const SYSTEM_INSTRUCTION = `
You are a sommelier and database clerk. Analyze the whisky bottle image. Extract precise metadata.
If the image is NOT a whisky bottle or if you are very unsure, set "is_whisky" to false and provide a low "confidence" score.
You are a sommelier and database clerk. Analyze the whisky bottle image.
PART 1: METADATA EXTRACTION
Extract precise metadata. If the image is NOT a whisky bottle or if you are very unsure, set "is_whisky" to false and provide a low "confidence" score.
If a value is not visible, use null.
Infer the 'Category' (e.g., Islay Single Malt) based on the Distillery if possible.
Search specifically for a "Whiskybase ID" or "WB ID" on the label.
Output raw JSON matching the following schema:
Search specifically for a "Whiskybase ID" or "WB ID" on the label.
PART 2: SENSORY ANALYSIS (AUTO-FILL)
Based on the identified bottle, select the most appropriate flavor tags.
CONSTRAINT: You must ONLY select tags from the following provided list. Do NOT invent new tags in this field.
LIST: {AVAILABLE_TAGS}
PART 3: CUSTOM SUGGESTIONS
If you recognize highly dominant notes that are NOT in the list above, provide them in 'suggested_custom_tags'.
Limit this to 1-2 very unique notes (e.g. "Marshmallow" or "Balsamico"). Do not repeat tags from the system list.
Output strictly raw JSON matching the following schema:
{
"name": string | null,
"distillery": string | null,
@@ -27,10 +39,12 @@ Output raw JSON matching the following schema:
"vintage": string | null,
"bottleCode": string | null,
"whiskybaseId": string | null,
"distilled_at": string | null (e.g. "2010" or "12.05.2010"),
"bottled_at": string | null (e.g. "2022" or "15.11.2022"),
"batch_info": string | null (e.g. "Batch 1" or "L12.03.2022"),
"distilled_at": string | null,
"bottled_at": string | null,
"batch_info": string | null,
"is_whisky": boolean,
"confidence": number (0-100)
"confidence": number (0-100),
"suggested_tags": string[] (from provided list),
"suggested_custom_tags": string[] (new unique notes)
}
`;

9
src/services/analyze-bottle-nebius.ts
View File

@@ -9,7 +9,7 @@ import { createHash } from 'crypto';
import { trackApiUsage } from './track-api-usage';
import { checkCreditBalance, deductCredits } from './credit-service';
export async function analyzeBottleNebius(base64Image: string): Promise<AnalysisResponse & { search_string?: string }> {
export async function analyzeBottleNebius(base64Image: string, tags?: string[]): Promise<AnalysisResponse & { search_string?: string }> {
const supabase = createServerActionClient({ cookies });
if (!process.env.NEBIUS_API_KEY) {
@@ -24,7 +24,6 @@ export async function analyzeBottleNebius(base64Image: string): Promise<Analysis
const userId = session.user.id;
// Check credit balance (using same gemini_ai type for now or create new one)
const creditCheck = await checkCreditBalance(userId, 'gemini_ai');
if (!creditCheck.allowed) {
return {
@@ -36,7 +35,6 @@ export async function analyzeBottleNebius(base64Image: string): Promise<Analysis
const base64Data = base64Image.split(',')[1] || base64Image;
const imageHash = createHash('sha256').update(base64Data).digest('hex');
// Check Cache (Optional: skip if you want fresh AI results for testing)
const { data: cachedResult } = await supabase
.from('vision_cache')
.select('result')
@@ -44,21 +42,20 @@ export async function analyzeBottleNebius(base64Image: string): Promise<Analysis
.maybeSingle();
if (cachedResult) {
console.log(`[Nebius Cache] Hit! hash: ${imageHash}`);
return {
success: true,
data: cachedResult.result as any,
};
}
console.log(`[Nebius AI] Calling Qwen2.5-VL...`);
const instruction = GEMINI_SYSTEM_INSTRUCTION.replace('{AVAILABLE_TAGS}', tags ? tags.join(', ') : 'No tags available') + "\nAdditionally, generate a 'search_string' field for Whiskybase in this format: 'site:whiskybase.com [Distillery] [Name] [Vintage]'. Include this field in the JSON object.";
const response = await aiClient.chat.completions.create({
model: "Qwen/Qwen2.5-VL-72B-Instruct",
messages: [
{
role: "system",
content: GEMINI_SYSTEM_INSTRUCTION + "\nAdditionally, generate a 'search_string' field for Whiskybase in this format: 'site:whiskybase.com [Distillery] [Name] [Vintage]'. Include this field in the JSON object."
content: instruction
},
{
role: "user",

14
src/services/analyze-bottle.ts
View File

@@ -8,7 +8,7 @@ import { createHash } from 'crypto';
import { trackApiUsage } from './track-api-usage';
import { checkCreditBalance, deductCredits } from './credit-service';
export async function analyzeBottle(base64Image: string): Promise<AnalysisResponse> {
export async function analyzeBottle(base64Image: string, tags?: string[]): Promise<AnalysisResponse> {
const supabase = createServerActionClient({ cookies });
if (!process.env.GEMINI_API_KEY) {
@@ -16,7 +16,7 @@ export async function analyzeBottle(base64Image: string): Promise<AnalysisRespon
}
try {
// Ensure user is authenticated for tracking/billing
// ... (auth and credit check remain same) ...
const { data: { session } } = await supabase.auth.getSession();
if (!session || !session.user) {
return { success: false, error: 'Nicht autorisiert oder Session abgelaufen.' };
@@ -24,7 +24,6 @@ export async function analyzeBottle(base64Image: string): Promise<AnalysisRespon
const userId = session.user.id;
// Check credit balance before making API call
const creditCheck = await checkCreditBalance(userId, 'gemini_ai');
if (!creditCheck.allowed) {
return {
@@ -33,12 +32,9 @@ export async function analyzeBottle(base64Image: string): Promise<AnalysisRespon
};
}
// 1. Generate Hash for Caching
const base64Data = base64Image.split(',')[1] || base64Image;
const imageHash = createHash('sha256').update(base64Data).digest('hex');
console.log(`[AI Cache] Checking hash: ${imageHash}`);
// 2. Check Cache
const { data: cachedResult } = await supabase
.from('vision_cache')
.select('result')
@@ -46,16 +42,14 @@ export async function analyzeBottle(base64Image: string): Promise<AnalysisRespon
.maybeSingle();
if (cachedResult) {
console.log(`[AI Cache] Hit! hash: ${imageHash}`);
return {
success: true,
data: cachedResult.result as any,
};
}
console.log(`[AI Cache] Miss. Calling Gemini...`);
const instruction = SYSTEM_INSTRUCTION.replace('{AVAILABLE_TAGS}', tags ? tags.join(', ') : 'No tags available');
// 3. AI Analysis
const result = await geminiModel.generateContent([
{
inlineData: {
@@ -63,7 +57,7 @@ export async function analyzeBottle(base64Image: string): Promise<AnalysisRespon
mimeType: 'image/jpeg',
},
},
{ text: SYSTEM_INSTRUCTION },
{ text: instruction },
]);
const responseText = result.response.text();

9
src/services/magic-scan.ts
View File

@@ -3,18 +3,23 @@
import { analyzeBottle } from './analyze-bottle';
import { analyzeBottleNebius } from './analyze-bottle-nebius';
import { searchBraveForWhiskybase } from './brave-search';
import { getAllSystemTags } from './tags';
import { supabase } from '@/lib/supabase';
import { supabaseAdmin } from '@/lib/supabase-admin';
import { AnalysisResponse, BottleMetadata } from '@/types/whisky';
export async function magicScan(base64Image: string, provider: 'gemini' | 'nebius' = 'gemini'): Promise<AnalysisResponse & { wb_id?: string }> {
try {
// 0. Fetch available tags for constrained generation
const systemTags = await getAllSystemTags();
const tagNames = systemTags.map(t => t.name);
// 1. AI Analysis
let aiResponse: any;
if (provider === 'nebius') {
aiResponse = await analyzeBottleNebius(base64Image);
aiResponse = await analyzeBottleNebius(base64Image, tagNames);
} else {
aiResponse = await analyzeBottle(base64Image);
aiResponse = await analyzeBottle(base64Image, tagNames);
}
if (!aiResponse.success || !aiResponse.data) {

2
src/services/save-bottle.ts
View File

@@ -67,6 +67,8 @@ export async function saveBottle(
distilled_at: metadata.distilled_at,
bottled_at: metadata.bottled_at,
batch_info: metadata.batch_info,
suggested_tags: metadata.suggested_tags,
suggested_custom_tags: metadata.suggested_custom_tags,
})
.select()
.single();

20
src/services/tags.ts
View File

@@ -35,6 +35,26 @@ export async function getTagsByCategory(category: TagCategory): Promise<Tag[]> {
return data || [];
}
/**
* Fetch all system default tags
*/
export async function getAllSystemTags(): Promise<Tag[]> {
const supabase = createServerActionClient({ cookies });
const { data, error } = await supabase
.from('tags')
.select('*')
.eq('is_system_default', true)
.order('name');
if (error) {
console.error('Error fetching all system tags:', error);
return [];
}
return data || [];
}
/**
* Create a custom user tag
*/

2
src/types/whisky.ts
View File

@@ -14,6 +14,8 @@ export const BottleMetadataSchema = z.object({
batch_info: z.string().nullish(),
is_whisky: z.boolean().default(true),
confidence: z.number().min(0).max(100).default(100),
suggested_tags: z.array(z.string()).nullish(),
suggested_custom_tags: z.array(z.string()).nullish(),
});
export type BottleMetadata = z.infer<typeof BottleMetadataSchema>;