Project
auto-mem
A private, on-device memory engine that makes a backlog of saved screenshots and reels searchable, and works out why each one was saved.
- 0.697
- MRR, hard retrieval
- 0.603
- R@1
- 0.045
- keyword baseline MRR
A local-first multimodal memory engine for Apple Silicon. It turns ~8k saved social-media screenshots and screen recordings into a searchable, temporally linked knowledge base: on-device OCR, speech and music separation with multilingual transcription, structured extraction with a vision-language model, and shared text-and-image embeddings over a vector store and a temporal graph. No content leaves the machine.
Decisions
- Every model choice was made on an audited benchmark before any bulk run. Each target hides in a pool of 500 documents among look-alikes and random distractors, behind 194 adversarial queries written to avoid the item’s own words.
- Multimodal embeddings plus a reranker reach MRR 0.697 and R@1 0.603, against 0.045 for keyword search over the OCR text.
- A tie-handling bug had been ranking the keyword baseline’s target first among ties. Fixing it dropped that baseline from 0.076 to 0.045: the bug flattered the comparison it was meant to be fair to.
- The larger embedding model beats the smaller one by +0.055 MRR, with a paired bootstrap confidence interval that only just clears zero, reported as such.
- Model weights load strictly. A silently random language-ID head is worse than none.
Where it falls short
- Ingestion into the graph and vector stores, the API and the UI are not built yet.
- The speech-recognition comparison has no ground truth, so it measures agreement, not accuracy.