How does it work under the hood?

[tleyden]

Awesome project!

It would be great if you could add some notes to the readme (or other wiki page) on how it works under the hood. For example here are a few questions I had:

• Does it use OCR or a multimodal vision model?
• Which model is used?
• Which apple silicon apis does it use?
• Does it depend on any other repos such as GGML?
• Were any optimizations done for reducing power consumption?

[tleyden]

I read through the code, here's what I learned:

• Does it use OCR or a multimodal vision model? Just OCR
• Which model is used? ImageAnalyzer text mode - built into OSX
• Which apple silicon apis does it use? None directly, but presumably ImageAnalyzer can leverage apple silicon
• Does it depend on any other repos such as GGML? No
• Were any optimizations done for reducing power consumption? WIP

The main capture loop is something like this:

1. Grab screenshot
2. Extract text via ImageAnalyzer
3. Add text to sqlite in a full-text indexed column, which includes the rest of the metadata about the captured image + timestamp info
4. Save the image to disk, while compressing with ffmpeg

When searching from the UI, it will do a fulltext search in sqlite for matching keywords. Or the database of captured screenshots can be browsed chronologically.

[jasonjmcghee]

Now it’s changed a bit- using the old ocr api, which image analyzer uses under the hood. Optionally in “fast mode”. Also optionally only ocr the current window. These together reduce energy usage a fair amount.

another piece not mentioned here is search and timeline.

search and timeline both use AVFoundation to extract a thumbnail from the mp4 for display. Timeline also runs ImageAnalyzer with overlay, so users can select text on the timeline.

search uses full text search to match, optionally filtering by application name.

[tleyden]

Thanks for the explanation!

so users can select text on the timeline.

Ah I didn't know about that. For copy/paste purposes?

One question on the Timeline view, are there any controls, or can the user just scroll through with gestures? I find that screen a bit hard to use.. like I don't know where I am in the timeline as I scroll through, and it's not visually differentiated from my desktop.

[jasonjmcghee]

That’d be a great enhancement. Currently just scrolling. It’d be great to add a ui to control without gestures / see where you are / annotate the timeline etc

[jasonjmcghee]

Mutable generated a wiki for us https://wiki.mutable.ai/jasonjmcghee/rem