> Canonical URL: https://codingcat.dev/post/firebase-2025-10-privacy-first-on-device-ai # Smart photo organization through privacy-first, on-device AI in Firebase: Web Architecture & Implementation Guide ![CodingCat.dev tutorial cover image](https://cdn.sanity.io/images/hfh83o0w/production/45ff770d5d01819c47dd48ea72207cc4f09a01df-1200x630.svg?w=1200&q=82&fit=max&auto=format)[![Alex Patterson](https://cdn.sanity.io/images/hfh83o0w/production/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)October 16, 2025 Calling frontier multimodal models directly from a web or mobile client is a security minefield if you expose raw API keys or hardcode brittle system prompts in your frontend bundle. Following the announcement of **Smart photo organization through privacy-first, on-device AI** by Cynthia Wang on the [**Official Firebase Blog**](https://firebase.blog/posts/2025/10/privacy-first-on-device-ai/) (2025-10-15), here is our hands-on CodingCat.dev breakdown of what actually shipped, how it works under the hood, and how we wire it into production TypeScript, Next.js, Astro, SvelteKit, React, and Angular applications. User privacy can be a big concern when you integrate AI into your app, especially when your app lets users upload personal data, like photos to a cloud server for processing. To show how you can safeguard user privacy while building with AI, we developed a progressive web app (PWA) that brings AI to the photos instead of the other way around. By running the model directly in the browser, we can ensure that it won’t compromise the privacy of the users. ## The Mental Model: Why Smart photo organization through privacy-first, on-device AI Matters for Web Devs As a Google Developer Expert (GDE) for Firebase and Senior Solutions Engineer at CodingCat.dev, I evaluate every Firebase update through one lens: does it reduce boilerplate while keeping our runtime security and cloud bills predictable? Firebase AI Logic lets you invoke Gemini and Imagen models from client and edge apps using the Firebase Web SDK while keeping your API credentials off the wire, enforcing Firebase App Check, and managing server-side prompt templates centrally. > AJ Pro-Tip from Alex: Combine Firebase AI Logic with server prompt templates, Firebase Remote Config feature flags, and project spend caps so you can iterate on prompts or swap Gemini models instantly without redeploying your Next.js or Astro frontend. ## How It Works: Technical Breakdown & Implementation Here’s a look under the hood to see how we approached this. ### Design considerations When handling personal photos, security and trust is everything. Thus our entire design philosophy revolved around ensuring the privacy of user data, which led to a few key technical decisions. Normally when building an AI feature, we default to the powerful cloud-based model like the Gemini Pro model. This typically involves sending data, in our case photos, to a data center or server for processing. While this model is capable of more complex reasoning, our main focus is on maximizing user privacy and control In your app. Since we are handling lots of personal photos, we want to consider that users may feel more comfortable having their photos not leave their device. That is why we chose to use a local model that is running in the browser. Instead of sending photos to the cloud, all AI analysis happens directly in the user’s machine with no data leaving the device and even allowing it to work offline. To complement this, we built the app as an installable progressive web app (PWA). Being an offline-first approach, this means that the user can launch the app without internet connection to reinforce the convenience and privacy of this app. For on-device processing, we opted for a model that is integrated directly into the web browser. This approach had some key advantages. Instead of having to load a large model, we simply need access to the model downloaded by the browser. This is where the Firebase AI Logic SDKs come in, it would be able to detect if a compatible on-device model like Gemini Nano is available, and if not, request the browser to download and cache the model for future use. The nice thing is that once the on-device model is downloaded by the browser, all web apps that want to use that model now have access to it. Of course, not all browsers or devices have a built-in AI model. For users on other platforms, we needed a reliable fallback. This is where hybrid AI inference comes in, You can use an on device model when available but fallback to a cloud model as needed. If an on-device model isn’t available, the SDKs can switch to a cloud based model, in our case `gemini-2.5-flash-lite`. Though many models would have worked in our case, we found that this model met all of our requirements: Flash-lite provided good results for both metadata generation and sorting based on the structured outputs, with the additional plus of being cost-effective. However, we thought that we should make the switch transparent to the user. Thus, before sending any data, we made a pop up modal to ask for permission to process their photos in the cloud. Here is a quick code snippet on how we did this in our react app: ``` useEffect const // Checking availability of the on-device model const if console. setShowNanoAlert } console. } }; checkNanoAvailability }, []); <> {showNanoAlert ``` ### The AI portion With building AI apps, how one asks a model to perform the task is probably just as important as the code itself, here is how we decided to approach it. For the app to organize our photos, it had to be able to see our photos. The metadata generation prompt is multimodal, meaning it combines both image (or some other media) and text. The text portion also meant that the user could provide specific instructions to guide the analysis, like “focus on the architecture”, “identify on the seasons” to get more relevant and useful categorization. To instantiate our model to accept multimodal inputs, we needed to set the createOptions in our getGenerativeModel call: ``` export mode: InferenceMode.PREFER_ON_DEVICE, inCloudParams: { model: onDeviceParams: { promptOptions: { responseConstraint: metadataSchema }, createOptions: { expectedInputs: [{type: expectedOutputs: [{type: }); ``` To use the AI’s output In your application, we need it to be clean and predictable. We enforce a strict JSON schema that the model’s response must follow. This ensures we always get back a parsable object with a description, an array of categories, and its dominant colors, eliminating the need for complex string parsing increasing the accuracy of the response. Here is your schema that we used for structuring the output: ``` import const properties: { description: Schema. categories: Schema. description: items: Schema. }), dominant_colors: Schema. description: items: Schema. }), }, required: [ }); const properties: { sorted_groups: Schema. description: items: Schema. properties: { group_name: Schema. images: Schema. }, required: [ }), }), }, required: [ }); ``` your app has two main tasks, so we created two distinct prompts: This prompt runs sequentially for each individual image. It takes the image and an optional user query, and returns a JSON object with a description, an array of categories, and dominant color hex codes. After all images have been analyzed, this second prompt takes the entire collection of generated metadata and a user’s sorting preference (e.g., “by event,” “by color”). It then returns a single JSON object with the images organized into logical groups. This separation makes the process more efficient and the user experience more responsive, as metadata for each image appears as soon as it’s ready. ### The code Here’s how we put everything together using the Firebase AI Logic SDK. First, we initialize the Firebase AI SDK. We then configure two different model instances based on our two-prompt strategy. Notice the mode: `InferenceMode.PREFER_ON_DEVICE` setting, this tells the SDK to handle the on-device-first logic with a cloud fallback, enabling the hybrid approach. We define both the cloud model (`inCloudParams`) and the on-device configuration (`onDeviceParams`), including the required JSON schemas for structured output. ``` import const const // Model for generating metadata for a single image export mode: InferenceMode.PREFER_ON_DEVICE, inCloudParams: { model: onDeviceParams: { promptOptions: { responseConstraint: metadataSchema }, createOptions: { expectedInputs: [{type: expectedOutputs: [{type: }); // Model for sorting a collection of image metadata const mode: InferenceMode.PREFER_ON_DEVICE, inCloudParams: { model: onDeviceParams: { promptOptions: { responseConstraint: sortingSchema }, createOptions: { expectedInputs: [{type: expectedOutputs: [{type: }); ``` When a user analyzes photos, we want them to see the results as soon as possible. Although not a true server stream, we created a stream-like experience in the client. We loop through the images and start the analysis. Using `Promise.all` would wait for all the images to finish, but by processing them individually in a loop, React can update its state as soon as an image’s metadata is returned. ``` export const let if prompt }; const return }; const // ... setup code ... setIsAnalyzing setStatus for try // Call the AI logic for a single image const // Update the state for this specific image, triggering a re-render setImageData itemIndex ? : )); } console. } } setIsAnalyzing setStatus }; ``` After the initial analysis, we gather all the generated JSON objects into an array. This collection is sent to our second model configuration, `sortAndCategorizeImages`, in a single request. This prompt instructs the model to act as a photo organizer and group the images based on the user’s chosen method. ``` const const const if setStatus return } setIsAnalyzing setStatus const setSortedData setStatus setIsAnalyzing }; export if try const const return } console. return } } ``` AI-generated tags are only useful if the user can use it somewhere. To give the user access to those tags, we provide an export option for the user to be able to save the tags their image. It uses the `piexifjs` library to embed the metadata as a JSON object within the `UserComment` field of the image’s EXIF data. This let users to then import their photos to other tools so that they may use the AI-generated tags with other desktop tools for further photo refinement. ``` const // Omitting for brevity const reader. const const " [piexif.ExifIFD.UserComment]: JSON. } }; const const // Omitting for brevity }; } ``` ## Production Web Architecture & Security Checklist Whether we are deploying on [**Firebase App Hosting**](https://firebase.google.com/docs/app-hosting), Cloudflare Workers, or Vercel, shipping AI features to production requires defense-in-depth across three layers: - **Enforce Firebase App Check & Attestation: ** Protect your web endpoints, Cloud Functions, Data Connect GraphQL resolvers, and Firebase AI Logic calls against unauthorized replay or billing abuse using [**Firebase App Check with reCAPTCHA Enterprise**](https://firebase.google.com/docs/app-check). - **Validate Locally with the Firebase Emulator Suite: ** Run deterministic integration tests against local Firestore, Auth, Functions, and Data Connect PostgreSQL emulators before promoting schema or rule changes to production. - **Keep Server & Client Boundaries Type-Safe: ** Use the modular Firebase Web SDK and generated TypeScript bindings so SSR frameworks like Next.js, Astro, SvelteKit, and Angular serialize state cleanly without hydration flickers. ## Conclusion Accelerate your full-stack web and AI development by pairing **Smart photo organization through privacy-first, on-device AI** with strict Security Rules, Firebase App Check, and local emulator testing. Come tell us on [**X (@CodingCatDev)**](https://x.com/CodingCatDev) and [**LinkedIn (CodingCat.dev)**](https://www.linkedin.com/company/codingcatdev) how you are using Firebase in your stack, and subscribe to the [**CodingCat.dev YouTube Channel**](https://www.youtube.com/@CodingCatDev?sub_confirmation=1) so you don't miss what we build next! ## Watch Our Related Firebase Video Tutorial on YouTube Want to see how we architect and debug Firebase web applications live? Watch our related CodingCat.dev session [**Firebase SQL with Data Connect**](https://youtu.be/N25khadfUag) below, and make sure to [**Subscribe to CodingCat.dev on YouTube**](https://www.youtube.com/@CodingCatDev?sub_confirmation=1) for weekly full-stack web and AI deep dives: ## Resources, Official Docs & Related CodingCat.dev Guides - **Primary Release Source: ** [Smart photo organization through privacy-first, on-device AI on the Official Firebase Blog](https://firebase.blog/posts/2025/10/privacy-first-on-device-ai/) (published 2025-10-15 by Cynthia Wang) - **Official Firebase Documentation: ** [Firebase Developer Docs & Web SDK Reference](https://firebase.google.com/docs/web/setup) - **Related on CodingCat.dev: [It’s graduation day for the Firebase MCP Server](/post/firebase-2025-10-firebase-mcp-server-ga)** - **Related on CodingCat.dev: [Understanding hybrid inference: the role of Firebase AI Logic](/post/firebase-2025-10-understand-ai-logic-hybrid-inference)** - **Related on CodingCat.dev: [Firebase SQL with Data Connect](/podcast/firebase-sql-with-data-connect)** - **Related on CodingCat.dev: [Build a Custom No-Code App in 90 Minutes](/podcast/build-a-custom-no-code-app-in-90-minutes)** ## Frequently Asked Questions (FAQ) ### What are the key capabilities introduced in Smart photo organization through privacy-first, on-device AI? Smart photo organization through privacy-first, on-device AI (announced 2025-10-15 by the Firebase team) enhances the platform for Firebase, Google Cloud, Web Dev, AI, React, TypeScript, Security workloads. User privacy can be a big concern when you integrate AI into your app, especially when your app lets users upload personal data, like photos to a cloud server for processing. To show how you can safeguard user privacy while building with AI, we developed a progressive web app (PWA) that brings AI to the photos instead of the other way around. By running the model directly in the browser, we can ensure that it won’t compromise the privacy of the users. Developers can review the primary announcement on the [**Firebase Blog**](https://firebase.blog/posts/2025/10/privacy-first-on-device-ai/) alongside our implementation patterns and production checklist above. ### How do web developers use Smart photo organization through privacy-first, on-device AI with Next.js, SvelteKit, React, or Angular? Web developers can integrate these features using the modular Firebase JavaScript/TypeScript SDK, Firebase Data Connect web bindings, and [**Firebase App Hosting**](https://firebase.google.com/docs/app-hosting). Always pair client-side SDK initialization with Firebase App Check and strict Security Rules, and test locally with the Firebase Local Emulator Suite before deploying. ### Where can I watch video tutorials on Firebase Web and AI development? You can watch our hands-on Firebase video walkthroughs on the [**CodingCat.dev YouTube Channel**](https://www.youtube.com/@CodingCatDev?sub_confirmation=1), including [**Firebase SQL with Data Connect**](https://youtu.be/N25khadfUag), and follow us on [**X (@CodingCatDev)**](https://x.com/CodingCatDev) for weekly full-stack tutorials. ## Recent Posts [View More](/blog/page/1)[![PHP2Shell (CVE-2026-109618): How AI Found a Decade-Old Bug in 70% of the Web](https://cdn.sanity.io/images/hfh83o0w/production/e83065d0a4e2f841ce64d3f4aea8ddbbf07e5cb9-1376x768.jpg?w=1376&q=82&fit=max&auto=format)](/post/php2shell-cve-2026-109618-ai-php-security-scramble) ### [PHP2Shell (CVE-2026-109618): How AI Found a Decade-Old Bug in 70% of the Web](/post/php2shell-cve-2026-109618-ai-php-security-scramble) October 22, 2026 Two critical pre-authentication RCE flaws in PHP (CVE-2026-109618 and CVE-2026-109619), uncovered by AI models from OpenAI and Google Gemini, sparked a global patching scramble across 70% of the web. Here is how PHP2Shell works and how to patch your servers. [![Alex Patterson](https://cdn.sanity.io/images/hfh83o0w/production/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)[![Sanity Knowledge Bases](https://cdn.sanity.io/images/hfh83o0w/production/38b0f021c51d97043c8a1ebf31d505a0ea877b7c-2752x1536.jpg?w=1920&q=82&fit=max&auto=format)](/post/sanity-knowledge-bases) ### [Sanity Knowledge Bases](/post/sanity-knowledge-bases) October 17, 2026 Eliminate query-time guessing in your AI agents. Learn how Sanity Context Knowledge Bases compile structured datasets, crawled documentation websites, and uploaded files ahead of time, resolve conflicting facts, and serve a compact outline over Context MCP. [![Alex Patterson](https://cdn.sanity.io/images/hfh83o0w/production/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)