https://tomhazledine.com/mapping-llm-embeddings-in-3d/ https://blog.gdeltproject.org/a-template-for-visually-comparing-embedding-models-exploring-capitalization-spacing-knowledge-cutoffs/ To generate 2D and 3D projections of the entire dataset—or a subset upon query—using TensorFlow's TensorBoard projector, you'll need to do the following: 1. **Generate Embeddings**: Before you can project the embeddings, you need to have them. It seems you're already generating embeddings using the `EmbeddingService.py/generate_embeddings()` function. 2. **Prepare Data for TensorBoard**: - You'll need to save the embeddings in a format that TensorBoard can understand. Typically, this is a `.tsv` file for the embeddings and another `.tsv` file for the metadata (labels). - The embeddings `.tsv` will have each row representing an embedding. If your embeddings are 100-dimensional, then each row will have 100 values. - The metadata `.tsv` will have labels. In your case, this could be the names of the contacts or any other identifier. 3. **Use TensorBoard's Projector**: - Launch TensorBoard and point it to the directory containing your `.tsv` files. - In TensorBoard, go to the Projector tab. Here, you can load your embeddings and metadata files. - Once loaded, you can use the UI to project the embeddings into 2D or 3D space. TensorBoard uses PCA, t-SNE, and UMAP for dimensionality reduction. 4. **Integrate with Django**: - To make this functionality available in your web application, you'll need a way to trigger the projection process and then display the results. - You can create a new Django view that handles the projection request. This view can call a service function that does the TensorBoard projection (as described above) and then returns the projected data. - On the frontend, you can use a visualization library (like D3.js) to display the projected embeddings. 5. **Challenges & Considerations**: - **Performance**: Projecting embeddings, especially using t-SNE, can be computationally intensive. Consider how often you need to do this and how you can optimize the process. - **Data Size**: If you have a lot of contacts, visualizing all of them might be overwhelming. Consider providing options to filter or sample the data. - **Interactivity**: The real power of such visualizations comes when they are interactive. Consider adding tooltips, zooming, and panning functionalities. From the files I've analyzed, I didn't find any existing code related to TensorBoard's projector. You'll need to integrate the TensorBoard projector into your Django application following the steps above. Would you like a more detailed guide or code snippets on any specific step?