Visual Genome Exploration

My A3 assignment focused on displaying scene graph triplets from the Visual Genome dataset, and focused on the displaying the distribution of relationships present in the dataset. For this project, I want to create a tool that allows for broader exploration of the dataset, including visualizing sample images from the dataset, similarity measures between subject/predicates/objects, and more organic exploration fo the dataset. Currently, I have implemented TSNE clustering based off the the learned GLOVE word embeddings. At each layer of the visualization, I display the sum of the word vectors used in the triplet, such that the first layer contains only subjects, the second layer contains subjects+predicates, and the third layer contains subjects+predicates+objects. In each layer, an sample image is displayed containing a triplet involving the highlighted word. The triplet is shown above the image, with the highlighted words being the ones selected in prior layers. The size of each bubble corresponds to the number of occurences that the selected triplet appears in the dataset. The code used to format and generate the data for this project can be found in the assignment's gitlab repository.

I used many sources to assist with this visualization: 1 2 3 4 5

I worked alone on this project. I used GitHub Copilot code completion during the code process.

Team Members: Jack Lowry

Instructions

Each dot represents a subject, predicate, or object from the Visual Genome dataset. The size of the dot represents the number of times that node appears in the dataset. Users can click on a dot to filter triplets that only contain that word. A sample image is shown, which includes that a triplet containing that word. Above, that triplet is displayed, with the highlighted words showing which words have been selected by the user.

Shirt On Man

Sample Image

Occurences of triplet in dataset

(at default scale)