MIT

A particularly difficult aspect of developing MosAIc was creating an algorithm that could find not only similarities in color or style, but also in meaning and theme, Hamilton said. Researchers examined a deep network of “activations,” or features, for each image in the open access collections of both museums. The distance between the “activations” of the deep network was how researchers judged similarity.

Researchers also used a new image search data structure called a “KNN Tree,” which groups images together in a tree-like structure. To find one image’s closest match, the algorithm starts at the “trunk” of the grouping, then follows the most promising “branch” until it’s found the closest image. The data structure improves on itself by allowing the tree to “prune” itself based on characteristics of the image.

Hamilton said he hopes the work started on MosAIc can be expanded upon to other fields, like humanities, social sciences and medicine. “These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts,” he said.