The MathSeer project, funded jointly by the NSF and Alfred P. Sloan Foundation, sought to create new math-aware search engines and interfaces to fully support formulas as well as text in searching technical documents (e.g., research papers, and math course materials), intended for use by mathematical non-experts. Our slogan was 'Math search for the masses.'

Products. By the end of the project, the MathSeer project had produced the following. Note that source code for all software described is available as open source from GitLab.

1. ARQMath Collection. A dataset created over three years through international competitions run by the MathSeer team for the CLEF conference. The collection was built from a large collection of Math Stack Exchange posts. ARQMath contains over 200 search topics (queries + scored results) for both math question answer retrieval and formula search tasks. ARQMath is the largest collection of its type, already serving as an international benchmark.

2. Graphics Extraction and Retrieval Framework. PDFs generally contain drawing instructions and images, with no indication of where formula regions are located. To address this, we created a framework to detect and recognize math formulas in PDFs. The framework includes new detectors for math formulas in document images (ScanSSD and a Yolov4 variant), and a math formula parsing technique designed to work for handwriting, images, and PDF symbols extracted directly from PDF documents (QD-GGA). We also created a tool for extracting symbol locations from PDF documents accurately (SymbolScraper). SymbolScraper works well enough that it was used in creating Allen AI's initial Semantic Reader prototype. In addition, we have created a framework that supports the execution of queries combining text and formulas, when queries are represented in text with formulas in LaTeX format.
   

3. MathDeck. We created a novel search interface to make it easier to create, edit, and use formulas in search queries. This led us to create both a new structure-based formlua editor supporting both visual operations and LaTeX, and the representation of formluas in 'chips' that can be used both in search and editing, allowing formulas to literally be built in pieces, and to move parts of formulas around. We also devised formula entity cards, which contain a functional formula 'chip' put on a card with a description from wikipedia attached. These cards appear during editing, as a form of information-rich autocompletion for formulas. Both chips and cards can be edited, saved, shared, and reused. A news story on MathDeck appeared in ACM TechNews in 2020 (story).
   

4. A variety of state-of-the-art formula retrieval models, including neural embeddings, spatial pyramid representations of symbols that can be used with standard search engine construction frameworks (PHOC), learning-to-rank models for combining representations, and a new multmodal representation for text and math, that captures the argument structure of formulas and words in sentences in a uniform representation (MathAMR). MathAMR is adapted from Abstract Meaning Representation graphs used in Natural Language Processing (NLP).
   

Publications. The project produced 28 peer-reviewed publications, and received two best paper awards (ICFHR 2018, ECIR 2019) and another best paper nomination at JCDL 2019. A number of papers appeared in top venues spanning different areas of computer science, including SIGIR, CHI, and CVPR.

Project Members. The project involved the participation of 20 students and 1 full-time research programmer between 2017 and 2022. The project provided a collaborative, suportive environment to develop research skills for students from the community college level through PhD. PhDs from the project have gone on to faculty positions and research posts in industry; a number of PhD and Master's students held industrial research internships, and some Master's students went on to work in industrial research as well. Companies that students went to intern and work for include Amazon, Apple, Microsoft, and start-up companies such as Petuum.

Next Steps. The Document and Pattern Recognition Lab at RIT is currently working to extend the systems and tools created for MathSeer to chemistry, in support of another NSF-funded project, the MMLI NSF AI Center. The project has explored new directions to advance the development of graphics-aware search engines, which we believe have a very bright future.