LARGE SCALE ENTITY MATCHING

Maisha Alam

Erik Saule

Motivations

Write your motivations here

Problem Statement:

• Challenge: Vast and growing volume of academic research papers.
• Issue: Difficulty in managing and discovering relevant materials.

​

• Why It's Necessary to Solve This Problem:
• Information Overload: Over 2.5 million new research papers are published annually. Impossible to manually filter through this deluge.
• Precious Time Lost: Researchers spend hours searching for papers, reducing time for actual research and innovation.
• Missed Opportunities: Failure to discover relevant studies can hinder progress and limit scientific breakthroughs.

​

• Outdated Methods: Traditional search engines and keyword-based searches are no longer sufficient.

​

• The Solution:
• Efficient Tools: Developing and using technology-driven tools to streamline paper management and improve discovery.��This slide communicates the urgency and necessity of solving the problem while hinting at the proposed solution.

​

​

​

• �

​

​

Backgrounds

Write the related and relevant background of your project here …

• Solution Overview: � 1. Our project aims to develop a system inspired by “theadvisor."� 2. It uses citation analysis and recommendation techniques to assist researchers in their literature search.� 3. In this project we aim to work with research papers that is recent and has comprehensive metadata and citation information. We are also focusing on the latest and most up-to-date research materials. In an era of information overload, ensuring access to recent and high-quality sources is critical for the success of any research endeavour.�
• Components of the System:� 1. Data Ingestion: Handles various input formats (e.g., TXT, XML) and parses and processes data.� 2. We can parse DBLP research papers as they are consistent, and similarly, parse MAG (Microsoft Academic Graph) files without storing the papers or taking up too much space in memory.�
• Data Sources:� 1. Academic paper databases, conference proceedings, and expert databases provide the foundation for the project's functionality.

Current Progress

Write the current progress of your project here …

• In our project, we've successfully implemented a codebase that efficiently parses extensive datasets. The core functionality revolves around a callback function.

​

• Data Handling: Our codebase is adept at handling large and complex datasets, which is a crucial requirement for academic research. Researchers often deal with extensive libraries of papers and references, and our system can seamlessly process this wealth of information. In the MAG file, we have designed a computer program that can be valuable for researchers and data analysts who want to work with large datasets of academic papers. It extracts specific information (paper ID, DOI, and title) from a compressed file, which is a common format for distributing large datasets. For the DBLP file, we have designed a program that can be useful for researchers, data scientists, or anyone interested in analysing academic publications. It provides a structured way to extract metadata from a large XML file, which is often used to store academic publication data. The code extracts and prints information such as the title, author, year, journal, and electronic edition (EE URL) of academic articles.

Future Plans

Write future plans regarding your project here …

In our future plans for this project, we envision the utilisation of locality-sensitive hashing, a powerful technique that allows us to efficiently assess the similarity of research papers. While we strive to parse extensive databases like DBLP and MAG without storing excessive data in memory, we recognise the importance of a robust algorithm for determining the likeness between papers. Locality-sensitive hashing offers a promising solution by enabling us to quickly evaluate the similarity of papers, facilitating tasks such as clustering related research, recommending relevant literature, and identifying common themes or trends in academic publications.

Thank You

Q & A