Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing a Research Publication's Contribution

Petr Knoth, Drahomira Herrmannova

Full text: http://dx.doi.org/10.1045/november2014-knoth

Find on Google Scholar

---

Abstract

We propose Semantometrics, a new class of metrics for evaluating research. As opposed to existing Bibliometrics, Webometrics, Altmetrics, etc., Semantometrics are not based on measuring the number of interactions in the scholarly communication network, but build on the premise that full-text is needed to assess the value of a publication. This paper presents the first Semantometric measure, which estimates the research contribution. We measure semantic similarity of publications connected in a citation network and use a simple formula to assess their contribution. We carry out a pilot study in which we test our approach on a small dataset and discuss the challenges in carrying out the analysis on existing citation datasets. The results suggest that semantic similarity measures can be utilised to provide meaningful information about the contribution of research papers that is not captured by traditional impact measures based purely on citations.

---

---

Ratings & reviews

• Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing a Research Publication's Contribution

  This is a very good idea

  Răzvan Valentin Florian

  Răzvan ValentinFlorian

  100 90 0

  

  I think that semantic similarity-based measures similar to the prototype proposed here have a great potential for improving scientometrics.

  However, the exact formula probably needs improvement (and this is expected for a first iteration of implementing this concept). The proposed formula of the contribution is proportional to the spread of cited publications and inversely proportional to the spread of citing publications. This reflects the stated hypothesis "that if a paper uses ideas from a narrow field, but has an impact on a very large field, it is a sign of the paper's contribution". However, when creating new concepts, a publication may cite papers from various fields (e.g., multiple fields in cognitive science, such as psychology, machine learning, cybernetics) but its application may be in a well defined field such as computational neuroscience. In this case the publication would cite from a large field and be cited in a relatively narrower field, but this, by itself, should not decrease the relevance of the publication.