HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> OpinioNet: the LiquidPub Reputation Module

OpinioNet

The LiquidPub Reputation Module

Project

The world of scientific publications has been largely oblivious to the advent of the Web and to advances in ICT. Even more surprisingly, this is the case even for research in the ICT area: ICT researchers have been able to exploit the Web to improve the (production) process in almost all areas, but not their own. We are producing scientific knowledge (and publications in particular) essentially following the very same approach we followed before the Web. Scientific knowledge dissemination is still based on the traditional notion of “paper” publication and on peer review as quality assessment method. The current approach encourages authors to write many (possibly incremental) papers to get more “tokens of credit”, generating often unnecessary dissemination overhead for themselves and for the community of reviewers. Furthermore, it does not encourage or support reuse and evolution of publications: whenever a (possibly small) progress is made on a certain subject, a new paper is written, reviewed, and published, often after several months. The situation is analogous if not worse for textbooks.

The LiquidPub project proposes a paradigm shift in the way scientific knowledge is created, disseminated, evaluated and maintained. This shift is enabled by the notion of Liquid Publications, which are evolutionary, collaborative, and composable scientific contributions. Many Liquid Publication concepts are based on a parallel between scientific knowledge artifacts and software artifacts, and hence on lessons learned in (agile, collaborative, open source) software development, as well as on lessons learned from Web 2.0 in terms of collaborative evaluation of knowledge artifacts.

This is the general idea of the project. IIIA-CSIC will collaborate providing a reputation module, OpinioNet, that will allow the user to choose a researcher or a piece of research work and compute the reputation of that entity. However, the reputation for each entity chosen (whether an author, a reviewer, an SKO, an SKO set, etc.) can be customised by the user. For example, the reputation of a paper may be computed based on citations only, opinions only, or both. The flexibility and customisation in calculating the reputation is key.

OpinioNet

The OpinioNet reputation module provides a set of algorithms for computing the reputation of two of the resources defined in the LP Platform: contribution and person. In both cases, reputation is computed depending on the information source. The table below shows the types of reputation and the different information sources considered by the reputation module. Other reputation metrics such as h-index and g-index are provided by the performance module.

The reputation of a scientific contribution is computed in terms of the opinions the contribution has received. The reputation of researchers, however, depends on the role that researchers play. We currently distinguish between two roles that can be played by researchers inside the LP world: author and reviewer. The reputation of an author is based on the reputation of its contributions. In contrast, the reputation of a reviewer is calculated by analyzing his/her reviews or his/her social network (which we leave for future work).

Here you can find the API specification of the OpinioNet reputation module published at Google Docs.

Propagation Application

You can download and test the application here .

The propagation algorithm is based on the idea that assigned opinions on SKO nodes (conference papers, journals, conferences, series...) also affect the reputation of neighbouring nodes and people related to them (authors, reviewers, ...).

We also note that OpinioNet is being integrated with the Liquid Journal use case. Actions such as subscribing to a Liquid Journal, or adding a paper to a Liquid Journal are translated into indirect opinions, and propagated accordingly. This is, however, ongoing work.

Additionally, OpinioNet has also been reading data from the Interdisciplines (or the Liquid Conference use case). In the following we provide a video of the propagation application, where we can see a simulation of the behaviour of the algorithm over time, using fake data of two conferences with thirty papers per conference.

Integrating OpinioNet with ItRank

When aggregating opinions, OpinioNet bases the reliability of each opinion on the reputation of the opinion holder. We say an entity that is considered very good in a certain field is usually considered to be very good as well in assessing how others are in that field. In other words, expertise is the bases for good reviews. This is based on the ex-catedra argument. An example of a current practice following the application of this argument is the selection of members of committees, advisory boards, etc. However, in addition to expertise, one may also consider assessing how good was the researcher's past opinions. For this, OpinioNet may make use of ItRank.

ItRank, or the Iterative Ranking module, takes possibly multidimensional ratings and uses them to obtain a list of user weights and a list of object scores. Depending to the chosen algorithm, both weights and scores may differ from one rating criterion to another. Supported algorithms include AA, YZLM, dKVD, YZLM*, and dKVD*. For further details on ItRank and its algorithms, we refer the interested reader to the LiquidPub deliverable D4.2.

Related Work

The research carried out by [11, 1, 3] studies the dynamics of opinion formation by focusing on the effect of social relations on how peoples’ opinions may influence each other in a social network.

Repage [8], ReGreT [7], and SUNNY [4] provide mechanisms for computing the confidence in a reviewer based on the social relations. These mechanisms mainly influence the reliability of the reviewer, which is crucial when aggregating opinions, yet outside the scope of our work. Similar to [8, 7, 4], additional aggregation mechanisms, such as [9, 6], may be viewed as complementary to our work, which mainly focuses on the propagation of opinions for one reviewer in the structural graph.

In the area of publications, SARA [5] and CiteRank [10] present algorithms on how reputation may propagate based on who is citing whom. Their reputation propagates along citation links. In our case, however, we focus on the propagation of reputation along structural links by focusing on the composition of entities and using the part of relation as an indication to the flow of opinions from one entity to another. The EigenTrust [13] algorithm also has a notion of propagation of trust from one trustee to another: e.g. if I trust you, then I will trust the people you trust. Again, our focus is on propagation along structural links.

Finally, research work on ontology-based recommender systems, such as [2, 12], makes use of the clustering or classification of information and uses machine learning and data mining techniques for ranking and recommending entities. One may draw similarities between the taxonomies used by such systems and that of the structural graph of our work; although the propagation mechanism of this paper is unique in both its algorithm and semantics.



[1] V D. Blondel, J L. Guillaume, J M. Hendrickx, C. Kerchove, and R. Lambiotte, ‘Local leaders in random networks’, Physical Review E, 77(3), 036114, (2008).
[2] Amancio Bouza, Gerald Reif, Abraham Bernstein, and Harald Gall, ‘Semtree: Ontology-based decision tree algorithm for recommender systems’, in Proceedings of the Seventh International Semantic Web Conference (ISWC2008), (October 2008).
[3] P. Klimek, R. Lambiotte, and S. Thurner, ‘Opinion formation in laggard societies’, EPL (Europhysics Letters), 82(2), 28008+, (2008).
[4] Ugur Kuter and Jennifer Golbeck, ‘Sunny: A new algorithm for trust inference in social networks using probabilistic confidence models’, in Proceedings of the 22nd national conference on Artificial intelligence (AAAI’07), pp. 1377–1382. AAAI Press, (2007).
[5] Filippo Radicchi, Santo Fortunato, Benjamin Markines, and Alessandro Vespignani. Diffusion of scientific credits and the ranking of scientists. http://arxiv.org/abs/0907.1050, Sep 2009.
[6] Matthew Richardson and Pedro Domingos, ‘Building large knowledge bases by mass collaboration’, in Proceedings of the 2nd international conference on Knowledge capture (K-CAP’03), pp. 129–137, New York, NY, USA, (2003). ACM.
[7] Jordi Sabater and Carles Sierra, ‘Social regret, a reputation model based on social relations’, ACM, SIGecom Exchanges, 3.1, 44–56, (2002).
[8] Jordi Sabater-Mir, Mario Paolucci, and Rosaria Contexs, ‘Repage: REPutation and imAGE among limited autonomous partners’, Journal of Artificial Societies and Social Simulation (JASSS’06), 9(2), (2006).
[9] Carles Sierra and John Debenham, ‘Information-based reputation’, in First International Conference on Reputation: Theory and Technology, ed., Mario Paolucci, pp. 5–19, Gargonza, Italy, (2009).
[10] Dylan Walker, Huafeng Xie, Koon-Kiu Yan, and Sergei Maslov. Ranking scientific publications using a simple model of network traffic. http://arxiv.org/abs/physics/0612122, 2006.
[11] Fang Wu and Bernardo A. Huberman. Social structure and opinion formation. http://arxiv.org/abs/cond- mat/0407252, 2004.
[12] Cai-Nicolas Ziegler, Lars Schmidt-Thieme, and Georg Lausen, ‘Exploiting semantic product descriptions for recommender systems’, in Proceedings of the 2nd ACM SIGIR Semantic Web and Information Retrieval Workshop, Sheffield, UK, (July 2004).
[13] S. D. Kamvar, M. T. Schlosser, and H. Garcia-Molina, The EigenTrust Algorithm for Reputation Management in P2P Networks, In Proceedings of the Twelfth International World Wide Web Conference, 2003.