======================================== Questions not addressed during the class ======================================== ## Some of the questions below may not be exactly what you asked, for I summarized similar questions. ## Some questions may be skipped. This happens (1) when answering the question needed some clarification or contexts behind the question, (2) when a similar question was asked in the class, or (3) when the presentation essentially answered the question. ## If you find your questions unanswered or if you have follow-up questions, please let me know. Q: Considering that evaluation based on multiple measures is costly and takes some effort, do you think that a single scalar (closed-down) measurement can be desirable in certain situations? A: No. At the end of the day, evaluators have to make a decision, where a single measure is created in a sense (someone winning and others losing is one dimensional). However, the decision should not be based on a single measure; if based on a single measure, it is not really that decision is made there, but the decision is made by someone who computed the measurement. Q: How can we avoid misuse of simple S&T indicators, especially once the measurements prevailed and are widely used? A: This is difficult question that the current research cannot answer. Future research is needed. Q: Once ranking or evaluation based on some indicators are accepted, scientists or organizations change their behaviors to improve their ranking, etc. Should we stop them doing that, or should we let them be? A: The goal of policy is to improve performance (in some sense). Whatever the indicators or the ranking are based on, they are only a proxy of the performance. Without understanding this, making effort to improve merely the ranking is a waste of resource. Q: Scientometrics seem able to dilute many biases present in peer-review such as individual specialization, personal interests and so on, but scientometrics is not completely free of such biases, for example, peer-review may perform better at the individual scientist level. How should we deal with this kind of problem? A: As mentioned, any measurement based on scientometrics involves some bias. Evaluations should not be based solely on metrics. As long as peer review is professionally done, the role of scientometrics should be supplementary; i.e., metrics informs reviewers. Q: What are the drawbacks of simple S&T indicators? Have there been any prominent cases where it has failed to capture the real picture? A: When the evaluation is based on those indicators, scientists should be pressured to publish their papers in high-impact journals. This can bias the contents of science. For example, my case study of business schools finds that they are not motivated to do research on climate change because it won't be published in prestigious journals, though its social relevance is obvious. The same problem occurs in natural sciences, too. Q: As mentioned in the report, manipulation of scoreboards is a potential danger that can lead to unreliable rankings. There are cartel-like groups that cites each othersf publications in order to inflate citations. How widespread do you think such cheating behaviors are? A: The cartel-like behavior is not very common, as far as I know. As a similar behavior, self-citation (to cite your own papers) is notorious, but it has been constantly observed over decades; no noticeable increase is observed due to the pressure for evaluation. There is a paper about the impact on research integrity; Martin, B. R. 2013. Whither research integrity? Research Policy, 42: 1005-1014. Q: If multi-dimensional measurements are introduced for evaluation, how will the system influence the researchers? After all, researchers will make effort only to improve the measurements than to really improve performance? A: Whatever the measurements are, there is always such a risk. But, it is certainly more difficult for researchers to play with the system when the dimensionality is higher than when it is uni-dimensional. Yet, it takes further empirical research to know how such system change affects the behavior of researchers. Q: Should we also use patent data in addition to publication data when computing S&T indicators? A: Patent data certainly adds another dimension of information for the decision-makers. But, we have to be careful that patent data can be trickier than publication data. The motivation for publication is rather universal for scientists, but that for patenting is less so. The practice of patenting significantly differs by fields, for example. Thus, measurement based on patents could be misleading if not carefully used. Q: How are the concepts of ebroadening outf and eopening upf related to technology assessment ? A: The same concept applies to technology assessment. See: Adrian, et al. (2013) Broadening out and opening up technology assessment. Research Policy (In Press). Q: In the paper, how do the indicators handle differences in citation and publication characteristics among scientific fields? A: As for citation and publication count, they are divided by field-average for standardization.