Open Commentaries to "Ten Steps Toward a Better Personality Science: How Quality may be Rewarded More in Research Evaluation" (Leising et al.)

Reading the target paper of this “Theme Bundle” (Leising et al., 2022), steering it through an extensive peer-review process, collecting commentaries and reading them too, and finally writing the present text belonged to the most pleasurable professional experiences I had in 2021. The target paper is clearly thought-provoking, intellectually rich, and provocative in many aspects. It was foreseeable right from the start, and even more so after receiving the reviewers’ comments to the original version of the paper, that the text would stir a range of reactions among its readership and motivate many people to voice their consent or dissent. This is why the editor-in-chief at Personality Science, John Rauthmann, and I decided to solicit both invited and open commentaries and give the authors of the target paper the opportunity to respond to these. The package that resulted is, in my view, a “must-read” for every scholar interested in personality science and psychology more broadly. Although many commentators disagree with some (or many) of the recommendations made by Leising et al. (2022; i.e., the “ten steps towards better personality science”), they all agree that personality science can (and should) improve, that (implicit or explicit) incentive structures are relevant for such improvement, and that it is very difficult to define what “improve” actually means. There is no ultimate criterion for “good” science – the “ought” state. Yet, fueled by the replication debate in psychology, we feel that there is a discrepancy between the ought and the is.

Leising et al. (2022) argue that consensus-building (Steps 1-5) and methodological rigor including openness and transparency (Steps 6-10) may be a way to reduce the is-ought discrepancy, and especially the “consensus-building” argument provoked some well-grounded criticism among the reviewers of the original paper and other commentators (e.g., see the comments by Asendorpf & Gebauer; Corker; Denissen & Sijtsma; Fernandes & Aharoni; Hagemann; Hilbig et al.; Hogan et al.). A problem that these authors have with the “consensus-building” idea is that 1) consensus is prone to biases (e.g., it favors the mainstream), that 2) the common ground for consensus-building may be too shaky (i.e., a “garbage-in-garbage-out” problem; see the comment by Corker), and that 3) consensus penalizes dissent and, thus, a healthy evolution of science. Some commentators add an important flavor to the first point by arguing that each consensus-building process is likely to be influenced by power and status hierarchies in the scientific community (see the comments by Adler; Beck et al.; Fedorenko et al.; Galang & Morales; Klimstra; McLean & Syed). Without proper safeguards against a violation of procedural justice standards (e.g., Leventhal, 1980), those who are high in power and status will define the consensus. The process itself will lack inclusiveness, and the outcome will lack representativeness – a point that Naomi Oreskes (2019) also makes in her book “Why trust science?”, a book that is of prime relevance for our present discourse.

Other comments focus on the reward scheme that Leising et al. are proposing (e.g., Asendorpf & Gebauer; Beck et al.; Friedman; Schmitt). The argument here is that such reward schemes disqualify certain scientific approaches to personality (e.g., qualitative research; see Klimstra; McLean & Syed) and that it is the wrong criteria that are eventually being rewarded, such as submissiveness, closed-mindedness, and compliance with the mainstream (see Asendorpf & Gebauer). As good as Leising et al.’s intentions are, the idea of a standardized reward scheme for “good science” may ultimately aggravate the problems we have with quantitative indicators for scientific “quality” (such as the infamous h-index) instead of solving them. I agree with these arguments, and I would like to add an aspect that one of the reviewers of the initial manuscript (Rainer Bromme, who explicitly agreed on having his name disclosed here) had touched upon. Referring again to Oreskes (2019) as well as microbiologist/science philosopher Ludwik Fleck (1980), Bromme noted that the social process of doing science is far more complex than implied by the authors of the target article. Most researchers are (hopefully) not merely interested in increasing their h-index or accumulating reward points – they are intrinsically motivated to work on something important, to learn with and from other people, and to find out something relevant. For the sake of making all this possible, they even behave altruistically: they agree to review other people’s work, they serve as committee chairs, and they devote their time to self-administration (see the comment by Schmitt).

As any other social system, academia already has implicit (“built-in”) incentive structures that reward altruism and punish egoism. Cynics may snort and provide exemplars of narcissistic, egotistic individuals who went far in their academic careers, but I am convinced that these exemplars are exceptions to the rule. Most scholars would probably argue that the partners they particularly enjoy collaborating with are helpful, supportive, diligent, and trustworthy. Mutual trust is likely a much stronger predictor of “doing good science” than extrinsic rewards (Altenmüller & Gollwitzer, 2022), so instead of reinventing grading schemes and replacing one quantitative indicator by another, we might want to think more carefully about a system in which both self-correction (and openness to each other’s criticism) and mutual trust-building in science can be optimized.

That said, there are many ways in which doing good science can be facilitated by methodology and infrastructure: this includes not only platforms (repositories) for sharing our data, materials, and papers so that they can be much more easily accessed by others (“open science”), but also databases that can provide us with up-to-date knowledge about the psychometric properties (including measurement invariance) of methods and measures, as Horstmann and Ziegler as well as Mazei et al. mention in their comments.

A final point mentioned in some comments is that many of the “ten steps” described by Leising et al. are exclusively applicable to quantitative, variable-centered approaches to personality science and to “contexts of verification” (see Hogan et al.). Some commentators rightfully note that this should by no means imply that these approaches are in any way more valid or esteemed than, for instance, person-centered, idiographic, and/or qualitative approaches (see Dunlop, Klimstra). I believe that Leising et al. agree with this, but it deserves being mentioned explicitly here.

In sum, I hope that readers will enjoy digesting this Theme Bundle and that the present conversation will be the start of a lively, engaged discussion within personality science and beyond. To repeat, it was an extreme pleasure and honor being guest editor of this edition, and I thank the editors of Personality Science, the authors of the target article, and all commentators for this delightful opportunity.

The authors of the target article had good intentions. They wanted to advance personality science by implementing 10 “steps” designed to boost the trustworthiness of research.

We are sympathetic with most of these steps. We strongly disagree, however, with 1) the authors’ overly strong focus on consensus about important research questions, and 2) their proposed reward scheme for “good scientific work.” This scheme directly translates into a social reward system for young scientists, thereby shaping the norms of scientific culture. The scheme rewards swimming with the mainstream and penalizes swimming off or against the mainstream. In effect, minority views, creativity, and innovation will be discouraged. We assume that the members of the task force do not intended to penalize minority views, creativity, and innovation and, thus, we encourage them to consider not only the potential merits of their proposal but also potential, negative side-effects on creativity and innovation.

Contrast the target article’s reward scheme with the first three evaluation criteria for a successful ERC grant proposal (European Research Council, 2021, p. 33):

The target article’s reward scheme would reward research that fulfills those criteria with zero points (unless the applicant was not the first to identify the “important challenge,” but borrowed it from some consensus list of important challenges). A small variation on a mainstream topic devoid of any new idea, by contrast, would gain 10 points if related to a consensus in a certain area, published open access and pre-registered with sufficient power – simple criteria that can be easily achieved. The target article briefly mentions creativity and innovation in passing, but these two criteria did not make it in the authors’ scheme. They defend their neglect of those cardinal criteria with the difficulty of judging creativity and innovation. But difficulty cannot be an excuse for proposing an unbalanced reward system, especially if the authors expect their reward scheme to shape scientific culture, setting norms for what kind of research should be admired and what kind should be frowned upon.

As the past 15 years have shown, quick and dirty measures of research quality such as the Hirsch index can gain a life of their own and mislead young scientists. Many tried to maximize their h-index through heavy conference tourism, excessive networking, and inflation of co-authorships and cross-citations, all of which kept them away from their core tasks as scientists.

Leising et al. (2022) report the consensual opinion of five psychologists with a clear mission: “Scientists need to be increasingly rewarded for doing good work, not just lots of work.” Because nobody can object to the proposition “quality matters”, this mission statement is indisputable. However, when it comes to specify what “good work” really is and how good work may be encouraged, things become more complicated and less clear-cut. The authors base their consideration on writings of Karl Popper, including key concepts such as cumulative progress, formalization, and of course, falsification. There is much to say about why science is not only an accumulative business (Kuhn, 1962), why formalization of theory and hypotheses has limitations (Heylighen, 1999), and why physicists do not believe in falsification (Singham, 2020). Due to space limitations, however, I will restrict my comment to what the authors call the building of “consensus”.

According to Leising et al. (2022), fostering “consensus” regarding research goals, terminology, measurement practices, data handling, and the current state of theory and evidence are the first five steps toward a better personality science. When they introduce “consensus” as an essential element of good science in their section “What is good research?”, they refer to work of Oreskes (2020). Because the formation of “scientific consensus” has been a target of intense studies in the sociology, history, and philosophy of science (Oreskes, 2020; Shwed & Bearman, 2010), it is worth to have a closer look at this concept.

In the field of science studies, “scientific consensus” is a term that refers to the transformation of empirical propositions into facts by a scientific community (e.g., Shwed & Bearman, 2010). In the typical case, some experts analyze the status of a scientific domain and report their conclusions. One prominent example is a report by Oreskes (2004), who analyzed the validity of an IPCC report from 2001 that stated a consensus on anthropogenic climate change. The repeated questioning of this consensus by some policymakers motivated her research. Oreskes (2004) classified 928 abstracts that contained the keyword “global climate change” into several groups, ranging from endorsement over indifference up to rejection of the consensus position. She found no single abstract that disagreed with the consensus position and concluded that “scientists publishing in the peer-reviewed literature agree with IPCC” (Oreskes, 2004, p. 1686). This “scientific consensus” is why the anthropogenic climate change may be considered to be a scientific “fact” (Shwed & Bearman, 2010) or scientific “knowledge” (Oreskes, 2020). Aside the proposition “human activity is producing global climate change “, other propositions that have become target for consensus research are “smoking causes cancer”, “vaccinations cause autism”, or “there are gravitational waves”, just to give some examples (for these and further cases, see Oreskes, 2020; Shwed & Bearman, 2010). In any case where Oreskes (2020) discusses the “scientific consensus”, she refers to empirical propositions that reflect a positive or reliable knowledge, which is to say that these propositions can be “true” or “wrong” because they represent empirical features of the real world (of course, consensus does not imply an eternal truth because scientists may err; c.f. “instability of scientific truth” in Oreskes, 2020, p. 74).

It is obviously not possible to extend this concept of “scientific consensus” to research goals, terminology, measurement practices, and data handling. Research goals may be interesting, terminology may be useful, and measurement practice and data handling may be valid. However, the attribute “true” is not in stock because none of these items can ever be scientific facts in the sense of an empirical proposition (see Shwed & Bearman, 2010). No matter if we belief that the first four steps of Leising et al. (2022) are useful for making personality science better or not, Oreskes (2020) does not provide the epistemological grounding.

Quite the contrary may be true. In order to integrate empirical findings from different studies into a reliable scientific fact or trustworthy knowledge, the integrated evidence and the involved scientific community needs to satisfy several criteria of good science. Oreskes (2020, pp. 143-144) lists five of them: (1) “Do the individuals in the community bring to bear different perspectives? Do they represent a range of perspectives in terms of ideas, theoretic commitment, methodological preferences, and personal values?” (2) “Have different methods be applied and diverse lines of evidence considered?” (3) “Has there been ample opportunity for dissenting views to be heard, considered, and weighted?” (4) “Is the community open to new information and able to be self-critical?” (4) “Is the community demographically diverse: in terms of age, gender, race, ethnicity, sexuality, country of origin, and the like?” Only when the answers to these questions is "yes" should we trust the consensus.

The simple reason why Oreskes (2020) demands diversity to such a large degree is that each individual research finding and each individual scientist is inevitably biased—and diversity is the method to counteract it. Of course, these criteria of good science echo many tenets in the methodology of psychology, such as using different operations and methods to counteract the mono-operation bias and the mono-method bias and to establish convergent and discriminant validity (Shadish et al., 2002). In any case, these criteria of Oreskes (2020) appear rather to be an anti-thesis to the first four steps of Leising et al. (2022) than an epistemological support for building “consensus” beyond the empirical facts.

To conclude my thoughts about the proposal of Leising et al. (2022), I think these authors have hijacked Oreskes' (2020) notion of "consensus" to give their proposal the air of epistemological grounding, but they really missed the point: Diversity of concepts and methods is the prerequisite for a consensus about empirical findings to be trustworthy. Their ten steps and in particular their reward scheme emphasize agreement and consensus but downplay or ignore the values of disagreement and diversity. Therefore, the proposal of Leising et al. (2022) will need further elaboration before it may help to establish a better personality science.

Leising, Thielmann, Glöckner, Gärtner, and Schönbrodt (2022) suggest better (personality) science can be achieved through greater consensus-building and improving the credibility of empirical research. Whereas the latter is a perfectly reasonable goal (even if some specifics remain debatable), the former strikes us as questionable to put it mildly. Specifically, Leising et al. (2022) call for and seek to heavy-handedly reward more consensus on (a) what we know and what we ought to find out next (their Steps 1 and 5) as well as (b) terminology, measurement, and analytical approaches (their Steps 2-4). Crucially, although they acknowledge that consensus must be seen as preliminary and envision consensus on the basis of a fair, transparent, and evidence-based process, their essential proposal – manifested as a reward-point-table – asks only whether there is self-ascribed¹ consensus.

First off, it is worth considering the intentions apparently feeding this call for consensus by Leising et al. (2022). Undeniably, lack of theoretical integration and too few serious efforts to pit competing theories against each other with the best available data and analyses techniques hinder progress. More obviously still, inconsistencies and idiosyncrasies in terminology, measurement, and analytical strategies can hinder the establishment of reliable evidence. Personality science traditionally rewards construct inflation (continuous invention of ever more constructs and operationalizations with dubious distinctiveness or incremental usefulness) over theoretical specification or integration and has not sought let alone found any effective means of preventing jingle-jangle-fallacies. Vague definitions render traits barely testable (beyond some correlation with something similar) and allow for multiple non-equivalent operationalizations of what is (often inductively) given the same label. There is no denying that (personality) science is beset by systematic problems along these lines. As such, the diagnosis by Leising et al. (2022) is spot-on.

But consensus is not the cure. Consensus is, by definition, a matter of majority belief or preference. It is ultimately whatever most agree to, for whatever reason, and thus not necessarily aligned with any logic or evidence. In science, however, authority is not wielded by some (self-proclaimed) majority but by logic and evidence, albeit preliminary and uncertain. To promote the role of consensus, Leising et al. (2022) go so far as to state that “facts are claims about which agreement has been reached among scientists in the respective field” (manuscript p. 5). They are not. Facts are consistent, independently verified, and reliable empirical observations – or, albeit more tentatively, explanations unequivocally supported by such observations. Climate change is not a fact because most climate scientists believe in it, but because of evidence² .

Indeed, in line with Kuhn’s very argument, major scientific progress typically requires upending the consensus (normal science), rather than cementing its power through still more rewards. The history of science is replete with examples that clearly demonstrate how the consensus produces drag or inertia, flat-out denial, or worse. The consensus shouted down Wegener's theory of continental drift despite consistent and strong evidence; the behaviorist consensus upheld proliferation of impoverished associationist theorizing for decades despite consistent and strong evidence for the role of insight and expectation/anticipation in learning and problem solving; and if the replication crisis has taught us anything then how much it takes to question let alone upend (allegedly) “consensual” knowledge.

The deck is already stacked against dissent and counterevidence to widely held views. For an individual scientist’s career, normal science is a safer bet whereas rocking the boat is risky at best. Anomalies are often kept under the rug by the gatekeepers and stewards of normal science (editors and reviewers) or, if occasionally published, met with some combination of pushback, defensive downplaying, and denial. In (personality) science, the symptoms as diagnosed by Leising et al. (2022) are upheld by the consensus, albeit often implicit. If anything, our field needs to shield critical arguments and “inconvenient” evidence from the consensus, not hand the consensus bigger guns.

By all means, let us promote and reward publications that serve theory specification and integration, that actively seek to apply Occam’s razor so the field may focus on the constructs, models, and operationalizations we (really) need, that expose shortcomings in our terminology and frameworks and offer up alternatives, that develop a single, authoritative means of measuring a (well-defined) construct – or any combination of these. But let us never vote on it.

Any of the above can be achieved by any number of individuals. Their faction size does not and should not matter, only their logic and evidence. Anyone who would rather be a majority whip is free to go into politics. Anyone who would rather abide by shared beliefs is free to practice religion. These are the dominions of consensus, lending legitimacy to politics and authority to religion. But unlike science, neither politics nor religion are tasked with nor remotely suited for closing in on the truth. The latter goal, distal and formidable though it is, will best be served by a (personality) science that, above all, rewards challenging theories and alleged facts – consensual or not – through logic and evidence.

We are deeply invested in the personality research enterprise; we found the “10 Steps” paper to be interesting and informative, but it also raised six issues as follows:

In conclusion, although we support many of the ideals and goals of the 10 Steps paper, we are somewhat skeptical of their proposed solutions. The authors suggest that the first step should be to establish common goals for the field. We agree, but we would argue that the grand problem for our discipline concerns predicting significant behavioral outcomes (i.e., longevity, status, relationship quality, well-being). If we keep this goal in mind, then progress will follow.

I enjoyed reading this thoughtful and thought-provoking target article (Leising et al., 2022), and I wholeheartedly agree with the authors’ overall goal of improving the quality of personality research. Who would not agree with this goal? I also appreciate the steps they propose to move toward it. None of them are new, and some have been requested for decades. The authors are aware of this. The value added by the target article to previous requests is that they are all compiled into one paper, which I hope will generate synergistic impact. I also agree with the authors’ belief that a change in incentives will be the most effective way to improve research quality. Again, the proposal to reward research quality rather than the number of publications is not new. But if the steps to better research quality have been known for so long, why are we still seeing so many papers of mediocre quality? In my view, two factors have contributed importantly to previous failures at improving research quality: (1) the timing and scope of changes in the incentive structure and (2) the insufficient quality of committee work. Considering these factors will be crucial for preventing continued failures.

Leising, Thielmann, Glöckner, Gärtner, and Schönbrodt (2022) diagnose an important problem that plagues psychology: An incentive structure that places much value on researchers' quantity of scientific output. The authors propose a novel structure: Scientists can earn points by investing in more consensus-based, cumulative, open, and replicable science. The authors predict that such structure might result in an improvement of the scientific enterprise because it moves away from the current incentivization of speed and competition, which plausibly underlies much of the crisis of faith in the replicability of research findings. Like any prediction, this one also is liable to empirical scrutiny.

An incentive structure not only needs to produce the right collective outcomes but also fit the psychology of human motives. There are, broadly speaking, two major and partly overlapping basic motive taxonomies. Self-determination theory features a need for affiliation (which it labels relatedness), achievement (which it labels competence), and autonomy. Classical motive theory also features the first two of these needs, but adds the need for power, called the need for status by Anderson et al. (2015). Both taxonomies allow for individual differences, though perhaps to differing extents (Soenens, Vansteenkiste, & Van Petegem, 2015). Here, we take an eclectic, broad-brush approach and assume four basic motives (affiliation, achievement, status, and autonomy) that differ between individuals. So how does the proposal of Leising et al. (2022) stack up?

Through a motivational lens, Leising et al.’s (2022) analysis might be grounded in a working hypothesis that the status motive has become too dominant in science. In their search for fame and recognition, scientists do not cooperate as much as they should, and invest in quantity and speed rather than replicability and quality. In extreme cases, this can produce tension with the need for achievement, which is fulfilled if people truly master something new and complicated – for which there might be too little time in a hyper-competitive regime. The status quote also threatens the need for affiliation if it sacrifices cooperation for competition. Leising et al. (2022) do not entirely ignore the status motive: In their proposal, researchers can still compete for points and related prestige, but they have to do so by acting responsibly and collaboratively. This requirement appeals to submission to textbook methodology and ethical principles; that is, fair play.

A first question is whether it is indeed possible to channel the status motive towards collaborative goals. While there is research that shows that “communal narcissism” can indeed be focused on prosocial outcomes (Gebauer et al., 2012), the proof of the pudding is in the eating. In addition, we can only guess to what extent truly pioneering researchers have been driven by a need for status (e.g., in the case to discover the structure of DNA)? Can we expect the same pace of risky and time-sensitive breakthroughs under a more collectivist incentive structure, which rewards people for relatively safe endeavors? The benefits of the newly proposed system might still outweigh the drawbacks, and any change of system has of course winners and losers. Still, the issue requires careful analysis and empirical evidence.

If our motivational analysis is correct, the greatest question mark is the need for autonomy. While it is true that hierarchical power structures also limit the autonomy of lower-ranked scientists, the newly proposed system appears more collectivistic. That is, it establishes a set of “consensus statements” regarding methods and theories, and scientists are highly rewarded with credit points if they conform. However, what about researchers, who prefer to work alone and/or are ahead of their time? The authors do discuss how innovation would still be possible in such a system, but without supporting evidence, we cannot be sure. Some skepticism is informed by a motivational insight from economics: Any quality indicator of work can trigger strategic behaviors that render the counting metrics themselves obsolete (this is known as Goodhart's law; Chrystal, Mizen, & Mizen, 2003). We would be concerned if collectivistic approaches suppress diversity of opinion, which runs counter to ideas about science being a Darwinian-like process in which the generation of diverse ideas is followed by consensus-based selection (Simonton, 2003). In our view, there should be ways to reward contrarian scientists that consistently “run against the grain” – and do so with determination but without breaking the rules of the game.

This relates to our final point, which is about individual differences. We have speculated that the status quo seems well attuned to the need for status and autonomy (at least for the powerful). Switching to the new system might create a focus on affiliation (cooperation) and perhaps on research that is performed with greater precision, which would be welcomed by many. Perhaps this is a good trade-off. Still, it might be even better to create diversity in incentive structures, so that individual differences of researchers can be put to productive use. After all, it has been suggested that team diversity contributes to better outcomes (Rock & Grant, 2016). The challenge would then be to create an incentive structure that values the contribution of everyone with useful skills and potential, whether they are quirky contrarians or agreeable consensus builders, avid reviewers or bold explorers, shy background types or active spotlight seekers. Whether such a diversity-promoting incentive structure is compatible with a score-keeping system as proposed by Leising et al. (2022), without doubt also raising discussion and perhaps even conflict among contestants, remains to be seen. A personnel assessment system that discourages questionable behavior but does not get in the way of autonomy (initiative, courage, and academic freedom) sets a daunting task, and we think that personality psychologists are well equipped to contribute to solving it!

Leising et al. (2022) suggest strategies to improve the quality of research in personality psychology. These include the need to develop consensus regarding research goals, terminology, measurements, data handling, and the current standing on theory and evidence. The authors also suggest that the development of such consensus should be rewarded, and they provide concrete criteria for rewarding good scientific practices. However, some of these strategies might not be feasible, and at worst, might do more damage than good. Below, we critique some of Leising et al.’s (2022) suggestions for implementing a consensus-building approach.

Leising et al. (2022) anticipate that, with proper incentives, various groups will work together to articulate possible versions of consensus, and that these positions will then be considered and debated by the field. For some aspects of our science this is already happening: the past decade provides a good example of how a consensus was built around the idea that a significant portion of our results in psychology were not replicable. It helps that the evidence was so provocative and compelling (e.g., Open Science Collaboration, 2015), and yet even then there was initial disagreement about the extent of the replication crisis (e.g., Gilbert et al., 2016). Because of these reforms, some kinds of consensus relating to better reporting standards, robust analyses, controlling for post-hoc theorizing, and greater transparency in methods overall are now less controversial.

In contrast, we believe that moving towards a consensus on substantial theoretical matters, without a thorough understanding of the current diversity of positions, might be problematic for three reasons: it will be less persuasive; it might discourage lines of investigation that might have led to a more robust convergence; once we have it, we might find the consensus less flexible than we would want.

Outside the numerous “grand” theoretical positions outlined in the standard textbooks, there is a whole menagerie of mid-level theories, and it is a genuine question whether the full range of ideas deployed by contemporary personality scholars are adequately documented in the usual reviews of the literature. Theorizing, of course, is not a matter of popularity, but consensus building is a social process. Legitimacy is accrued if the procedure is seen as having involved a wide sample of colleagues from the field. So it matters what the starting conditions are for what kind of consensus emerges, and our contention is that maybe we do not yet have a good handle on the full range of ideas at play.

And because such breadth is necessary, it also matters who is involved in the process of consensus building. The validity and legitimacy of any consensus will depend upon the extent of collaboration between researchers from the global north and those from the global south. Psychological studies, including those in personality science, have always been mostly reliant on WEIRD (Westernized, Educated, Industrialized, Rich, and Democratic) samples, a problem adequately discussed elsewhere (Henrich, Heine, & Norenzayan, 2010; Rad, Martingano, & Ginges, 2018). The much touted “consensus” on the Big 5 illustrates the challenges for future concordances. Findings from large samples around the world generally corroborate a five factor structure, but not without significant exceptions (Church, 2016). It is a credit to the field that, in our own case, a less-WEIRD population like the Philippines features prominently as part of the cross-cultural evidence base for the Big 5 (e.g., del Pilar, 2017; Katigbak et al., 2002), but this is just a demonstration that a non-English Big 5 instrument can be constructed in a non-WEIRD culture. Future attempts at convergence should do better by encouraging the search and development of viable alternatives from non-WEIRD scholars to test current frameworks against; to include more theorists rather than just more populations.

A premature narrowing of conceptual and methodological horizons might itself be detrimental to progress. If the rationality of a research program depends on its ability, in the long-run, to identify lines of investigation that are more fruitful than others, there is persuasive work from Mayo-Wilson, Zollman, and Danks (2011) formally demonstrating that a research strategy that is rational for an individual investigator (or individual lab) might not necessarily be rational for the group (or the field) if adopted by all of its members. The upshot of this is that investigators can still converge by independently pursuing what they believe to be the best course of action, as long as the results of their investigations are freely and transparently shared within the field. This is not inconsistent with the aims of the target article, but more explicitly bottom-up: a consensus paper or convention might be thought of as merely the articulation of an already existing convergence rather than an attempt to legislate it into being. This is “consensus-finding”.

Leising et al. (2022) indicate that they are aware of the problems of reaching a sub-optimal consensus. They defuse these concerns by appealing to the idea that the consensus view should be constantly updated as the field develops. We find this view too optimistic. A consensus is political in nature, even when the content of the consensus is scientific. Franz (1997) documented the emergence of an international consensus on climate change and she observed that momentum towards a consensus gathered pace within a period when there was no significant change in the documented scientific findings. This implies that agreement was driven by factors outside of the data, factors arguably of a political nature. Because of this a scientific consensus, once established, might also be difficult to abandon since a change of direction could incur costs in terms of credibility and alliances. The cited example of defining the term “planet” is inapt: the change had no substantial impact on the theories and methods of astronomers, being mainly a cultural wrangle (Messeri, 2010). We cannot imagine that a vote to define the word “trait” would pass so sedately.

Given what we’ve said above, we advocate for the following counter-proposals:

Ten years into the replication crisis, psychologists have learned that we need to do better. Many initiatives have been proposed and implemented to improve research practices, but a neglected domain for improvement has been changing incentives. The target article (Leising et al., 2022) provides a concrete proposal for reshaping how we evaluate research, which they propose will better align incentives and improve research quality. I suspect other commentaries will focus their critiques on the particulars of the proposed point system for scoring research quality, which, although laudable for its specificity, contains some shortcomings related to its ability to properly reward research beyond very common research designs.

I will instead focus on where the target paper differs from other papers in this genre: its focus on fostering consensus. Leising et al. (2022) propose that “the field cannot go on without greater and more explicit consensus as to (1) what the most important issues to be investigated are, (2) how things shall be named, (3) how things shall be measured, (4) how data shall be analyzed, and (5) what the current state of theory and knowledge is” (p. 8). Given this focus on consensus, I was surprised that the authors gave only scant attention to meta-analysis, which is often upheld as a consensus building tool (Chan & Arvey, 2012). Here, I explain why meta-analysis has often failed to create consensus, illuminating a more general flaw in the Leising et al. (2022) vision of consensus building, and I describe what we should do instead to achieve a more cumulative psychology.

Meta-analysis is a tool for providing summaries of sets of studies, both in terms of central tendency (what a typical study looks like) and variability (how much studies vary from one another). Furthermore, meta-analysis is the quantitative part of a broader research synthesis method known as systematic review. Systematic reviews are designed to take stock of all current knowledge on particular research questions, and they involve a repeatable, comprehensive search strategy, coupled with evidence quality evaluation. Systematic reviews may include meta-analyses, but they can also be qualitative (see Corker, 2022 for additional discussion on strengths and weaknesses of both methods).

As such, systematic reviews and meta-analyses seem to be promising candidates for establishing consensus in a given research area. Indeed, Chan and Arvey (2012) said as much: “Meta-analysis may contribute to the advancement of knowledge and normal science … by facilitating the building of consensus in a field or topic” (p. 85). The idea is that a review will provide a birds-eye overview of current evidence on a topic, showing where consensus vs. disagreement exists in the knowledge base. Interestingly, although consensus would seem to be an unabashed good, historically there have been worries that meta-analyses would provide too definitive an answer to a question. If strong consensus was revealed through a meta-analysis, further research on a topic might be quelched: “Meta-analyses may declare a ‘winner’ or ‘loser’ before an ongoing trial is over” (Feinstein, 1995, p. 77).

Although promising as a consensus building tool, the reality of meta-analysis is less sanguine. Anyone who has attempted to perform a meta-analysis or systematic review will report how frustrating it is to discover how idiosyncratically researchers work. In most areas of psychology, there is little overlap between studies in terms of measurement, methodology, and even theoretical approach. Crucially, the lack of commonalities between studies sometimes makes synthesis well-nigh impossible. How can the results of such studies even begin to be combined?

Leising et al. (2022) lament this exact state of affairs and propose that we reward those who build consensus positions and use consensual methods in their research to encourage the proliferation of this kind of work. A researcher eager to adopt their recommendations might therefore gather several research groups together to engage in a meta-analysis or systematic review, with the goal of uncovering the group’s collective understanding of knowledge on a research question. But our eager colleague would almost immediately be met with a problem: in the presence of such diversity in measurement, methods, and theory, exactly where is consensus supposed to come from?

Put succinctly, my argument is the following. Given the current state of affairs in psychology – in which 50% or more of our results fail to replicate, the vast majority of measures are ad hoc or of unknown validity, theories are so vague as to be mostly unfalsifiable, and there is little to no overlap in approach between studies in supposedly related topical areas – neither meta-analysis nor any other currently known consensus generating procedure is going to be sufficient to allow us to fashion gold from lead.

If true, where does that leave us? Before we “quit the academy and make an honest living selling shoes” (Meehl, 1990, p. 237), we might consider the alternative course proposed by Paul Rozin twenty years ago. Rozin (2001) argued that detailed, precise descriptions of reliably occurring social phenomena should be the basis for later experimentation and model testing. In contrast with biology and other developed sciences, he argued, psychologists have often jumped straight to hypothesis testing and skipped over the rich observational research that could inform and shape those experiments.

Certainly, it could be argued that personality psychology has valued this kind of descriptive work more than some other subfields of psychology. Indeed, the existence of the Big Five as an organizing framework is a testament to how generative careful descriptive work can be. But given our current levels of progress and understanding, we could surely do even more such work. Is my argument a call for totally atheoretical, inductive work? No, because even descriptive work is theory-laden and must begin somewhere (Oreskes, 2019). But if theories are explanations of available facts, then to strengthen our theories – and therefore our consensual understanding – it would be nice to first have some reliable facts to explain. But attempting to jump straight to consensus on the basis of our current (highly flawed) research literature, as Leising et al. (2022) recommend, is likely to fail.

Efforts to improve research in personality psychology often prioritize the way science is conducted and overlook the importance of what gets studied, by whom, and for what reasons. Leising et al.’s (2022) proposal of ten steps towards a better personality psychology are certainly laudable recommendations for innovation in our field but, like many contemporary scientific reform efforts, they emphasize the how and largely overlook the what, who, where, and why of research. While progress in the conduct of scientific research is absolutely important, we should not pursue this progress without attending to the vital matter of science’s broader aims: improving our world.

Leising et al. (2022) propose a 10-step checklist that they argue will facilitate “a better personality science.” Although we agree with many of the proposed steps, whether the checklist separates “good research” from bad is an empirical matter. We therefore consider whether the checklist would have caught one of the replication crisis’s most infamous papers—namely, Bem’s (2011) “Feeling the future” in the Journal of Personality and Social Psychology. Table 1 demonstrates the difficulty we faced in coming to a consensus on the score to assign Bem’s paper, which is the first of our several criticisms.

The midpoint of our scores was 12.0 (building consensus: 3.3, using consensus: 2.3, formalization: 0.0, preregistration: 0.0, replication: 1.0, informativeness: 3.5, and open science: 0.8). Does this score separate the bad from the good, the reproducible from the irreproducible? These questions are difficult to answer for at least two reasons. Since most published research hasn’t been scored, individual scores are difficult to contextualize. However, even if most published research was scored, and a consensus between scorers was reached, we contend that conceptual problems built into the checklist render scores difficult to interpret, and that the scope of the checklist misses important things.

A CRediT Alternative

Creating a better science requires a shift in academia’s reward structures. The current system rewards producing more publications with little reference to contributions to those publications. Even with Leising et al.’s (2022) ten steps toward a better science, researchers who produce more “quality” research with minimal contribution will be most rewarded. The contributor roles taxonomy (CRediT¹ )—fourteen high-level roles that specify the researcher’s contribution to a publication—offers a simple yet effective way of weighing the quality of researcher contributions rather than quantity alone. We propose adding a CRediT section to CVs to move us toward contribution-based standards (see Figure 1 for an example).

Click to enlarge

Figure 1

Example of CRediT Section in a CV

Note. “X” indicates contributorship role, “–” indicates not applicable for a given project, and blanks indicate roles not undertaken. “Concept” = Conceptualization; “Data Cur.” = Data Curation; “Invest” = Investigation; “Method” = Methodology; “Admin” = Project Administration; “Supervis.” = Supervision; “Vis.” = Visualization; “Draft” = Writing: Original Draft; “Review” = Writing: Review & Editing.

Besides making a researcher’s contributions clear, the CRediT section offers insights into expertise and team science. A quantitative role may include “analysis.” A supervisory role may include “conceptualization” and “funding.” “Writing: original draft” indicates that the researcher contributed to their content area. Different combinations of contributions reflect different yet important roles in team science.

There is no question that there is a need for a better personality science. Less clear, however, are the answers to what we think “better” actually means and how exactly we get there. At the 2021 conference of the Association for Research in Personality, all of the invited sessions focused on how to create a more diverse, equitable, and globally-minded personality science. The conference is largely focused on North American representation and perspectives, but there were other voices as well, and one of the invited sessions pertained to diversity and inclusion in personality research in Europe. Collectively, the sessions made plain that there is a hunger for change, and a need for change that, if not addressed, could derail the field. Attending that conference and reading the target article by Leising et al. (2022) left us with the sense that we have entered into two different understandings of reality.

Sessions at the conference focused heavily on reflecting on and improving our methods to generate a stronger knowledge base, just as Leising et al. (2022) did. The point of departure, however, is that improving our methods are not enough. Like many of our colleagues at the conference, we emphasize the need to understand how our methods are intertwined with the tremendous lack of inclusivity and diversity in our field, and how they reify existing power structures. Improving our methodological approach can not only lead to more robust conclusions, it can also make us think more deeply about what we do and how we do it. Perhaps most importantly, doing so will help us to understand and acknowledge the limits of what we do. This kind of humility and reflection is at the core of “steps towards a better personality science,” though as we elaborate, this might be characterized as more of a sea change than steps.

Although we are fans of common ground and understand the need to speak the same language and share tools to work together, we found the emphasis on consensus decision-making much too strong and much too premature. First, as we elaborate below, we do not yet have the appropriate data or methodological development to make these decisions. Second, there was no proposal to integrate the necessary work of reflecting on the biases that would impact such decisions. Third, there was no discussion of who would be in the room to make these consensus decisions, or of the power dynamics intrinsic to our existing systems that are perpetuated with such practices. The risk of Leising et al.’s (2022) proposal is that decisions made by a select few will not only contain unacknowledged bias, but will also systematically marginalize and restrict the very kind of work that is needed to understand our own biases in measurement and the experiences and voices that have been left out of our science. The risk is the potential for no meaningful change at all.

This point is not hypothetical, as it is indeed already what we see in the field. The majority of our common tools have not been subject to sufficient tests of whether they are appropriate across diverse groups of people. In her talk at the conference, Monisha Pasupathi (2021) eloquently argued that if one takes diversity in the human condition seriously, we will need to understand that our common constructs, methods, and measures are not appropriate or relevant for all people. We are not physicists. We are not evolutionary biologists. We are scientists who study humans in all their messy complexities, which include their messy and complex cultural-historical contexts. We will have to realize the limits of what we are doing and do something different, which means we need to slow down and deepen our understanding, rather than prioritize rushed consensus based on limited knowledge.

The overarching problem is that the paper is targeted at the wrong level. Before we can make the large-scale movements towards the better science Leising et al. (2022) have envisioned, we need to have a much broader scope on where we have gone wrong - beyond the under-powered and unreplicated studies that have given rise to the current crisis. We need to understand the power dynamics and inequitable systems that supported and celebrated those scholars and programs of research that were based on faulty data and designs. We need to understand the historical context of why we value what we value (e.g., quantitative over qualitative; Syed, 2021a). We need to understand how our biases and values as researchers shape why we study what we study (e.g., Dunn et al., 2021). We need to take a good hard look at what is missing in our science (Syed, 2021b). We need to do the work.

Engaging in this kind of reflection highlights how the current emphasis on “good science” comes from an entirely quantitative perspective. Descriptive and qualitative work are the very kinds of tools that will help us to understand the potential limits of our theories and measures, the appropriateness of our measures for different populations, and the variety in the human condition that we should be capturing if we are doing our job. Unfortunately, the scoring system proposed by Leising et al. (2022) places this kind of descriptive and qualitative work at an even greater disadvantage. Qualitative research - mentioned only very briefly in passing - would receive the very lowest of scores, not even scorable on multiple components of the system.

We also find it ironic that the authors argue against the quantity-focused form of assessment that dominates the field, instead advocating for an approach that is more nuanced and grounded in the actual practices of the field (even if incomplete). Perhaps the same appreciation of the bias of quantitative work, and the potential benefits of qualitative work, can be applied to our science.

In short, we don’t offer tidy solutions because the first step is serious reflective work. We need to truly examine the science we have created, and let some of it go. Who knows what we might discover.

The target article (Leising et al., 2022) offers ten helpful steps toward a better personality science, but I believe several more steps will help advance the field even further. These suggestions come from first taking a step back to allow broader perspective.

The target article focuses mostly on current topics in open science and knowledge cumulation such as sharper theory and prediction, and better statistical power and reliability. The authors rightly propose appreciating quality over quantity, especially in determining the value of the work and of the scholar. Perhaps ironically, although the target article nicely refers to the “replication crisis,” it would be helpful to go back nearly half a century to the beginnings of the Personality and Social Psychology Bulletin (PSPB journal), which discussed some of the same issues as a “crisis” in our field.

In one article well-read nearly a half century ago, the then-prominent psychologist Zick Rubin discussed “On Measuring Productivity by the Length of One’s Vita,” in which he bemoaned the rating of Psychology departments by counting the number of articles published by their faculty (Rubin, 1978). He concluded his piece by saying, “I am not an opponent of quantification in psychology. It may even be useful for psychologists to develop, if they can, “objective” indices of the quality of psychological work. But, for heaven’s sake, let’s not measure ourselves by the number of articles we publish” (p. 198). Similarly, a founder of our field, Muzafer Sherif (1977), bemoaned the wheat versus chaff ratio in the skyrocketing volume of research and publication. Sherif went on to discuss many of the important issues now being reiterated by Leising et al. (2022) in the current era.

So I propose that we need another criterion added to the steps proposed in the target article. It might be termed: “Contribution to the comprehensive cumulation of knowledge and understanding.” That is, published research should be more highly valued if it has an extensive knowledge of the century-long history of our field and builds deeply upon it, in an integrative fashion. Such an accomplishment does not simply mean citing relevant, important articles from the past, but also means keeping (or resurrecting) the best ideas and showing how they can now be further refined given new theories and new research methods.

A second step that would extend and complement the suggestions offered by Leising et al. (2022) is partly derivative from rightful concerns with the emphasis on the use of publication counts and large grant funding to decide which research projects (and which Psychology departments) are superior. This comparison process necessarily sets up a competition. Of course, some researchers and some departments are undoubtedly better at advancing the field of personality, and should be rewarded (or at least recognized) for their achievements. But the system does not have to be one of severe competition.

In part, this new structure would reward the incorporation of and cooperation with traditionally under-represented people and points of view, bringing both new perspectives and increased external validity. Further, if research publications are better appreciated if they involve multiple, cooperating laboratories—including cross-cultural considerations—then the quality and generality of personality research will likely improve. Thus teamwork could be an explicit criterion for evaluating faculty research.

That is, there sensibly could be less emphasis on individual comparisons and more attention to teamwork. This criterion would not have to apply to all faculty, because some sorts of scholarship are not as amenable to team projects. But overall lessened concern with constant comparative evaluation of individuals would likely help focus researchers on long-term quality.

Third and finally, the quality of research (and investigators) can be looked at not only in terms of instantly-applied criteria but also as to how impactful the work is on the field of personality science over time. Some research reveals instantly recognizable advances, but other research needs to pass the test of time, and the rewards can come later. To use an example from my own research: When, in 1993, we published an article finding that conscientiousness, in both childhood and adulthood, could predict longevity across the decades in longitudinal data (Friedman et al., 1993), no one knew whether this was a function of the particular cohort, or sample, or personality measures, or perhaps was even a chance finding. It was only after multiple other research teams turned to examining this finding and increasing their attention to the whole field of personality and health and longevity that the full impact of the initial findings could be appreciated, two decades later (Friedman & Kern, 2014).

Note that long-term retrospective judgment of research importance (e.g., 5 years, 10 years or 20 years later) would still impact promotions (advancement) of faculty. Promotions to tenure are usually based on 5 or more years of work, promotions to full professor are usually based on 10 or more years of work, and promotions to the very senior research positions (e.g. endowed chairs) are often based on 20 or more years of work. The point here is that more explicit emphasis on the long-term importance and value of personality research would encourage researchers to remember to take a long view of the possible long-term impact of high quality research.

In sum, the three steps sketched here are usefully included in an explicit enumeration of considerations of how to best reward high quality research in personality science.

Those who choose to read the target article by Leising et al. (2022) will be glad they did. I was struck by several of its features, not the least of which being the sheer amount of time, energy, and expertise required to pull this article off, and pull it off so well. Papers like this work best when they aspire for root-and-branch overhauls of the way things are, rather than minor revision. Our authors could be accused of little else. When presented as a unified whole, their ten steps signal nothing less than a seismic shift in the way research in personality psychology is to be conducted, evaluated, and rewarded.

The authors offer a tight, logical path forward for personality psychologists interested in conducting empirical research with the necessary credibility. Such specificity carries numerous benefits, many of which are outlined in detail by the authors themselves. It also leaves certain types of research, types of research that most assuredly should have a place in personality science, on the outside looking in. I am thinking here of personological pursuits examining whole persons and lives. Although researchers doing this type of work may have something to gain by engaging with our author’ first five steps, the ‘back five’ appear to be written with someone else in mind.

Leising et al. (2022) present an ambition plan for improving personality science. Their paper begins with a nuanced picture of what “good” science should look like, including greater openness about its shortcomings. I agree that it would be great to see criticism towards our own past work being embraced as a strength rather than a weakness.

Leising et al. (2022) also raise important concerns about the current focus on numbers of publications, citations, and the amount of funding received. This often plays out favorably for researchers who had access to good-quality data early in their career and gives such researchers an unfair head start. I am among those who have benefitted from this broken system. Furthermore, I agree that these numbers are easily manipulated, for example by researchers who disproportionally suggest citations of their own articles when serving as editors and reviewers. Therefore, I agree that we need drastic changes.

However, I believe that the steps and reward scheme proposed by the authors do not address some other underlying problems of our field. I propose that to improve personality science, we need to reflect on the scientific paradigm dominating the field and – related to that – the lack of diversity (of all kinds) among personality researchers.

In personality science, there is a predominant focus on quantitative approaches. For example, the rise of the Five-Factor Model was clearly driven by a belief in the relative objectivity of such approaches (i.e., factor analysis; Goldberg, 1990). Leising et al.’s (2022) procedure of counting “good” qualities of articles, with more “important” qualities weighing heavier, further fits in this tradition. These practices align with post-positivism, representing a view that significant, substantial, and replicable effects as observed in statistical analyses represent an objective truth, or at least a close approximation thereof.

Post-positivism can be mistaken for being the only “real” scientific paradigm, which can lead to reduced interest in, knowledge about, and appreciation of alternative approaches to science. Perhaps for this reason, Leising et al. (2022) misrepresent qualitative research with their comment that ‘…we think that qualitative research may benefit substantially from stricter formalization. A prime example would be the formalization of individual belief systems.’ First, “qualitative” methods represent a wide variety of methods. For many of those methods, there are guidelines for what should be reported and how (e.g., Levitt et al., 2018). Second, qualitative researchers often already make their individual belief systems explicit, for example by referring to theories that imply adherence to a particular belief system (e.g., Critical Race Theory; Delgado & Stefancic, 2017). Therefore, qualitative researchers do not need recommendations on stricter formalization. Instead, a greater appreciation of scientific paradigms other than post-positivism, and methods other than quantitative ones (i.e., qualitative methods, mixed methods), may lead to better, more inclusive, personality science.

Other scientific paradigms, such as constructivism (e.g., personal construct theory; Kelly, 1955), have previously been used for the study of personality. More recently, research aligned with the transformative paradigm (e.g., Mertens, 2007) or using practices from different paradigms (the pragmatic approach; e.g., Feilzer, 2010) seem to become increasingly common. Reviews by Arshad and Chung (2022) and Atherton et al. (2021) are examples of this. The reward scheme of Leising et al. (2022) may underappreciate such work, especially when pursued using qualitative methods. For example, criterion 9b (’has an expected type I error rate of ≤ .05 and type II error rate of ≤ .20, based on realistic effect size estimates’) can only be applied to quantitative research.

From paradigms other than post-positivism, Leising et al.’s (2022) call for consensus regarding research goals, terminology, measurement practices, data handling, and the current state of theory and evidence can also be perceived as problematic. For example, the transformative paradigm holds ‘that realities are constructed and shaped by social, political, cultural, economic, and racial/ethnic values indicates that power and privilege are important determinants of which reality will be privileged in a research context’ (Mertens, 2007, p. 212). This perspective directly points to potential problems caused by underrepresentation in personality science of most groups other than white male-identifying researchers from North American and Western European countries and represents the underlying philosophy of science guiding recent critical reviews of our field (e.g., Arshad & Chung, 2022). Arshad and Chung (2022) provide excellent recommendations for how to address these, and related, problems.

Systemic inequality beyond personality science may have affected even the very constructs we focus on. A good example of this could be the Big Five. Derived from lexical analyses on the English language (Goldberg, 1990), a still large set of adjectives were grouped into a more manageable set of broad dimensions using factor analyses (the Big Five). However, Goldberg’s (1990) samples primarily included English-speaking college students. To get into college, one needs to fluently use language fitting with what is privileged in one’s educational systems. Furthermore, what is considered “standard” language is decided by humans with language use by marginalized groups often considered (by the dominant group) as inferior and non-standard (e.g., Siegel, 2006). Therefore, words that are meaningful to many in describing personality likely were excluded from lexical analyses. Hence, a five-factor structure of personality might reflect a useful way of thinking about broad individual differences among a particular (relatively privileged) segment of the population in countries where the lexical method and subsequent factor analyses were employed, but Big Five measures may poorly resonate with others. This is not to say that the Big Five are not useful but to think that they came about in a bias- and value-free manner is naive.

This commentary is meant as an illustration of how relying on consensus for evaluating “quality” could lead to perpetuated inequality. The past and (to a somewhat lesser degree) current climates have been rather non-inclusive in personality science (Atherton et al., 2021), which likely led to values upheld by a small group being privileged. From any other perspective than a post-positivist one, these omissions are problematic. Therefore, we can only make personality science better if we recognize and reflect on the scientific paradigms we use and become more inclusive.

Leising et al. (2022) suggested ten steps toward a better personality science. Without a doubt, following or implementing the spirit underlying these ten steps would further improve the quality of personality science, it would make life easier for a lot of (young) researchers, and would culminate in a more integrative and manageable research field. We agree with this spirit and most of these propositions, cannot comment on some, but disagree with one. Specifically, Leising et al. (2022) suggested that the field should pursue common measurement practices, especially with respect to the measurement of the Big Five personality traits. The authors write "there needs to be a single, standard way of assessing the Big Five personality factors, as well as standard ways of assessing everything else that is considered worth assessing." (p. 12). The authors suggest that this should be achieved to avoid jingle-jangle fallacies, to facilitate meta-analyses, and to allow for cumulative knowledge building. Although we strongly agree with the goals that should be achieved, we disagree with the proposed solution, specifically with the idea of one standard measure to assess personality. Instead, we suggest an alternative that could be implemented for any type of test that uses individual items (e.g., statements about oneself as in personality tests, stimuli in reaction time tasks, intelligence tests items).

Psychological tests that assess one variable are usually developed for a specific purpose (e.g., comparing groups vs. assessing individual levels), and for a specific population (e.g., younger people, older adults, from different cultures) (Ziegler, 2014). To achieve this, items are, for example, tailored towards the language level of the targeted participants or to reflect specific living circumstances. As a result, different measures fulfil the needs of different researchers. Using just one tool would leave a gap in our tool kit. To provide a metaphor: There is no one tool to measure distance. At home, where precision does not matter too much, one can use a simple yardstick, for longer distances, a laser distance meter can be used. Of course, this example neglects the problem of different scales of measurement in psychology which potentially prohibit comparing results derived with different measures. A risk we see when using one authoritative measure alone is an artificial reduction of the potential construct breadth. The field of personality research underwent a similar development before, when the original Big Five were developed, based on items and adjectives selected by specific researchers. The result was that the field, after years of otherwise very productive work, realized that many, often darker aspects of personality, were not reflected (Thalmayer, Saucier, & Eigenhuis, 2011).

Instead of forcing the same measure upon everyone, we would suggest providing, in the long run, an online repository of measures and data. Such a data base should ideally contain individual items and their scores, plus background information on the sample characteristics and the survey method (e.g., rating scale used, language, introduction, online vs. offline). Similar to automatically updating websites such as curatescience.org, such a website could generate item and scale statistics for different samples, estimates of reliability, validity, and even measurement invariance across different populations. To this end, the website would need to match uploaded items to existing scales or other items, compute (latent) scale composites, and provide (latent) correlations to other scales, if available.

Such a website or repository would ideally provide three services: First, researchers could search for items and scales that have been used and validated in samples similar to the ones they intend to collect. Second, items (without data) could be uploaded, and be compared to existing items based on linguistic features, thereby avoiding the jingle-jangle fallacies that Leising et al. (2022) point out as a potential threat to the test score's validity (similar to https://rosenbusch.shinyapps.io/semantic_net/; Rosenbusch, Wanders, & Pit, 2020). Finally, researchers could upload their data and their items to share both at the same time in a well-documented format.

From our standpoint, such a dynamic approach has several advantages over one authoritative measure to rule them all:

Finally, we envision an even more ambitious goal: For each scale, it could be possible to generate a unique identifier or code (similar to a DNA sequence or a DOI) that unambiguously represents the items that have been used in a scale, the rating type scale used, and even the instructions. Published in an article, this code could then be submitted to the website, generating the exact questionnaire that was used (e.g., for reuse in online survey platforms such as formr.org, or as a word-template), additional statistics, and also further publications in which the scale has been used.

We agree, this proposal is not modest, and it will require financial investment as well as the investment and commitment of our community to share their data. Luckily, however, this project could be launched right away with the existing open data from personality research (Horstmann, Arslan, & Greiff, 2020). If successful, it could serve as a beacon for the integration of different measures in personality psychology and beyond.

We fully agree with the general tenor of the article by Leising et al. (2022)—that research quality is vital—as well as with most of the suggested actions (e.g., implementing Open Science Practices and conducting more replications; e.g., Nelson et al., 2018). Leising et al. (2022) outlined central issues and specific next steps for increasing the rigor and relevance of our discipline—including those that concern the measurement of constructs (Step 3). In fact, measures are a key building block of psychological science—literally. With our measures, we collect data that are subjected to statistical analysis, which then provide results used to draw inferences about hypotheses, which ultimately inform theories to organize “knowledge” in psychology (e.g., Bacharach, 1989).

Yet, always pursuing “standard ways of assessing everything else that is considered worth assessing” (Leising et al., 2022, p. 12) can leave crucial (yet overlooked in their article) benefits untapped. Admittedly, pursuing standards in measurement can make perfect sense, for instance, when various measures of the same construct exist but differ in quality. Objectivity, reliability, and validity have to be examined, so that a single best measure, if it exists, can prevail (Leising et al., 2022; we also agree with the problems associated with unsystematically adapting measures; pp. 11–12). However, the opposite approach—systematically pursuing a “diversity” in measurement—has notable benefits unconsidered by Leising et al. (2022). Diversity in measurement can contribute to rigor and informativeness of psychological science—precisely the kind of goals addressed in their article. It depends on the aims of a research endeavor whether a standard measure is needed or not (see Leising et al., 2022, p. 13).