This is an open access article distributed under the terms of the Creative Commons
Attribution License (People can comment on almost everything online nowadays. On social network sites, for instance, people can, with little censorship, comment on anything they hear, read, watch, or whatever is on their mind. Similarly, on webpages such as Google Maps or Yelp, people can comment on businesses, places they have visited, or whatever they have consumed. In online research, participant recruitment platforms such as Amazon Mechanical Turk or Prolific Academic allow participants to leave private comments for researchers or to discuss things publicly in forums. Considering online research more generally, researchers may even ask participants directly if they have any comments they would like to add before finishing a study (Schonlau, 2015). By doing so, researchers can collect valuable information regarding participants’ thoughts about a study such as whether it was too long, if some questions or tasks were difficult to understand, or if it was fun and engaging to participate (McLauchlan & Schonlau, 2016). At the same time, participants may point to interesting new research questions or highlight important issues that the researchers failed to consider (Decorte et al., 2019; O’Cathain & Thomas, 2004). Ultimately, all this information can be used to improve (online) research. In the context of panel studies, such information may further be used to decrease attrition by allowing researchers to identify and subsequently adapt or discard any features of a study that participants find particularly frustrating or bothersome, as well as to add features that participants might request (McLauchlan & Schonlau, 2016). Moreover, giving voice to panel participants by providing the mere option to comment might increase their engagement and, in turn, the likelihood that they will continue to participate.
As in other online contexts, such as social network sites (Liu & Campbell, 2017), some people will make use of the opportunity to comment in online scientific studies while others will not, which can potentially influence the type of comments made. Indeed, if only a small subset of participants with very specific personality characteristics tend to comment (in a certain way), the comments made may not provide researchers with an accurate picture of how participants more generally perceived their study. In other words, researchers might get a biased view of their study and, in turn, end up making changes that only cater to a small group of participants, are largely unnecessary, or, at worst, negatively affect many other participants.
In order to take full advantage of the comments participants make and to determine whether changes should be made on the basis thereof, researchers thus need to know if, and if so, to what extent, those who comment differ from those who do not, as well as whether participants with certain personality characteristics are more likely to write certain types of comments. To the best of our knowledge, these two aspects have not been addressed, so far. Herein, we test whether inter-individual differences in participants’ personality are related to their tendency to comment as well as to the types of comments they make. More specifically, we link participants’ levels in the HEXACO dimensions to whether or not they commented in an online scientific study, and if so, how much they wrote, as well as whether they commended the study, wrote a neutral comment, or complained about the study.
The HEXACO Model of Personality
The HEXACO Model of Personality summarizes people’s personality in six basic trait dimensions: Honesty-Humility, Emotionality, Extraversion, Agreeableness vs. Anger, Conscientiousness, and Openness to Experience (Ashton & Lee, 2007; Zettler et al., 2020). Three HEXACO dimensions—Conscientiousness, Extraversion, and Openness to Experience—show strong conceptual and empirical overlap to their Big Five counterparts, whereas Emotionality and Agreeableness vs. Anger are rotated variants of Big Five Neuroticism and Agreeableness, respectively (Thielmann et al., 2022). More precisely, Emotionality differs from Neuroticism especially in that it, among other characteristics, captures variance with regard to one’s tendency to be sentimental and close to others (which is rather captured by Agreeableness in the Big Five framework). Agreeableness vs. Anger, in turn, differs from Big Five Agreeableness especially in that it, among other characteristics, captures variance with regard to one’s tendency to feel irritability and anger (which is rather captured by Neuroticism in the Big Five Framework; Thielmann et al., 2022). The HEXACO model also includes Honesty-Humility as a sixth basic trait dimension, representing people’s “tendency to be fair and genuine in dealing with others, in the sense of cooperation with others even when one might exploit others without suffering retaliation” (Ashton & Lee, 2007, p. 156). Correspondingly, Honesty-Humility shows substantial negative relations to aversive personality characteristics and outcomes in the realm of unethical behavior (for meta-analyses, see Zettler et al., 2020).
The Link Between Personality and Online Commenting Behavior
Several studies found links between people’s personality and their online commenting behavior. With regard to the Big Five, research has found Neuroticism (Seidman, 2013), Extraversion (Choi et al., 2017; Lee et al., 2014; Liu & Campbell, 2017; Seidman, 2013; J.-L. Wang et al., 2012; K. Wang et al., 2018), Agreeableness (Choi et al., 2017; Lee et al., 2014; Seidman, 2013; J.-L. Wang et al., 2012; Wu & Atkin, 2017), and Openness to Experience (Barnes et al., 2018) to correlate positively with people’s tendency to comment on social network sites, whereas Conscientiousness has been reported to be negatively associated with this tendency (Buckels et al., 2014; Lee et al., 2014; Wu & Atkin, 2017). Relatedly, research has observed a positive relation between, on the one hand, Extraversion, Agreeableness, and Openness to Experience, and, on the other hand, people’s intention to write product reviews online (Picazo-Vela et al., 2010). While each of the Big Five traits thus has been linked to people’s tendency to comment online, relations appear to be particularly robust across studies regarding Extraversion and Agreeableness. Moreover, some studies have investigated links between people’s personality and the types of comments they write. Koban et al. (2018), for instance, found Agreeableness and Openness to Experience to correlate negatively with uncivil commenting intentions on Facebook, and Sorokowski et al. (2020) reported Psychopathy to be positively linked with hateful commenting on Facebook (for similar research, see, e.g., Beckert & Ziegele, 2020; Buckels et al., 2014).
Comments in Online Scientific Studies
The specific context of online scientific studies differs substantially from that of other online contexts. The comments participants make are, for instance, rarely public and typically non-interactive (i.e., researchers tend not to respond to comments and often cannot even do this due to collecting data anonymously). Generally, only little is known about who comments in online scientific studies as well as whether specific personality characteristics are related to certain types of comments.
In fact, only a few studies have investigated participants’ commenting behavior in online scientific studies, and these studies have largely focused on what participants write rather than who comments. In one study, Schonlau (2015) categorized participants’ comments from two large panel studies and found that while most participants did not comment, those who did wrote more neutral and negative comments as compared to positive ones. More recently, Decorte et al. (2019) categorized participants’ comments from a cross-national online study of small-scale cannabis growers and found that most participants did not leave a comment, but that those who did, contrary to the findings of Schonlau (2015), wrote more positive than negative comments. In the light of the potential benefits of knowing who is more likely to comment in a certain way, more research investigating who actually comments in online scientific studies as well as who writes certain types of comments is needed.
The Present Investigation
Relying on data from a repeated cross-sectional online COVID-19 survey, we herein investigate these questions, crucially extending current knowledge on participants’ commenting behavior in the context of online scientific studies. Based on previous theorizing and robust links between Big Five Extraversion and people’s tendency to comment online (Choi et al., 2017; Liu & Campbell, 2017; K. Wang et al., 2018), as well as between HEXACO Extraversion and an overall proclivity for positivity (for theorizing and meta-analytic findings, see Zettler et al., 2020), we hypothesized that:
-
Hypothesis 1: Participants with higher levels of Extraversion are more likely to comment.
-
Hypothesis 2: Participants with higher levels of Extraversion are more likely to commend.
In line with the conceptualization of HEXACO Agreeableness vs. Anger as a basic dimension capturing characteristics such as being forgiving, lenient, patient, and tolerant vs. being choleric, ill-tempered, quarrelsome, and stubborn (Ashton & Lee, 2007), we further hypothesized that:
-
Hypothesis 3: Participants with higher levels of Agreeableness vs. Anger are more likely to commend.
-
Hypothesis 4: Participants with lower levels of Agreeableness vs. Anger are more likely to complain.
Not stating any directional hypotheses, we also tested all other potential links between, on the one hand, age, gender, and the HEXACO dimensions and, on the other hand, commenting behavior in terms of commenting or not, and if commenting, comment length, commending, writing a neutral comment, or complaining.
Open Science Statement
This study was pre-registered after data collection had ended, but before analyzing the data (see Lilleholt et al., 2020). The analyses presented herein do not deviate from our pre-registered analysis plan. Any analysis that was not pre-registered is labelled as exploratory. Data and analysis scripts are available in the Supplementary Materials.
Method
Procedure
Our data comes from the COVID-19 Snapshot MOnitoring in Denmark (COSMO-Denmark; https://cosmo-denmark.dk/). From March 2020 to September 2021, COSMO-Denmark assessed Danish citizens’ perceptions, emotions, and behavioral reactions to the COVID-19 pandemic via both a panel and a repeated cross-sectional study (Zettler et al., 2021). Herein, we use the data from the fourth measurement occasion (week of April 4, 2020) to the sixteenth measurement occasion (week of September 13, 2020) of the repeated cross-sectional study. The repeated cross-sectional study included the Brief HEXACO Inventory (BHI; de Vries, 2013) as well as an opportunity to comment on the study at the very end of it (“If you have any other comments regarding the survey, you can write them here:”).1 A screenshot of the “comment page” is provided in Figure S1 in the Supplementary Materials. The “comment page” was preceded by pages with various questions that differed from measurement occasion to measurement occasion but was always followed by a “thank you page” that thanked participants for their participation.
The sampling procedure was as follows: In 2018, following data handling approval, the last author received contact information for a, concerning age and gender, representative sample of 100,136 adult Danish citizens via Statistics Denmark. From this sample, between 4,999–7,500 people were randomly invited every week (from May 11, 2020: every other week) to participate in the repeated cross-sectional study of COSMO-Denmark via the official digital mail system in Denmark, e-Boks (https://www.e-boks.com/danmark/en). The studies were set up in formr (Arslan et al., 2020). Across measurement occasions, some variables were assessed consistently while others were only measured once or twice.2 Responding to the repeated cross-sectional study took around 25–35 minutes typically. Participants were not compensated for their participation but were informed that their responses would contribute to the advancement of science and understanding of Danish people’s responses to the pandemic (including a webpage reporting about selected variables).
Participants
We only use data from participants who completed the repeated cross-sectional study, provided information about their age and gender, and had no missings on the BHI. A total of 8,809 participants fulfilled these criteria. This subsample (Mage = 56.38, SD = 15.59 years; 55.4% women, 44.3% men, 0.3% other) is almost perfectly representative of the full Danish COSMO sample in terms of age and gender (N = 32,228; Mage = 56.05, SD = 15.87 years; 54.5% women, 45.3% men, 0.3% other). Moreover, in terms of age and gender, this subsample is fairly, although not fully, representative of the entire adult population of Denmark in 2020 when the study was conducted (N = 4,666,625; Mage = 49.38, SD = 19.01 years; 50.6% women, 49.4% men).3
Measures
HEXACO Dimensions
Participants’ HEXACO dimensions were assessed via the BHI, translated by us into Danish. The BHI assesses each HEXACO dimension via four items only, one item referring to each of the four facets per factor in the HEXACO conceptualization (see https://hexaco.org/). A sample item is “I like to talk with others” (Extraversion Sociability), and the answer scale ranged from 1 = “Strongly disagree” to 5 = “Strongly agree”. The explicit aim of the BHI is to assess the HEXACO dimensions both broadly and briefly—to maximize the content validity of the scale while keeping it as brief as possible—partly explaining the low internal consistency estimates (de Vries, 2013). Specifically, Cronbach’s α/McDonald’s ω for the HEXACO dimensions in this study were: .37/.37 (Honesty-Humility), .35/.35 (Emotionality), .63/.65 (Extraversion), .47/.47 (Agreeableness vs. Anger), .49/.49 (Conscientiousness), and .53/.54 (Openness to Experience). While some of these estimates are low or—especially concerning Honesty-Humility and Emotionality—very low, similar ranges have been reported by other studies using the BHI (e.g., Garbe et al., 2020; van Sintemaartensdijk et al., 2022). Despite some low internal consistencies, the BHI factors have been found to show concurrent validity in terms of correlating relatively highly with their respective counterpart factors of both the 100-item and the 200-item version of the HEXACO-PI-R (r’s = .57–.84; de Vries, 2013; Julian et al., 2022; Schumacher & Zettler, 2019). Moreover, while factors in short personality inventories often have lower internal consistencies than their respective counterpart factors in longer inventories, they generally seem to have good predictive validity for a wide range of relevant criteria (Rammstedt et al., 2021). This notwithstanding, it is important to recognize that the low internal consistency estimates of the BHI factors are bound to attenuate any relations that might exist between these factors and the observed commenting behavior considered herein (Salkind, 2010). All effects reported herein should thus be seen as fairly conservative estimates of the tested relations.
Comment Categorization
We first dummy-coded participants who wrote a comment (1) and those who did not (0). Next, all comments were categorized into mutually exclusive categories based on their valence. In particular, comments with a neutral tone overall were categorized as neutral, whereas comments with a positive or negative tone overall were categorized as commendations and complaints, respectively (for examples, see Table 1). In cases where the valence of the comment was rather ambiguous, an overall assessment was made. The categorization of all comments was done by two research assistants who did not know about the purpose of this research. To ensure consistency in the categorization process, we first instructed the research assistants on how to categorize the comments and then asked them to categorize 25 randomly selected comments. We then calculated Cohen’s Kappa to check the inter-rater reliability. Because Cohen’s Kappa was below .60 for the neutral category, we set up a meeting with the research assistants and resolved any disagreements about the categorization of the first 25 comments. Subsequently, the research assistants were asked to categorize 25 new randomly selected comments, after which we again calculated Cohen’s Kappa. This time, Cohen’s Kappa was found to be above .60 for all three categories, so that the research assistants proceeded to code the remaining comments. The inter-rater reliability for the remaining comments were found to be substantial for the neutral category (Cohen’s Kappa = .69), and very high for the commendation (Cohen’s Kappa = .82) and complaint (Cohen’s Kappa = .90) categories. Relatedly, the overall inter-rater reliability across categories was found to be very high (Cohen’s Kappa = .85). Finally, any disagreements were settled in a meeting between the two research assistants.
Table 1
Examples of Commendations, Complaints, and Neutral Comments
| Example | Commendation | Complaint | Neutral |
|---|---|---|---|
| Example 1 | “Thank you for giving me the opportunity to express my opinion” | “You are using way too many public resources on something that is useless” | “I hope the survey shows something that can be used“ |
| Example 2 | “Interesting project. I was happy for the opportunity to participate” | “Some of the questions were difficult to answer” | “Some answers would have been very different 6 weeks ago. I have based my answers on the present” |
| Example 3 | “A nice an easy survey. Well done” | “Several times I think there were too few response options” | “My answers are probably affected by the fact that I am pregnant” |
Note. Please note that the examples have been translated from Danish to English. In doing so, we aimed to retain the meaning of the comments, rather than simply translating them word by word.
Exploratory Comment Sentiment Analysis
To explore the valence of the comments made in a continuous, rather than in a categorical, fashion, we conducted a sentiment analysis using SENTIDA (Lauridsen et al., 2019). SENTIDA is a simple model for general sentiment analysis in Danish which has been shown to perform well as compared to other models developed for the purpose of automated sentiment analysis in Danish (e.g., AFINN; Nielsen, 2011). The output of the SENTIDA model is a sentiment score for each separate string of text provided that can range from -∞ to ∞ with values below zero indicative of increasingly negative sentiments, a value of exactly zero indicative of neutrality, and values above zero indicative of increasingly positive sentiments.
Results
Descriptive Statistics and Exploratory Data Visualization
Comment Type, Length, and Sentiment
A total of 1,025 (11.64%) participants made a comment, of which 346 (33.76%) were categorized as neutral, 153 (14.93%) as commendations, and 526 (51.32%) as complaints. The average comment length including spaces was 160.70 (SD = 167.48) characters and the average sentiment score was 1.62 (SD = 5.36). Density distributions for the comment length and sentiment score are presented in Figure 1, together with means, standard deviations, and raincloud plots for the comment length and sentiment based on whether participants commended, wrote a neutral comment, or complained. Means, standard deviations, and raincloud plots for the six HEXACO dimensions based on whether participants commented and what type of comment they wrote are presented in Figure 2 and Figure 3.
Figure 1
Density Distribution for Comment Length and Comment Sentiment
Note. n = 153 for commend; n = 346 for neutral; and n = 526 for complain.
Figure 2
Means, Standard Deviations, and Raincloud Plots for the Six HEXACO Dimensions Based on Whether Participants Commented
Note. n = 1,025 for commented = Yes; n = 7,784 for commented = No.
Figure 3
Means, Standard Deviations, and Raincloud Plots for the Six HEXACO Dimensions Based on Whether Participants Commended, Wrote a Neutral Comment, or Complained
Note. n = 153 for commend; n = 346 for neutral; and n = 526 for complain.
Pre-Registered Analyses
Who Comments?
Testing our first hypothesis regarding who is more likely to comment, we performed a logistic regression with comment yes/no as the dependent variable and the six HEXACO dimensions, age, and gender as the independent variables. The results are presented in Table 2, showing that women, older participants, and participants with higher levels of Openness to Experience were more likely to comment, whereas participants with higher levels of Honesty-Humility and Agreeableness vs. Anger commented less. Correcting for multiple testing using the Bonferroni correction, only age, OR = 1.19, 95% CI [1.11, 1.28], padjusted < .001, and Openness to Experience, OR = 1.30, 95% CI [1.21, 1.40], padjusted < .001, continued to be significant predictors of who commented. In turn, we did not find support for Hypothesis 1 that participants with higher levels of Extraversion are more likely to comment, OR = 1.06, 95% CI [0.99, 1.14], padjusted = .984.
Table 2
Logistic Regressions Predicting Who Comments and Types of Comments
| Variable | OR (AME) [OR 95% CI] |
|||
|---|---|---|---|---|
| Comment | Commend | Complain | Neutral | |
| Intercept | 0.14*** [0.12, 0.15] |
0.13*** [0.10, 0.18] |
1.24* [1.04, 1.48] |
0.46*** [0.38, 0.56] |
| Age | 1.19*** (.001) [1.11, 1.28] |
1.15 (.001) [0.94, 1.39] |
0.90 (-.002) [0.79, 1.03] |
1.04 (.001) [0.91, 1.20] |
| Gender (Male) | 0.83* (-.019) [0.72, 0.96] |
1.18 (.020) [0.79, 1.74] |
0.73*(-.075) [0.55, 0.97] |
1.29 (.056) [0.96 1.74] |
| Honesty-Humility | 0.91** (-.017) [0.85, 0.97] |
1.08 (.016) [0.89, 1.30] |
0.88 (-.054) [0.77, 1.01] |
1.11 (.039) [0.96 1.28] |
| Emotionality | 0.97 (-.004) [0.90, 1.04] |
0.94 (-.011) [0.78, 1.13] |
0.79*** (-.083) [0.69, 0.91] |
1.34*** (.093) [1.16, 1.55] |
| Extraversion | 1.06 (-.009) [0.99, 1.14] |
1.36** (.056) [1.10, 1.70] |
0.85* (-.057) [0.74, 0.98] |
1.03 (.008) [0.89, 1.18] |
| Agreeableness vs. Anger | 0.91** (-.014) [0.85, 0.98] |
1.07 (.013) [0.90, 1.29] |
0.90 (-.038) [0.79, 1.02] |
1.08 (.025) [0.94, 1.24] |
| Conscientiousness | 0.98 (-.002) [0.92, 1.05] |
1.09 (.016) [0.90, 1.31] |
0.88 (-.046) [0.77, 1.01] |
1.09 (.029) [0.95, 1.25] |
| Openness to Experience | 1.30*** (.038) [1.22, 1.40] |
1.33** (.049) [1.09, 1.62] |
0.95 (-.018) [0.83, 1.08] |
0.91 (-.030) [0.79 1.04] |
| n | 8,786 | 1,021 | 1,021 | 1,021 |
| AIC | 6226.94 | 845.80 | 1394.56 | 1297.42 |
| BIC | 6290.67 | 890.16 | 1438.92 | 1341.78 |
| Pseudo R2 | 0.02 | 0.06 | 0.05 | 0.03 |
Note. Continuous variables are mean-centered and scaled by 1 standard deviation. All p-values are two-tailed and uncorrected for multiple testing. Bolded values indicate that p < .05 after applying the Bonferroni correction. Participants with Gender = “Other” (n = 23) are not included in the regression analyses. OR = odds ratio, CI = confidence interval, AME = average marginal effect.
*p < .05. **p < .010. ***p < .001.
Who Commends, Complains, and Writes Neutral Comments?
Next, we investigated whether participants with different levels in the HEXACO dimensions tend to commend, write neutral comments, or complain once they have decided to comment in the first place. To this end, we conducted three separate logistic regression analyses with the three comment types as the dependent variables. In support of Hypothesis 2, we found that participants with higher levels of Extraversion were more likely to commend (see Table 2). This result even holds after applying the Bonferroni correction, OR = 1.36, 95% CI [1.10, 1.70], padjusted = .046. On the contrary, we found no support for Hypothesis 3 that participants with higher levels of Agreeableness vs. Anger commend more, OR = 1.07, 95% CI [0.90, 1.29], padjusted = 1.00. Further, we found that people with higher levels of Openness to Experience were more likely to commend, OR = 1.33, 95% CI [1.09, 1.62], padjusted = .044.
Concerning complaints, we did not find support for Hypothesis 4 that participants with lower levels of Agreeableness vs. Anger were more likely to complain, OR = 0.90, 95% CI [0.79, 1.02], padjusted = .972. Conversely, even after applying the Bonferroni correction, we found that participants with higher levels of Emotionality were less likely to complain, OR = 0.79, 95% CI [0.69, 0.91], padjusted = .007, and more likely to make a neutral comment, OR = 1.34, 95% CI [1.16, 1.55], padjusted = .001. This result should be interpreted with caution, however, given the low internal consistency estimate of the Emotionality factor. Further, men, OR = 0.73, 95% CI [0.55, 0.97], p = .030, and participants with higher levels of Extraversion, OR = 0.85, 95% CI [0.74, 0.98], p = .022, appeared to complain less. These results do not hold after applying the Bonferroni correction (all padjusted > .05), though.
Non-Pre-Registered Exploratory Analyses
Comment Length and Sentiment
Comment Length
Next, we explored the relation between the HEXACO dimensions and comment length, using ordinary least square regression analysis. As shown in Table 3, we found that women, younger participants, participants low in Emotionality, and participants high in Openness to Experience wrote longer comments. Moreover, we found commendations to be shorter and complaints to be longer than neutral comments. However, only the results concerning age, β = -.201, 95% CI [-.262, -.141], padjusted < .001, Openness to Experience, β = .110, 95% CI [.049, .171], padjusted = .004, and comment type, βcommend = -.310, 95% CI [-.493, -.126], padjusted = .011; βcomplain = .241, 95% CI [0.109, 0.372], padjusted = .004, remained significant after applying the Bonferroni correction.
Table 3
Ordinary Least Square Regressions Predicting Comment Length and Sentiment Using Scales
| Variable | β [95% CI]
|
|
|---|---|---|
| Comment length | Sentiment | |
| Intercept | .000 [-.119, .120] | -.139* [-.257, -.021] |
| Age | -.201*** [-.262, -. 141] | -.059 [-.119, .001] |
| Gender (Male) | -.176** [-.308, -.044] | .034 [-.097, .165] |
| Honesty-Humility | .015 [-.051, .080] | -.018 [-.082, .047] |
| Emotionality | -.055 [-.118, .008] | .033 [-.030, .096] |
| Extraversion | .027 [-.037, .091] | .032 [-.031, .096] |
| Agreeableness vs. Anger | -.027 [-.091, .036] | .065* [.002, .128] |
| Conscientiousness | .035 [-.026, .096] | .011 [-.050, .072] |
| Openness to Experience | .110*** [.049, .171] | .046 [-.014, .106] |
| Commend | -.310*** [-.493, -.126] | .875*** [.693, 1.057] |
| Complain | .241*** [.109, .372] | -.014 [-.144, .117] |
| n | 1,021 | 1,021 |
| AIC | 2806.69 | 2793.51 |
| BIC | 2865.83 | 2852.65 |
| R2 | 0.11 | 0.12 |
Note. Continuous variables are mean-centered and scaled by 1 standard deviation. All p-values are two-tailed and uncorrected for multiple testing. Bolded values indicate that p < .05 after applying the Bonferroni correction. Participants with Gender = “Other” (n = 23) are not included in the regression analyses.
*p < .05. **p < .01. ***p < .001.
Sentiment
To investigate who writes more positive or negative comments in a continuous fashion, we conducted another ordinary least square regression analysis with the sentiment score from the SENTIDA model as the dependent variable. As shown in Table 3, we found that participants with higher levels of Agreeableness vs. Anger wrote more positive comments. Moreover, we found the comment type to be a significant predictor of the SENTIDA sentiment score. After applying the Bonferroni correction, only the result concerning comment type, βcommend = .875, 95% CI [.693, 1.057], padjusted < .001, remained significant.
Item-Level (Facet) Analyses
Considering the low internal consistency estimates of the BHI factors as well as recent calls to focus more on personality facets and single items (e.g., Mõttus et al., 2020), we reran all analyses reported above using each item of the BHI as individual predictors in the regression models. As shown in Table 4, we found the following items to be significant predictors of who commented after applying the Bonferroni correction: “I want to be famous” (Honesty-Humility, Greed Avoidance; OR = 0.90, 95% CI [0.84, 0.96], padjusted = .050), “I easily approach strangers” (Extraversion, Social boldness; OR = 1.16, 95% CI [1.06, 1.26], padjusted = .020), “I often express criticism” (Agreeableness vs. Anger, Gentleness; OR = 0.89, 95% CI [0.83, 0.96], padjusted = .042), and “I think science is boring” (Openness to Experience, Inquisitiveness; OR = 1.31, 95% CI [1.20, 1.42], padjusted < .001). Interestingly, after applying the Bonferroni correction, none of the items were found to be a significant predictor of comment type, comment length, nor sentiment (see Table 4 and Table 5).
Table 4
Logistic Regressions Predicting Who Comments and Types of Comments Using BHI Items
| Variable | Facet | Item text | OR (AME) [OR 95% CI] |
|||
|---|---|---|---|---|---|---|
| Comment | Commend | Complain | Neutral | |||
| Intercept | — | — | 0.13*** [0.12, 0.15] |
0.13*** [0.09, 0.17] | 1.23* [1.02, 1.48] |
0.45*** [0.37, 0.54] |
| Age | — | — | 1.18***(.001) [1.10, 1.27] |
1.07 (.001) [0.87, 1.32] |
0.94 (-.001) [0.81, 1.09] |
1.03 (.000) [0.88, 1.20] |
| Gender (Male) | — | — | 0.78**(-.025) [0.67, 0.91] |
1.12 (.013) [0.73, 1.71] |
0.73* (-.075) [0.54, 0.98] |
1.35 (.064) [0.98, 1.85] |
| Honesty-Humility 1 | Sincerity | I find it difficult to lie. | 1.03 (.003) [0.96, 1.11] |
1.28* (.024) [1.05, 1.57] |
0.84* (-.034) [0.74, 0.97] |
1.06 (.011) [0.92, 1.22] |
| Honesty-Humility 2 (R) | Fairness | I would like to know how to make lots of money in a dishonest manner. | 0.97 (-.003) [0.91, 1.04] |
1.10 (.011) [0.91, 1.34] |
0.92 (-.019) [0.81, 1.05] |
1.04 (.008) [0.91, 1.19] |
| Honesty-Humility 3 (R) | Greed Avoidance | I want to be famous. | 0.90** (-.012) [0.84, 0.96] |
0.78** (-.034) [0.65, 0.92] |
1.02 (.005) [0.89, 1.16] |
1.13 (.030) [0.98, 1.30] |
| Honesty-Humility 4 (R) | Modesty | I am entitled to special treatment. | 0.93* (-.010) [0.87, 0.99] |
1.12 (.017) [0.93, 1.35] |
0.97 (-.008) [0.86, 1.10] |
0.97 (-.008) [0.85, 1.11] |
| Emotionality 1 | Fearfulness | I am afraid of feeling pain. | 0.96 (-.004) [0.89, 1.03] |
1.09 (.008) [0.90, 1.32] |
0.93 (-.014) [0.81, 1.07] |
1.03 (.006) [0.90, 1.19] |
| Emotionality 2 (R) | Anxiety | I worry less than others. | 0.92* (-.007) [0.86, 0.99] |
0.89 (-.012) [0.74, 1.07] |
0.98 (-.004) [0.85, 1.12] |
1.09 (.016) [0.95, 1.26] |
| Emotionality 3 (R) | Dependence | I can easily overcome difficulties on my own. | 1.07 (.007) [0.99, 1.15] |
0.76* (-.035) [0.61, 0.94] |
0.91 (-.023) [0.79, 1.05] |
1.26** (.053) [1.08, 1.46] |
| Emotionality 4 | Sentimentality | I have to cry during sad or romantic movies. | 1.04 (.003) [0.97, 1.12] |
0.95 (-.005) [0.78, 1.16] |
0.86* (-.027) [0.75, 0.98] |
1.22** (.031) [1.05, 1.41] |
| Extraversion 1 (R) | Social Self-esteem | Nobody likes talking with me. | 1.01 (.001) [0.94, 1.08] |
1.22 (.029) [0.94, 1.59] |
0.90 (-.031) [0.77, 1.05] |
1.04 (.011) [0.89, 1.22] |
| Extraversion 2 | Social Boldness | I easily approach strangers. | 1.16*** (.012) [1.06, 1.26] |
1.16 (.014) [0.90, 1.50] |
0.90 (-.020) [0.76, 1.07] |
1.03 (.006) [0.87, 1.23] |
| Extraversion 3 | Sociability | I like to talk with others. | 1.01 (.001) [0.93, 1.09] |
1.12 (.015) [0.89, 1.41] |
0.88 (-.034) [0.76, 1.02] |
1.09 (.021) [0.93, 1.28] |
| Extraversion 4 (R) | Liveliness | I am seldom cheerful. | 0.95 (-.005) [0.88, 1.02] |
0.97 (-.004) [0.79, 1.19] |
1.14 (.030) [0.99, 1.31] |
0.88 (-.026) [0.76, 1.02] |
| Agreeableness vs. Anger 1 (R) | Forgiveness | I remain unfriendly to someone who was mean to me. | 1.06 (.005) [0.99, 1.14] |
0.90 (-.010) [0.74, 1.10] |
1.05 (.010) [0.92, 1.21] |
1.00 (.000) [0.87, 1.16] |
| Agreeableness vs. Anger 2 (R) | Gentleness | I often express criticism. | 0.89** (-.011) [0.83, 0.96] |
1.21 (.021) [1.00, 1.48] |
0.89 (-.026) [0.77, 1.02] |
1.03 (.005) [0.89, 1.19] |
| Agreeableness vs. Anger 3 | Flexibility | I tend to quickly agree with others. | 0.89** (-.013) [0.83, 0.96] |
0.98 (-.003) [0.80, 1.20] |
0.98 (-.005) [0.85, 1.13] |
1.03 (.008) [0.89, 1.20] |
| Agreeableness vs. Anger 4 | Patience | Even when I'm treated badly, I remain calm. | 0.99 (-.001) [0.93, 1.07] |
1.02 (.002) [0.84, 1.23] |
0.92 (-.016) [0.80, 1.06] |
1.08 (.014) [0.93, 1.24] |
| Conscientiousness 1 | Organization | I make sure that things are in the right spot. | 0.90** (-.010) [0.84, 0.96] |
1.19 (.018) [0.98, 1.44] |
0.88 (-.028) [0.77, 1.00] |
1.06 (.011) [0.92, 1.22] |
| Conscientiousness 2 (R) | Diligence | I postpone complicated tasks as long as possible. | 0.98 (-.002) [0.91, 1.05] |
0.99 (-.001) [0.81, 1.21] |
0.87 (-.027) [0.76, 1.01] |
1.16 (.028) [1.00, 1.35] |
| Conscientiousness 3 | Perfectionism | I work very precisely. | 1.06 (.007) [0.98, 1.14] |
0.85 (-.022) [0.70, 1.03] |
1.05 (.012) [0.91, 1.20] |
1.03 (.007) [0.89, 1.19] |
| Conscientiousness 4 (R) | Prudence | I often do things without really thinking. | 1.07 (.007) [1.00, 1.15] |
1.09 (.010) [0.90, 1.33] |
0.99 (-.002) [0.86, 1.14] |
0.96 (-.007) [0.83, 1.12] |
| Openness to Experience 1 | Aesthetic Appreciation | I can look at a painting for a long time | 1.11** (.008) [1.03, 1.19] |
1.33** (.025) [1.09, 1.62] |
0.95 (-.010) [0.82, 1.09] |
0.91 (-.015) [0.79, 1.05] |
| Openness to Experience 2 (R) | Inquisitiveness | I think science is boring. | 1.31*** (.026) [1.20, 1.42] |
0.86 (-.017) [0.70, 1.05] |
1.14 (.029) [0.98, 1.32] |
0.95 (-.012) [0.81, 1.10] |
| Openness to Experience 3 | Creativity | I have a lot of imagination. | 1.05 (.004) [0.97, 1.13] |
1.20 (.020) [0.96, 1.48] |
0.88 (-.028) [0.76, 1.02] |
1.04 (.008) [0.89, 1.21] |
| Openness to Experience 4 | Unconventionality | I like people with strange ideas. | 0.98 (-.002) [0.91, 1.05] |
1.10 (.011) [0.90, 1.34] |
0.98 (-.004) [0.85, 1.13] |
0.97 (-.007) [0.83, 1.12] |
| n | 8,786 | 1,021 | 1,021 | 1,021 | ||
| AIC | 6172.81 | 844.48 | 1405.99 | 1317.16 | ||
| BIC | 6363.99 | 977.55 | 1539.06 | 1450.23 | ||
| Pseudo R2 | 0.04 | 0.12 | 0.08 | 0.06 | ||
Note. Continuous variables are mean-centered and scaled by 1 standard deviation. All p-values are two-tailed and uncorrected for multiple testing. Bolded values indicate that p < .05 after applying the Bonferroni correction. Participants with Gender = “Other” (n = 23) are not included in the regression analyses. Items with an (R) have been reversed scored.
*p < .05. **p < .01. ***p < .001.
Table 5
Ordinary Least Square Regressions Predicting Comment Length and Sentiment Using Items
| Variable | Facet | Item text | β [95% CI]
|
|
|---|---|---|---|---|
| Comment length | Sentiment | |||
| Intercept | — | — | .014 [-.107, .135] | -.125* [-.246, -.004] |
| Age | — | — | -.192*** [-.256, -.127] | -.050 [-.115, .015] |
| Gender (Male) | — | — | -.191** [-.329, -.052] | .014 [-.124, .153] |
| Honesty-Humility 1 | Sincerity | I find it difficult to lie. | .028 [-.034, .091] | .012 [-.050, .075] |
| Honesty-Humility 2 (R) | Fairness | I would like to know how to make lots of money in a dishonest manner. | -.009 [-.070, .052] | -.014 [-.076, .047] |
| Honesty-Humility 3 (R) | Greed Avoidance | I want to be famous. | -.019 [-.083, .045] | -.066* [-.130, -.002] |
| Honesty-Humility 4 (R) | Modesty | I am entitled to special treatment. | .043 [-.020, .107] | .035 [-.028, .099] |
| Emotionality 1 | Fearfulness | I am afraid of feeling pain. | -.032 [-.095, .031] | .024 [-.039, .087] |
| Emotionality 2 (R) | Anxiety | I worry less than others. | -.042 [-.104, .021] | -.032 [-.095, .031] |
| Emotionality 3 (R) | Dependence | I can easily overcome difficulties on my own. | -.037 [-.103, .029] | .019 [-.047, .085] |
| Emotionality 4 | Sentimentality | I have to cry during sad or romantic movies. | .035 [-.028, .098] | .041 [-.022, .104] |
| Extraversion 1 (R) | Social Self-esteem | Nobody likes talking with me. | -.031 [-.098, .036] | .003 [-.064, .070] |
| Extraversion 2 | Social Boldness | I easily approach strangers. | .048 [-.026, .122] | -.049 [-.123, .025] |
| Extraversion 3 | Sociability | I like to talk with others. | -.002 [-.071, .067] | .032 [-.038, .101] |
| Extraversion 4 (R) | Liveliness | I am seldom cheerful. | .025 [-.041, .090] | .065 [-.000, .131] |
| Agreeableness vs. Anger 1 (R) | Forgiveness | I remain unfriendly to someone who was mean to me. | .006 [-.058, .069] | .018 [-.046, .081] |
| Agreeableness vs. Anger 2 (R) | Gentleness | I often express criticism. | -.051 [-.116, .014] | .030 [-.035, .095] |
| Agreeableness vs. Anger 3 | Flexibility | I tend to quickly agree with others. | -.080* [-.146, -.013] | .016 [-.051, .083] |
| Agreeableness vs. Anger 4 | Patience | Even when I'm treated badly, I remain calm. | .055 [-.009, .120] | .031 [-.034, .096] |
| Conscientiousness 1 | Organization | I make sure that things are in the right spot. | -.018 [-.081, .045] | .035 [-.027, .098] |
| Conscientiousness 2 (R) | Diligence | I postpone complicated tasks as long as possible. | -.049 [-.113, .016] | .005 [-.060, .069] |
| Conscientiousness 3 | Perfectionism | I work very precisely. | .085** [.021, .149] | -.005 [-.069, .059] |
| Conscientiousness 4 (R) | Prudence | I often do things without really thinking. | .026 [-.037, .089] | -.029 [-.092, .034] |
| Openness to Experience 1 | Aesthetic Appreciation | I can look at a painting for a long time | .062 [-.002, .127] | .033 [-.031, .097] |
| Openness to Experience 2 (R) | Inquisitiveness | I think science is boring. | .064* [.003, .126] | .007 [-.055, .069] |
| Openness to Experience 3 | Creativity | I have a lot of imagination. | .032 [-.033, .097] | .035 [-.030, .100] |
| Openness to Experience 4 | Unconventionality | I like people with strange ideas. | -.036 [-.101, .028] | .003 [-.061, .068] |
| Commend | — | — | -.315*** [-.499, -.130] | .848*** [.663, 1.034] |
| Complain | — | — | .228*** [.097, .360] | -.016 [-.148, .116] |
| n | 1,021 | 1,021 | ||
| AIC | 2808.51 | 2812.69 | ||
| BIC | 2956.37 | 2960.55 | ||
| R2 | .11 | .13 | ||
Note. Continuous variables are mean-centered and scaled by 1 standard deviation. All p-values are two-tailed and uncorrected for multiple testing. Bolded values indicate that p < .05 after applying the Bonferroni correction. Participants with Gender = “Other” (n = 23) are not included in the regression analyses. Items with an (R) have been reversed scored.
*p < .05. **p < .01. ***p < .001.
Discussion
Relying on a large and age- and gender-wise heterogeneous sample of 8,809 adult Danish citizens, the present study extends the current knowledge on the relations between personality and online commenting in several ways. First, it focuses on commenting in online scientific studies, whereas most previous studies focused on commenting on social media and/or company websites (e.g., Liu & Campbell, 2017; Picazo-Vela et al., 2010). Second, it considers the HEXACO dimensions, whereas previous studies considered the Big Five dimensions (e.g., Choi et al., 2017; Wu & Atkin, 2017) or other personality characteristics (e.g., Sorokowski et al., 2020). Third, it investigated actual commenting behavior, whereas most previous research investigated self-reported commenting behavior.
In contrast to previous research looking at social media and/or company websites, which found robust links between self-reported online commenting and Big Five Extraversion (Choi et al., 2017; Liu & Campbell, 2017) and Agreeableness (Choi et al., 2017; Lee et al., 2014; Wu & Atkin, 2017), our results indicate that only HEXACO Openness to Experience is linked to who actually commented in the specific context of this study; that is, an online scientific study without financial or individualized incentives (e.g., automatized personality feedback) on a timely topic (i.e., COVID-19). This finding highlights the importance of using behavioral observations as opposed to self-report measures (e.g., King, 2010) when investigating the relation between personality and online commenting. The finding also suggests that the influence of different personality dimensions on online commenting might be highly contextual. For instance, it might be that contextual factors such as whether there are incentives to comment (e.g., social reputation) affect which personality dimensions play a role in commenting or not. With regard to the current study, one possible explanation for why Openness to Experience was found to be the only predictive dimension with regard to who comments is that the opportunity to comment presented itself in the context of a scientific study on a timely socio-political topic (COVID-19). More precisely, people high in Openness to Experience tend to be intellectual and interested in science (Ashton & Lee, 2007), as well as concerned about political issues (e.g., Blanchet, 2019; Jordan et al., 2015). Thus, it might be that people with higher levels in Openness to Experience commented more on our survey because they found participating in a study on a timely socio-political topic interesting or relevant. This interpretation is in line with the observation that respondents with higher levels of Openness to Experience wrote longer comments and were more likely to commend once they had decided to comment in the first place. This interpretation is further in line with the finding that only the Openness to Experience item “I think science is boring”, but none of the other Openness to Experience items, which did not refer to science, significantly predicted who commented when controlling for multiple testing.
The fact that people with higher levels of Extraversion were more likely to write commendations aligns well with the conceptualization of this dimension. In fact, Zettler et al. (2020) found that especially Extraversion was related to criteria in the realm of positivity (e.g., happiness, positive affect, satisfaction). Notably, the exploratory item analyses suggest that it is the overall Extraversion factor which drives this link, rather than any single administered Extraversion item. Relatedly, the Extraversion item “I easily approach strangers” was linked to who commented or not even when controlling for multiple testing. This appears plausible given that writing online comments to unknown researchers clearly is a behavioral manifestation of approaching strangers.
On the trait dimension-level, we found that individuals high in Emotionality wrote more neutral comments and refrained from writing complaints. These results can be considered in line with existing research showing that Emotionality is negatively related to risk-taking (Zettler et al., 2020) and social risk taking in particular (Weller & Tikir, 2011). Specifically, people high in Emotionality might have aimed to avoid upsetting the recipients of the comments, given their generally high levels of anxiety and avoidance tendencies (Ashton & Lee, 2007; Zettler et al., 2020). Again, this tendency appears to be driven by the overall factor rather than any of the administered Emotionality items.
With regard to the exploratory item analyses, two further items turned out to be linked to who commented when controlling for multiple testing. The Honesty-Humility (Greed Avoidance) item “I want to be famous” and the Agreeableness vs. Anger (Gentleness) item “I often express criticism”. Overall, these results might be seen as support for the idea that personality items are important to consider, especially in contexts (and/or with regard to criteria) which are obvious expressions of an item in question.
Although we did observe several significant relations between people’s trait dimension/item levels and their commenting behavior, none of these relations were particularly strong. In fact, per the guidelines put forward by Chen et al. (2010), all of the estimated odds ratios in Tables 2 and 4 can be considered very small. Looking at the estimated average marginal effects (AME), a similar pattern emerges. As an example, the AME for the link between Openness to Experience and commenting was estimated to be .038, meaning that an increase of one standard deviation in Openness to Experience is associated with a 3.8% increase in the predicted probability of commenting. While our results provide evidence for several links between people’s personality and their commenting behavior in online scientific studies, they thus also show that these links are not overwhelmingly strong.
From a practical perspective, our findings suggest that researchers need to be mindful of how they evaluate and deal with comments in online scientific studies, given that people with certain personality characteristics tend to comment (in specific ways) more than others. Yet, in light of the modest strength of the observed relations, our results also suggest that researchers need not to be overly concerned that the comments they receive in online scientific studies are severely biased.
Limitations
There are several limitations of this investigation. First, the context of the study was highly specific which might limit the generalizability of the results. Yet, so are most other contexts in which online commenting has been studied, such as social network sites which may provide social incentives to comment on specific topics in a particular way (e.g., to gain reputation). As described above, we interpret the differences and similarities of our results with the results of other studies as support for the idea that the links between personality and online commenting are rather context-specific.
Second, although the BHI explicitly aims to assess the HEXACO dimensions both broadly and briefly and thus is expected to have relatively low internal consistency estimates (de Vries, 2013), some of the estimates found herein were particularly low. More precisely, based on the Spearman–Brown prophecy formula, one would expect an alpha of .43 for the BHI factors (four items per factor, with each item referring to a different facet; de Vries, 2013), and two of the found estimates—concerning Honesty-Humility and Emotionality—fell below this threshold. Importantly, we do not consider this as a problem of the translation process (for how to assess a potential translation bias, see, e.g., Bader et al., 2021), given that other researchers using the BHI found similarly low internal consistency estimates, including in Dutch samples, and, thus, the language in which the BHI was developed (e.g., Garbe et al., 2020; van Sintemaartensdijk et al., 2022). Nonetheless, we urge future research to investigate the link between the HEXACO dimensions and actual commenting behavior using other and more reliable inventories. At the same time, we want to emphasize that anyone who is particularly worried about the low internal consistency estimates of some of the BHI factors can disregard any of the trait-level findings and focus exclusively on the item-level analyses.
As a third limitation, we could only compare participants who commented with those who did not within our sample. Hence, it is possible that other people with certain personality characteristics who did not participate in this study would have commented differently. Unfortunately, it is difficult tackle this issue, given that people virtually always self-select into studies. Aiming for fully representative samples would, however, be one way to address this issue.
Finally, given that only very few participants actually wrote a comment (11.64%), it might be that most participants thought that no one would read their comments, which, in turn, might have kept them from commenting in the first place as well as influenced the type of comments made by those who commented after all.
Conclusion
Relying on a large repeated cross-sectional COVID-19 survey, we provide novel insights into the relations between personality and commenting behavior in online scientific studies. Our results show that, as compared to their counterparts, people high in Openness to Experience comment more; that people high in Extraversion and Openness to Experience commend more; and that those high in Emotionality write both more neutral comments and fewer complaints. At face value, these findings suggest that researchers need to be mindful of how they evaluate and deal with comments in online scientific studies, given that people with certain personality characteristics comment (in specific ways) more than others. At the same time, our results also suggest that researchers need not to be overly concerned that the comments they receive in online scientific studies are severely biased, given that none of the observed links between people’s personality and their commenting behavior was particularly strong.