: Re: Metrics for assessing the persuasiveness of a paper? Are there any known variables to measure, or questions to ask in survey, in order to asses the persuasiveness of a paper? For example,

@Deb2945533

Vote Up Vote Down

"Persuasiveness" is highly subjective. I can't imagine how you would measure it other than to perform an experiment with real people.

You didn't say what the subject of your paper is, which is probably good because that helps us to discuss the question of persuasiveness without being biased by whether or not we agree with you.

I know of three basic approaches to measuring persuasiveness:

(1) Get a group of test subjects. Have them read the paper. Ask them to rate how persuasive it is on some scale, 1 to 10 or whatever.

Catch: People who already agree with you will likely say that it is very persuasive, while people who disagree with you will tend say it is not persuasive at all, no matter how compelling the actual arguments are. You may well be measuring how many people agreed with you before reading the paper then anything about the paper itself. Often people disagree not over the facts themselves but over which facts are important.

For example -- and I'm going to use a real example here because it's hard to make this point purely in the abstract, but I'm trying to concentrate on the meta-issue of persuasiveness and not any particular issue -- so for example, suppose you were arguing against legalizing marijuana, and so you gave evidence that marijuana is bad for people's health. Someone who was pro-legalization might well say that even if true, this is irrelevant and therefore unpersuasive, because people should have the freedom to decide for themselves whether the perceived benefits outweigh the harm. On the other hand, suppose you were arguing for legalizing marijuana, and you pointed out how many people were jailed or lost their jobs or whatever just because they were caught smoking. Someone who was anti-legalization would likely reply that this is unpersuasive because the whole idea of making something illegal is to impose sanctions on people who break the law so that people will be afraid to do it: if these laws didn't hurt people, then they would be ineffective. Etc.

(2) Get a group of test subjects. Have them read the paper. Then discuss with them what they thought the strong and weak points were. This is called a "focus group". The advantage over a simple rating is that it gives you information about hows and whys. The disadvantage is that it gives no cut-and-dried answers. Advertisers often do this with planned advertising campaigns to see how people react to them. Just hearing that on a scale of 1 to 10 this advertisement got an average score of 4 tells you little. Is that good or bad? Compared to what? Does that mean people would buy the product or not? Is there something we could do to make it more effective or is that the best that we can hope for under the circumstances? Etc.

(3) Get a group of test subjects. Give them a pre-test asking their opinion on the subject matter. Then have them read the paper. Then give a post-test. See how many changed their minds.

I think this is the most effective method because it directly measures what you are presumably trying to accomplish: changing people's minds.

Catch: The more controversial the subject, the fewer people will change their minds. With a small test group, the numbers may be so small that they are not statistically significant. People's opinions about controversial subjects are built up over the course of a lifetime, from many many individual facts, ideas, experiences, etc. If a person believes that, say, political position X is good for the country, he likely has hundreds of reasons for thinking so. If you present him with a reason to think it's bad, his first reaction is likely to be that your argument MUST be flawed, because he KNOWS that X is good. Even if you convince him that your argument is valid, it's still one argument against versus a hundred arguments for. He's likely to see it as a curious anomaly, not a game-changer.

So no matter how valid and persuasive your arguments are, any effects may be very subtle, and thus difficult to measure. There may be some number of people who were undecided and who you managed to bring into your camp. But likely there are many more who were previously 80% sure that X was right but who are now only 79% sure. Etc.

(0)

Login to post a comment!