This post is a continuation of my last post about the logical fallacy within negative categories.  In the introductions to that post and the preceding post, I described the need to challenge the content of data that goes into automated or obligated decision-making from analytics.  My suggestion is to use classical rhetoric (including grammar and logic) as a guide for how the population can demand valid data by identifying and removing fallacies in data.   In the last post I proposed that something similar to syllogistic fallacies can occur in negative categories: categories representing the population not otherwise positively categorized.

One of my examples of a negative category was non-participating labor because it is the population of working eligible adults who are not working.   Frequently economic policy and debate focuses on this category as the one that needs attention.   The problem is that this is not a positive category.  We derive this number as the population remaining after we exclude the members who have jobs of some type.   The remainder has much more diversity of life situations and job expectations than the positive categories of people who have jobs.   The positive information about a person with a job is that he has found an acceptable job for the moment.    In contrast, there is no comparable positive information about what an acceptable employment situation would be like for those who do not have jobs.    The focus on this negative category risks a syllogistic fallacy of concluding that this population needs in terms of employment.

The last post ended with this statement:

I need more time to think about my own experiences with negative labels and how they interfere with decision-making.

I realized that a major challenge of my data projects have been in appropriately interpreting negative categories, and in particular to set expectations of analysts of what to expect from such categories.   Although I was working with a completely different kind of data, I was confronting the same kind of negative categories that economists have with unemployment.

In order to keep this a blog discussion, I am going to try to translate my experiences into an analogy.   The analogy I have in mind concerns comment sections of blog sites that are more popular than the one you are reading now.   Popular blog sites attract large communities who contribute many comments to each post.   Some blogs attract some very fascinating and informative comments.   Consider a project that attempts to analyze the comments of the posts in terms of the sentiment of the comment where the sentiment may be one of three options: agreement, disagreement, or request for clarification.   For each post, there may be different category labels for different sections or key points within the post.   Other dimensions (columns) of information for a comment may be the unique identifier (handle) of the commenter, the time, and an optional hyperlink to the commenter’s site.    This imagined project attempts to extract analytic information about the entire collection of comments for a busy site.

I’m assuming you have similar experience as I have in reading comments of other blogs or news site articles.   I have nostalgic (that I admit not reliable) memories of a period when there was wide cooperation in keeping comments specific to the points in the article.   I imagine the original intent of the comment section was similar to offering the article for editorial or peer review where the editors or peers were the readers themselves.   When I read comments today, I seek out the comments that follow that model of being reviewers of what the writer has written.   When I participate, I prefer to keep my comment contributions similarly constrained.   If some other blog or news article inspired some original thought, then that is more appropriate a post on my own blog (here) than as a reviewer comment on that site.   As I said, I have an old-fashioned view of what should be found in comment sections: comments should stay on topic of the parent article or post.

Modern commenting practice is very different from this old-fashioned view.  Despite my disappointment over the departure from the ideal I’d prefer, I find reading comments to be fascinating (though sometimes aggravating) to read.   There is so much going on in the comments that sometimes the off-topic or trolling exchanges will eventually come back on topic of the post and present something meaningful where perhaps that meaningful comment would have been volunteered otherwise.

For this thought experiment, I imagine a project interested in analyzing the on-topic comments to understand how well the stories are written.  Lessons may include what value readers find in the article, what trends appear in the criticisms (such as whether the articles need more research or more proof-reading), or the level of familiarity of the topic by the readers (such as whether they need more background information to understand the points in the post).   For relevant comments, I would assign a positive category to identify their handle, what post or section they are commenting on, when they made the comment relative to the post’s publication date, and what their sentiment was about it.

On busy sites, the majority of comments may be irrelevant.   It is still useful to quantify the number of irrelevant comments or perhaps even their relative times.   However, my proposed “sentiment” categories do not apply to irrelevant comments and I have no motivation find categories to assign the comments to positive categories.   I may lump them all into a sentiment label of “irrelevant”.   This is a negative label.  It merely means it is not one of the positive labels (agree, disagree, question).   My intention of having the negative label is to have a reference of scale such as to compare the number of positive labels with the overall number of comments.   I would discourage myself from attempting to interpret the negative label in any way.   The “irrelevant” label may have a wide range of meaningful positive categories to distinguish them from each other and those positive labels may defined mutually exclusive and perhaps antagonistic groups.    My purpose of placing them in a single group is merely set them aside from any direct consideration.  The label will appear as column option but I would ignore it as a principle topic of analysis.

In some ways, the previous post’s discussions of unemployed is like the “irrelevant” label.   The surveys seek out how many people have jobs through various techniques and then compares the number of jobs with the overall population.  The remainder is the unemployed (or not participating) and this is the irrelevant category in terms of a positive identification of a job.  The hazard comes when we attempt to treat the unemployed label as equivalent to a positive label.   There are many mutually exclusive positive labels for circumstances of unemployment.   We just are not bothering to identify those labels.

As I stated at the beginning, I’m using the comment analysis as a metaphor for the project I worked on.  In that project I ended up with many ways things can be irrelevant, and I came up with different names to distinguish them.   Each of these various names had a similar meaning of a category for those entities that I set aside from attempting to assign positive labels.   I had different reasons for setting them aside.   Although the project was not involving text data like comments on a blog, the following describe the different negative categories with similar scenarios with comments.

Spam: nuisance comments

Initially, my project did not have a problem with something comparable to spam, but eventually there came times when the project was overwhelmed by attempting to assign positive categories to a group that was generating a lot of records.  The positive categories we might have assigned were not that important and their presence did not impact the project.  To keep the project running smoothly, I create a category to assign these nuisance records so that they would be excluded from further study.

Comment spam is very similar to this negative category.   This blog site uses a service that automatically detects and isolates spam comments so they never appear in the publicly available comments.   For my humble blog in nearly a year of existence, about 95% of all comments end up in the spam folder.  Although the spam filter is automatic, I do check the folder in order to release any falsely labeled spam.   So far I’ve found only one comment that shouldn’t have been identified as a spam, so I think the service is doing a great job.

The service proudly announces to me that so far it has caught a certain number of spam comments.  As I mentioned, for this site, the number is many times greater than the justified comments.   Since spam comments eventually get deleted, all I’m left with is just a single number, the number of comments having the label of “spam”.  This is analogous to my nuisance category where all I’d end up with the aggregate sum without any distinction within that group.

When I look at the temporary folder for recent spam, I do see possibilities for meaningful positive labels to distinguish different types of spam.  Some spam, for instance has machine-generated text messages that almost seem intelligible in the body of the comment.   In this group, some of the comments include hyperlinks and others use the user-identifier field to supply a web site.   Other spam comments consist almost exclusively of hyperlinks.   I can list perhaps a dozen other recurring themes that can distinguish different positive categories for spam messages.

If I were interested in studying spam, I would not be satisfied with the single label “spam” but instead seek to distinguish the different strategies for spammers.   For this imagined project, the spam messages are uninteresting and I’m content to collect them all into a single spam category.   The number of comments is still recorded and has the label of spam, but I do not intend to make any sense of this information.   Spam will never be a topic of investigation.  All I need to know is that it exists and what its volume is.

This is analogous to one of the negative categories that I spend a lot of time dealing with on my project.  We needed to account for their presence but we had a disclaimer that if someone wanted to study this category then we would have to change the design to provide positive labels to replace the catch-all negative label.

Undifferentiated: aggregate one-time commenters

On this blog site, there are just a couple comments over the entire year so it is not a good example.   However, many more popular blog sites have are a large community of commenters.   As I read other site’s comments, I begin to pay attention to commenters who return and who begin to offer a consistent quality of response.   As I do this, I’m filtering for the top  commenters out of familiarity with a reputation I’ve learned from their past comments.   These repeat relevant commenters may present a minority of total comments but they are the ones I find most interesting.

Meanwhile, my imagined project is interested in all relevant comments so I will observe the sentiment for each relevant comment.   Many relevant comments will come from one-time commenters.    Mentally, I not pay attention to the name of the commenter because they never appeared before and they may never return.

Distinguishing comments by name can end up fragmenting the contributions of rare commenters.   I admit that this analogy is weak because in most sites the number of commenters is not that large.   However, in my real-life project, there was a large population of potential talkers and the goal of the project was to focus on the top talkers.    This lead to threshold criteria where if the talker contributes less than that threshold we will stop distinguishing him by name.   All of the low-volume talkers get aggregated into a group that together can have a measurable volume relative to the top talkers.   We called this the undifferentiated group.

The undifferentiated group is like the all-other category in pie charts where the individual slices are too tiny to present in the diagram and so numerous it would be too dense to distinguish in a printable chart.   One approach is to collect all of these into a category representing all minor slices so that it makes one large slice that we can compare with the top slices.

In this type of negative category, we replace many positive category with a single aggregate category that has the negative meaning of not being part of the big slices.  My imagined project would aggregate the infrequent commenters while continuing to distinguish the top commenters.

As the pie-chart example illustrates, there is a risk of treating this new slice (representing all other commenters) as a single voice that is a peer with the top commenters.   We need to recognize that this new slice is a negative category meant only to give perspective of the relative size of the top commenters.  This negative category may be so large that the it dominates over the top commenters.

In my project, we often found that the top group were just a small fraction of the total and this encouraged us to look elsewhere for guidance.   In these cases the largest slice of the pie was the undifferentiated group but it was meaningless to pursue any policy based on the inherent (and known) diversity within this negative group consisting of a large number of distinct individuals.   The negative category’s size convinced us to pursue a different approach to analysis.   Distinguishing by top talker was not going to help.

Again, the negative category is different from the positive categories.   If the best opportunity for policies reside in the undifferentiated group, we need to find new ways to categorize them than by using their names when individually each makes just a single comment.   The diversity within the negative group is too high to base any policy on.

Undetermined: ambiguous sentiment

Yet another type of negative category is one that collects all cases where the items are too ambiguous to assign to a particular positive category.   In my real-life project, this negative category was the biggest challenge.   It is easy to see the challenge also in the analogy of blog-site comments.   Many comments are relevant and on topic and may even come from frequent commenters but the sentiment of the comment is ambiguous.   These are relevant comments but we can not decide whether they are supporting the topic, arguing against it, or asking a question about it.   Such comments may involve further engagement (perhaps by the author) but even these subsequent messages may be ambiguous about the overall sentiment.

In blog comments, these appear as comments at either extreme of being too short or too long.

Examples of too short comment would be short expression such as “Thanks”, or “I like this”.   These expressions convey information about the commenter’s appreciation of the post, but they do not say anything about the sentiments I am seeking.   Someone who expresses thanks may intend to use this as an confirmation of why he disagrees with me.   Alternatively, he may be expressing a gratitude for inspiring a thought he did not wish to share with me.   These comments are relevant because they are on topic, but they can not be placed in one of the available positive categories.

At the other extreme or comments that are too long, the wall-of-text responses that ends that does not clearly state support, disagreement, request clarification for the post or one of its sections.  One example of a too-long comment are trackbacks where other blogs reference the current post.   These remote blog posts are often lengthy discussions using this post as one of its references.   Trackbacks are usually relevant to the post but their content is unlikely to be fall into one of the available positive categories.   I mention trackbacks because the host of this site (WordPress) treats trackbacks as the same thing as a comment.  Most of the non-spam comments reported for this blog are actually trackbacks.   Nearly all of the trackbacks are to my other posts on this same blog site in my attempt to cross-link the posts.   The trackback illustrates the difficulty of assigning this to one of the available positive categories.   The trackbacks are relevant but the purpose is to build upon the content instead of clearly agreeing or disagreeing with it.

The undetermined category is another negative category.   In the context of this imagined project, it is one where we recognize that the comment is relevant to the content but it is ambiguous as to which of the available positive categories it should belong to.   In some cases, this category may get so prevalent that there is a need to make new categories.  For example, on this site, I would distinguish my own cross-link trackbacks from the rest because these serve a consistent purpose.   However, if the scenario were different where the abundant trackbacks came from other sites, I suspect there would be no easy way to distinguish them with separate positive categories.   More likely each would have its own distinct purpose for referencing the content so that their category will be the negative category of undetermined.

Unexpected: misplaced comment

The last family of negative categories concerns the unexpected.   This was an important category in my real-life project because the unexpected was not supposed to happen at all, but we had a category to capture the cases when they did occur.   In my imagined comment-analysis project, these are the comments that clearly don’t belong to this particular topic.   In popular sites with lots of recent articles, many users will have multiple tabs open to different articles on the same site and write a good comment but submit it to the wrong article.   These comments are mistakes as opposed to spam, and they may come from frequent contributors and they may have a definite sentiment such as agreeing to some post.   The problem is that they do not belong to the post where they are submitted.

This last category best illustrates the purpose of a negative category.   The category of unexpected results is similar to an exception catcher in code.  Anything that appears in the unexpected category is something that needs investigation.   In my real-world project, entities falling into this category demanded immediate attention.   A non-empty unexpected category was a trigger for in depth explanation to understand what caused this to happen.   The causes may be a flaw in the handling of the data, or it may be clue of errors by the subjects of observations.   More importantly, this may be the early clue for discovering something new about the real world.

The unexpected category is never a topic of analysis itself.  There is no value in making policy based on volume of unexpected results.   Instead, the unexpected category justified and directs more in depth investigation into explaining the members within that category.

This should be the same same response for all of the negative categories.  If the negative category draws attention, the appropriate reaction is to dive in and find out how to divide it into new positive categories so that they may support analysis of specific policies or decisions based on these categories.   We have negative categories for the uninteresting elements.