This involves a "hate speech" or "spam" detection algorithm. (depending on the training data).
This will be based on ANN theory , and involve the backpropagation algorithm while not based on an actual neural network.
The training data will be labeled into two classes, "hate speech" and "non hate speech". Hate speech will have an expected value of 1 , representing certain hate speech, while non hate speech will have an expected value of -1.
So what we would like is for the software to be input a post and gives it a value of 1 or -1, or something much closer to one or the other.
During training The program will compute a value by performing the equivalent of forward propagation through the parameters we assign to each of multiple structures we will embed in the input ...then using this value and the epected value we will compute a cost function. We will then backpropagate through the heirachical structure of parameters in the direction that minimises the cost function.
When choosing a set of properties we would like to properly understand what we want the network to do. We know that both posts classified as hate speech and those not contain samples of words from a common pool (they share words). but we still want that the results for opposite classes to polarise. On top of this i beleive that this represents a complex system , in that small changes to the post (such as the prsence of the word "NOT" in an otherwise innocent statement) could complete reverse the polarity of the classification. And also the converse where large changes do nothing..which should be easy to imagine as a lengthy elaboration of either a well meaning message or the converse...even when that word NOT is a part of it, but in a different role. we would like to capture all of this information somehow , or else the system will not converge.
To do this we introduce three variables that are associated with each of the following structures. Individual words, whole sentences, and optionally whole posts.
Adjusting the values of these variables will be how the systems implements a type of POS tagging that is optimised for classifying different posts as either hate speech or not.
I will call them "permeability", "stream", and "angle", that will function as follows.
Every word will emit a stream from the left side and the right side, that will end at the limits of the sentence, i.e. the two periods at either end.
Each words stream will pour out at a unique rate (determined during training). Note that the reach of the different streams of each word in a sentence will have different symmetrical aspects, where a word nearer one boundary has more asymmetric reach, while a word near the center has more symmetric reach.
Every word will have a permeability associated with it. This is what it should do. The permeability will either scale up or scale down the value of another word in the sentences stream towards it. Permeability is local to a word in the sense that the other words have no direct way of knowing what the permeability of other words in the sentence have.
Then each word will have an angle. This is a value between 1 and -1. This value moves along with stream to every other word.
Finally we compute a value for the entire sentence and save it in a temporary variable.
This will be of the form of...for each word in the sentence, calculate the stream it has received form each word multiplied by the angle and divided by its own permeability.....then sum the results and store in a temporary variable temp_word .
Then the values of each temp_word in the sentence are summed stored in a variable temp_sentence.
And then the values of stream, permeability and angle for that particular sentence are calculated by scaling them by computing a function which takes these three variables as inputs and outputs a vector with each of them parametricised by the variable temp_sentence.
A little explanation...after training how should we classify the following two examples.
A. You are a monkey.
B. You are not a Monkey.
We should expect that we have trained for A to get a value of 1 and B a value of -1, despite that they share mostly the same words.
So i would like that all the angles of the words in A to be positive, while the results of stream and permeability should make all the temp_word variables to be large... the result of temp_sentence_A will be a large positivenumber..(All words have positve angle X large stream and high permeability)
Then for temp_sentence_B..it will have the roughly the same absolute value but the sign will be flipped i.e. negative , as we expect "Not" to have a negative angle , large stream and low permeability.....the negative angle with very large positive stream will do this sign change, while low permeability will keep the effective stream input low , from the other words...as a large stream input would result in a large positive temp_word_Not. offsetting the required negativity ,but with low permeability the value of the other words will dominate giving the resultant temp_sentence_B a large negative value.
So this is as we expect.
what about :
the inverse of that situation , with the NOT negating a good sentence instead of a bad one, will the same NOT need to have different properties?
C.You are very good.
D. you are not very good.
Well C should evaluate to -1 while D goes to 1
To be negative the words in C need negative angle, large stream and high permeability,
while to be negative D needs The exact same "Not" as in the other example,negative angle,very large positive stream and low permeability.
What we have arranged is for Not to affect the words in both examples by making their temp_words opposite of what they otherwise would be in sign ..separating two sentences with mostly the same words to two vastly opposing poles.
It should be noted that "You" had two opposing requirements for this to work...meaning that is a flaw...but this will be corrected by simply giving it in particular a lower permeability and a larger stream, so that it will have a more neutral effect in each example.
With the ability to adjust parameters we can have a large amount of detail in the system.
What happens is that we have exactly ONE equation that we use to find the value of any word. And hence sentence. That equstion takes in as parameters 3 variables from every other word in the sentence. Each word has similar variables with similar names but with different values on different words. A words variables do NOT contribute to its value in the equation. Only the other words in the sentence’s variables values affect it. This goes true round robin for calculating the value of each word. This may seem counter intuitive,,,but it will always result in the same word from two different sentences having different values….that is how we go about differentiating largely similar sentences. Ofcourse even when we initialise the 3 variables for each word randomly this will be true…however the system will be hopeless for classification because ultimately the pattern of the value of the final sentence (as calculated from the values of its words) will be random and you would not be able to tell between hate speech and good speech, or differentiate anything else either. But we will fix this by training the system to adjust the values of each words variables so they are no longer random , but optimised to generate sentences with one value for hate speech and a dimatrically opposite value for good one…based on the way each other word in the sentence influence the value of the sentence. So we will have an iterative loop were we adjust the variables of each word along the negative gradient of the cost function…..so we will do this .
ReplyDeleteRandomly initialise variables -----------(so cant distinguish anything between sentences…their values don’t correlate with anything)
Take a post of hate speech and give it an expected value of 1( good speech registers -1)
Calculate its value.with the randomly innitialised weights and the Equation
We will almost definitely NOT get the expected value—(because variable initialisation was random)
Maybe we get -0.2 for example
Calculate the cost as C = (expected value – ground value)----here it is 1-(-0.2) or 1,2.
Perform backpropagation, using the chain of values as a bsis for direction…i.e. the sentence value is the output note…then backpropagate to the word value..which is the hidden node…the input layer of nodes will be the 3 variables values or the first layer of nodes.
This is surprisingly like an ANN, but there are no weights or biases, or even neurons, despite that we can still backpropagate the error.
The backpropagation should adjust the value of the variables in the input layer (so their less random) …which will have an ultimate effect of determining indirectly the value each sentence has to a less random one and one either close to 1 or close to -1…by reducing the error…in this example its 1,2 untill the erro goes to 0 meaning that sentence will have the expected value of 1 for hate speech instead of the initial -0.2.
This supervised training will have as inputs many sentences that we have independentakly verified as either hate speech or not..in order to train the system.
So then after training we can input a random sentence and determine with a forward pass Using the now trained/non random 3 variables of each word--to get the value of the sentence, whether it is hate speech or not based on that value of the sentence.