We learn words at an early age so that we can communicate. We use words to communicate our needs, make new friends, and learn about various topics throughout our education. We spend our lives constantly learning new words and how to use them. This allows us to make our lives more successful and enjoyable. Who can't love words? My answer to this question is Anyone doing localization or translation of natural language!
The use of words creates problems that make these tasks even more difficult.
The first problem with words is that they are too many!
A large number of words necessitate large vocabulary dictionaries. Many words are also used at different times for different grammatical purposes.
Instead of individual words, this abstraction can use strings of words as the basis. These sequences will be different from the familiar clauses and clauses of a sentence. Each word string will perform a new, high-level grammatical function and will be grouped into string categories that perform the same function.
The first advantage of this approach will be the reduction in the number of discrete language units that need to be manipulated. Too many individual words will be reduced to a few categories.
If chosen carefully, each category performs one and only one high-level grammar function. Mapping to a single grammar function will remove the possible grammar and meaning of words in a sentence and replace it with a single grammatical structure. The variability of the probability of a sentence will decrease.
Using grammatical categories does not increase the likelihood of a particular text being low. However, each category will contain multiple strings of words, each of which will add to the category's possibilities. This will increase the overall probability of that category.
You can find the rest of the news
at this link:
