Text Classification Algorithms: A Survey
Repo kk7nc/Text_Classification, read from GitHub 1 min ago. kk7nc has not joined Take My Repo.
kk7nc/Text_Classification is a public repo on GitHub by kk7nc. Text Classification Algorithms: A Survey. It declares the MIT license on GitHub. It is written mainly in Python, has 1,819 stars and last changed on 2025-04-01. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-18.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Referenced paper : Text Classification Algorithms: A Survey
Text feature extraction and pre-processing for classification algorithms are very significant. In this section, we start to talk about text cleaning since most of documents contain a lot of noise. In this part, we discuss two primary methods of text feature extractions- word embedding and weighted word.
In Natural Language Processing (NLP), most of the text and documents contain many words that are redundant for text classification, such as stopwords, miss-spellings, slangs, and etc. In this section, we briefly explain some techniques and methods for text cleaning and pre-processing text documents. In many algorithms like statistical and probabilistic learning methods, noise and unnecessary features can negatively affect the overall perfomance. So, elimination of these features are extremely important.
Tokenization is the process of breaking down a stream of text into words, phrases, symbols, or any other meaningful elements called tokens. The main goal of this step is to extract individual words in a sentence. Along with text classifcation, in text mining, it is necessay to incorporate a parser in the pipeline which performs the tokenization of the documents; for example:
sentence:
After sleeping for four hours, he decided to sleep for another four
In this case, the tokens are as follows:
{'After', 'sleeping', 'for', 'four', 'hours', 'he', 'decided', 'to', 'sleep', 'for', 'another', 'four'}
Here is python code for Tokenization:
from nltk.tokenize import word_tokenize text = "After sleeping for four hours, he decided to sleep for another four" tokens = word_tokenize(text) print(tokens)
Text and document classification over social media, such as Twitter, Facebook, and so on is usually affected by the noisy nature (abbreviations, irregular forms) of the text corpuses.
Here is an exmple from geeksforgeeks
from nltk.corpus import stopwords from nltk.tokenize import word_tokenize example_sent = "This is a sample sentence, showing off the stop words filtration." stop_words = set(stopwords.words('english')) word_tokens = word_tokenize(example_sent) filtered_sentence = [w for w in word_tokens if not w in stop_words] filtered_sentence = [] for w in word_tokens: if w not in stop_words: filtered_sentence.append(w) print(word_tokens) print(filtered_sentence)
Output:
['This', 'is', 'a', 'sample', 'sentence', ',', 'showing', 'off', 'the', 'stop', 'words', 'filtration', '.'] ['This', 'sample', 'sentence', ',', 'showing', 'stop', 'words', 'filtration', '.']
Sentences can contain a mixture of uppercase and lower case letters. Multiple sentences make up a text document. To reduce the problem space, the most common approach is to reduce everything to lower case. This brings all words in a document in same space, but it often changes the meaning of some words, such as "US" to "us" where first one represents the United States of America and second one is a pronoun. To solve this, slang and abbreviation converters can be applied.
text = "The United States of America (USA) or America, is a federal republic composed of 50 states" print(text) print(text.lower())
Output:
"The United States of America (USA) or America, is a federal republic composed of 50 states" "the united states of america (usa) or america, is a federal republic composed of 50 states"
Slangs and abbreviations can cause problems while executing the pre-processing steps. An abbreviation is a shortened form of a word, such as SVM stand for Support Vector Machine. Slang is a version of language that depicts informal conversation or text that has different meaning, such as "lost the plot", it essentially means that 'they've gone mad'. Common method to deal with these words is converting them to formal language.
Another issue of text cleaning as a pre-processing step is noise removal. Text documents generally contains characters like punctuations or special characters and they are not necessary for text mining or classification purposes. Although punctuation is critical to understand the meaning of the sentence, but it can affect the classification algorithms negatively.
Here is simple code to remove standard noise from text:
def text_cleaner(text): rules = [ {r'>\s+': u'>'}, # remove spaces after a tag opens or closes {r'\s+': u' '}, # replace consecutive spaces {r'\s*<br\s*/?>\s*': u'\n'}, # newline after a <br> {r'</(div)\s*>\s*': u'\n'}, # newline after </p> and </div> and <h1/>... {r'</(p|h\d)\s*>\s*': u'\n\n'}, # newline after </p> and </div> and <h1/>... {r'<head>.*<\s*(/head|body)[^>]*>': u''}, # remove <head> to </head> {r'<a\s+href="([^"]+)"[^>]*>.*</a>': r'\1'}, # show links instead of texts {r'[ \t]*<[^<]*?/?>': u''}, # remove remaining tags {r'^\s+': u''} # remove spaces at the beginning ] for rule in rules: for (k, v) in rule.items(): regex = re.compile(k) text = regex.sub(v, text) text = text.rstrip() return text.lower()
An optional part of the pre-processing step is correcting the misspelled words. Different techniques, such as hashing-based and context-sensitive spelling correction techniques, or spelling correction using trie and damerau-levenshtein distance bigram have been introduced to tackle this issue.
from autocorrect import spell print spell('caaaar') print spell(u'mussage') print spell(u'survice') print spell(u'hte')
Result:
caesar message service the
Text Stemming is modifying a word to obtain its variants using different linguistic processeses like affixation (addition of affixes). For example, the stem of the word "studying" is "study", to which -ing.
Here is an example of Stemming from NLTK
from nltk.stem import PorterStemmer from nltk.tokenize import sent_tokenize, word_tokenize ps = PorterStemmer() example_words = ["python","pythoner","pythoning","pythoned","pythonly"] for w in example_words: print(ps.stem(w))
Result:
python python python python pythonli
Text lemmatization is the process of eliminating redundant prefix or suffix of a word and extract the base word (lemma).
from nltk.stem import WordNetLemmatizer lemmatizer = WordNetLemmatizer() print(lemmatizer.lemmatize("cats"))
Different word embedding procedures have been proposed to translate these unigrams into consummable input for machine learning algorithms. A very simple way to perform such embedding is term-frequency~(TF) where each word will be mapped to a number corresponding to the number of occurrence of that word in the whole corpora. The other term frequency functions have been also used that represent word-frequency as Boolean or logarithmically scaled number. Here, each document will be converted to a vector of same length containing the frequency of the words in that document. Although such approach may seem very intuitive but it suffers from the fact that particular words that are used very commonly in language literature might dominate this sort of word representations.
Original from https://code.google.com/p/word2vec/
I’ve copied it to a github project so that I can apply and track community patches (starting with capability for Mac OS X compilation).
There seems to be a segfault in the compute-accuracy utility.
To get started:
cd scripts && ./demo-word.sh
Original README text follows:
This tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research purposes.
this code provides an implementation of the Continuous Bag-of-Words (CBOW) and the Skip-gram model (SG), as well as several demo scripts.
Given a text corpus, the word2vec tool learns a vector for every word in the vocabulary using the Continuous Bag-of-Words or the Skip-Gram neural network architectures. The user should specify the following: - desired vector dimensionality (size of the context window for either the Skip-Gram or the Continuous Bag-of-Words model), training algorithm (hierarchical softmax and / or negative sampling), threshold for downsampling the frequent words, number of threads to use, format of the output word vector file (text or binary).
Usually, other hyper-parameters, such as the learning rate do not need to be tuned for different training sets.
The script demo-word.sh downloads a small (100MB) text corpus from the web, and trains a small word vector model. After the training is finished, users can interactively explore the similarity of the words.
More information about the scripts is provided at https://code.google.com/p/word2vec/
An implementation of the GloVe model for learning word representations is provided, and describe how to download web-dataset vectors or train your own. See the project page or the paper for more information on glove vectors.
ELMo is a deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). These word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. They can be easily added to existing models and significantly improve the state of the art across a broad range of challenging NLP problems, including question answering, textual entailment and sentiment analysis.
ELMo representations are:
Tensorflow implementation
Tensorflow implementation of the pretrained biLM used to compute ELMo representations from "Deep contextualized word representations".
This repository supports both training biLMs and using pre-trained models for prediction.
We also have a pytorch implementation available in AllenNLP.
You may also find it easier to use the version provided in Tensorflow Hub if you just like to make predictions.
pre-trained models:
We have got several pre-trained English language biLMs available for use. Each model is specified with two separate files, a JSON formatted "options" file with hyperparameters and a hdf5 formatted file with the model weights. Links to the pre-trained models are available here.
There are three ways to integrate ELMo representations into a downstream task, depending on your use case.
We have used all of these methods in the past for various use cases. #1 is necessary for evaluating at test time on unseen data (e.g. public SQuAD leaderboard). #2 is a good compromise for large datasets where the size of the file in is unfeasible (SNLI, SQuAD). #3 is a good choice for smaller datasets or in cases where you'd like to use ELMo in other frameworks.
In all cases, the process roughly follows the same steps. First, create a Batcher (or TokenBatcher for #2) to translate tokenized strings to numpy arrays of character (or token) ids. Then, load the pretrained ELMo model (class BidirectionalLanguageModel). Finally, for steps #1 and #2 use weight_layers to compute the final ELMo representations. For #3, use BidirectionalLanguageModel to write all the intermediate layers to a file.
Architecture of the language model applied to an example sentence [Reference: arXiv paper].
fastText is a library for efficient learning of word representations and sentence classification.
Github: facebookresearch/fastText
Models
Supplementary data :
FAQ
You can find answers to frequently asked questions on Their project website.
Cheatsheet
Also a cheatsheet is provided full of useful one-liners.
Term frequency is Bag of words that is one of the simplest techniques of text feature extraction. This method is based on counting number of the words in each document and assign it to feature space.
The mathematical representation of weight of a term in a document by Tf-idf is given:
Where N is number of documents and df(t) is the number of documents containing the term t in the corpus. The first part would improve recall and the later would improve the precision of the word embedding. Although tf-idf tries to overcome the problem of common terms in document, it still suffers from some other descriptive limitations. Namely, tf-idf cannot account for the similarity between words in the document since each word is presented as an index. In the recent years, with development of more complex models, such as neural nets, new methods has been presented that can incorporate concepts, such as similarity of words and part of speech tagging. This work uses, word2vec and Glove, two of the most common methods that have been successfully used for deep learning techniques.
from sklearn.feature_extraction.text import TfidfVectorizer def loadData(X_train, X_test,MAX_NB_WORDS=75000): vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS) X_train = vectorizer_x.fit_transform(X_train).toarray() X_test = vectorizer_x.transform(X_test).toarray() print("tf-idf with",str(np.array(X_train).shape[1]),"features") return (X_train,X_test)
| Model | Advantages | Limitation |
| Weighted Words |
|
|
| TF-IDF |
|
|
| Word2Vec |
|
|
| GloVe (Pre-Trained) |
|
|
| GloVe (Trained) |
|
|
| FastText |
|
|
| Contextualized Word Representations |
|
|
Principle component analysis~(PCA) is the most popular technique in multivariate analysis and dimensionality reduction. PCA is a method to identify a subspace in which the data approximately lies. This means finding new variables that are uncorrelated and maximizing the variance to preserve as much variability as possible.
Example of PCA on text dataset (20newsgroups) from tf-idf with 75000 features to 2000 components:
from sklearn.feature_extraction.text import TfidfVectorizer import numpy as np def TFIDF(X_train, X_test, MAX_NB_WORDS=75000): vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS) X_train = vectorizer_x.fit_transform(X_train).toarray() X_test = vectorizer_x.transform(X_test).toarray() print("tf-idf with", str(np.array(X_train).shape[1]), "features") return (X_train, X_test) from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target X_train,X_test = TFIDF(X_train,X_test) from sklearn.decomposition import PCA pca = PCA(n_components=2000) X_train_new = pca.fit_transform(X_train) X_test_new = pca.transform(X_test) print("train with old features: ",np.array(X_train).shape) print("train with new features:" ,np.array(X_train_new).shape) print("test with old features: ",np.array(X_test).shape) print("test with new features:" ,np.array(X_test_new).shape)
output:
tf-idf with 75000 features train with old features: (11314, 75000) train with new features: (11314, 2000) test with old features: (7532, 75000) test with new features: (7532, 2000)
Linear Discriminant Analysis (LDA) is another commonly used technique for data classification and dimensionality reduction. LDA is particularly helpful where the within-class frequencies are unequal and their performances have been evaluated on randomly generated test data. Class-dependent and class-independent transformation are two approaches in LDA where the ratio of between-class-variance to within-class-variance and the ratio of the overall-variance to within-class-variance are used respectively.
from sklearn.feature_extraction.text import TfidfVectorizer import numpy as np from sklearn.discriminant_analysis import LinearDiscriminantAnalysis def TFIDF(X_train, X_test, MAX_NB_WORDS=75000): vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS) X_train = vectorizer_x.fit_transform(X_train).toarray() X_test = vectorizer_x.transform(X_test).toarray() print("tf-idf with", str(np.array(X_train).shape[1]), "features") return (X_train, X_test) from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target X_train,X_test = TFIDF(X_train,X_test) LDA = LinearDiscriminantAnalysis(n_components=15) X_train_new = LDA.fit(X_train,y_train) X_train_new = LDA.transform(X_train) X_test_new = LDA.transform(X_test) print("train with old features: ",np.array(X_train).shape) print("train with new features:" ,np.array(X_train_new).shape) print("test with old features: ",np.array(X_test).shape) print("test with new features:" ,np.array(X_test_new).shape)
output:
tf-idf with 75000 features train with old features: (11314, 75000) train with new features: (11314, 15) test with old features: (7532, 75000) test with new features: (7532, 15)
from sklearn.feature_extraction.text import TfidfVectorizer import numpy as np from sklearn.decomposition import NMF def TFIDF(X_train, X_test, MAX_NB_WORDS=75000): vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS) X_train = vectorizer_x.fit_transform(X_train).toarray() X_test = vectorizer_x.transform(X_test).toarray() print("tf-idf with", str(np.array(X_train).shape[1]), "features") return (X_train, X_test) from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target X_train,X_test = TFIDF(X_train,X_test) NMF_ = NMF(n_components=2000) X_train_new = NMF_.fit(X_train) X_train_new = NMF_.transform(X_train) X_test_new = NMF_.transform(X_test) print("train with old features: ",np.array(X_train).shape) print("train with new features:" ,np.array(X_train_new).shape) print("test with old features: ",np.array(X_test).shape) print("test with new features:" ,np.array(X_test_new))
output:
tf-idf with 75000 features train with old features: (11314, 75000) train with new features: (11314, 2000) test with old features: (7532, 75000) test with new features: (7532, 2000)
Random projection or random feature is a dimensionality reduction technique mostly used for very large volume dataset or very high dimensional feature space. Text and document, especially with weighted feature extraction, can contain a huge number of underlying features. Many researchers addressed Random Projection for text data for text mining, text classification and/or dimensionality reduction. We start to review some random projection techniques.
from sklearn.feature_extraction.text import TfidfVectorizer import numpy as np def TFIDF(X_train, X_test, MAX_NB_WORDS=75000): vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS) X_train = vectorizer_x.fit_transform(X_train).toarray() X_test = vectorizer_x.transform(X_test).toarray() print("tf-idf with", str(np.array(X_train).shape[1]), "features") return (X_train, X_test) from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target X_train,X_test = TFIDF(X_train,X_test) from sklearn import random_projection RandomProjection = random_projection.GaussianRandomProjection(n_components=2000) X_train_new = RandomProjection.fit_transform(X_train) X_test_new = RandomProjection.transform(X_test) print("train with old features: ",np.array(X_train).shape) print("train with new features:" ,np.array(X_train_new).shape) print("test with old features: ",np.array(X_test).shape) print("test with new features:" ,np.array(X_test_new).shape)
output:
tf-idf with 75000 features train with old features: (11314, 75000) train with new features: (11314, 2000) test with old features: (7532, 75000) test with new features: (7532, 2000)
Autoencoder is a neural network technique that is trained to attempt to map its input to its output. The autoencoder as dimensional reduction methods have achieved great success via the powerful reprehensibility of neural networks. The main idea is, one hidden layer between the input and output layers with fewer neurons can be used to reduce the dimension of feature space. Specially for texts, documents, and sequences that contains many features, autoencoder could help to process data faster and more efficiently.
from keras.layers import Input, Dense from keras.models import Model # this is the size of our encoded representations encoding_dim = 1500 # this is our input placeholder input = Input(shape=(n,)) # "encoded" is the encoded representation of the input encoded = Dense(encoding_dim, activation='relu')(input) # "decoded" is the lossy reconstruction of the input decoded = Dense(n, activation='sigmoid')(encoded) # this model maps an input to its reconstruction autoencoder = Model(input, decoded) # this model maps an input to its encoded representation encoder = Model(input, encoded) encoded_input = Input(shape=(encoding_dim,)) # retrieve the last layer of the autoencoder model decoder_layer = autoencoder.layers[-1] # create the decoder model decoder = Model(encoded_input, decoder_layer(encoded_input)) autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
Load data:
autoencoder.fit(x_train, x_train, epochs=50, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
T-distributed Stochastic Neighbor Embedding (T-SNE) is a nonlinear dimensionality reduction technique for embedding high-dimensional data which is mostly used for visualization in a low-dimensional space. This approach is based on G. Hinton and ST. Roweis . SNE works by converting the high dimensional Euclidean distances into conditional probabilities which represent similarities.
Example:
import numpy as np from sklearn.manifold import TSNE X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]]) X_embedded = TSNE(n_components=2).fit_transform(X) X_embedded.shape
Example of Glove and T-SNE for text:
The first version of Rocchio algorithm is introduced by rocchio in 1971 to use relevance feedback in querying full-text databases. Since then many researchers have addressed and developed this technique for text and document classification. This method uses TF-IDF weights for each informative word instead of a set of Boolean features. Using a training set of documents, Rocchio's algorithm builds a prototype vector for each class which is an average vector over all training document vectors that belongs to a certain class. Then, it will assign each test document to a class with maximum similarity that between test document and each of the prototype vectors.
When in nearest centroid classifier, we used for text as input data for classification with tf-idf vectors, this classifier is known as the Rocchio classifier.
from sklearn.neighbors.nearest_centroid import NearestCentroid from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', NearestCentroid()), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
Output:
precision recall f1-score support 0 0.75 0.49 0.60 319 1 0.44 0.76 0.56 389 2 0.75 0.68 0.71 394 3 0.71 0.59 0.65 392 4 0.81 0.71 0.76 385 5 0.83 0.66 0.74 395 6 0.49 0.88 0.63 390 7 0.86 0.76 0.80 396 8 0.91 0.86 0.89 398 9 0.85 0.79 0.82 397 10 0.95 0.80 0.87 399 11 0.94 0.66 0.78 396 12 0.40 0.70 0.51 393 13 0.84 0.49 0.62 396 14 0.89 0.72 0.80 394 15 0.55 0.73 0.63 398 16 0.68 0.76 0.71 364 17 0.97 0.70 0.81 376 18 0.54 0.53 0.53 310 19 0.58 0.39 0.47 251 avg / total 0.74 0.69 0.70 7532
Boosting is a Ensemble learning meta-algorithm for primarily reducing variance in supervised learning. It is basically a family of machine learning algorithms that convert weak learners to strong ones. Boosting is based on the question posed by Michael Kearns and Leslie Valiant (1988, 1989) Can a set of weak learners create a single strong learner? A weak learner is defined to be a Classification that is only slightly correlated with the true classification (it can label examples better than random guessing). In contrast, a strong learner is a classifier that is arbitrarily well-correlated with the true classification.
from sklearn.ensemble import GradientBoostingClassifier from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', GradientBoostingClassifier(n_estimators=100)), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
Output:
precision recall f1-score support 0 0.81 0.66 0.73 319 1 0.69 0.70 0.69 389 2 0.70 0.68 0.69 394 3 0.64 0.72 0.68 392 4 0.79 0.79 0.79 385 5 0.83 0.64 0.72 395 6 0.81 0.84 0.82 390 7 0.84 0.75 0.79 396 8 0.90 0.86 0.88 398 9 0.90 0.85 0.88 397 10 0.93 0.86 0.90 399 11 0.90 0.81 0.85 396 12 0.33 0.69 0.45 393 13 0.87 0.72 0.79 396 14 0.87 0.84 0.85 394 15 0.85 0.87 0.86 398 16 0.65 0.78 0.71 364 17 0.96 0.74 0.84 376 18 0.70 0.55 0.62 310 19 0.62 0.56 0.59 251 avg / total 0.78 0.75 0.76 7532
from sklearn.ensemble import BaggingClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', BaggingClassifier(KNeighborsClassifier())), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
Output:
precision recall f1-score support 0 0.57 0.74 0.65 319 1 0.60 0.56 0.58 389 2 0.62 0.54 0.58 394 3 0.54 0.57 0.55 392 4 0.63 0.54 0.58 385 5 0.68 0.62 0.65 395 6 0.55 0.46 0.50 390 7 0.77 0.67 0.72 396 8 0.79 0.82 0.80 398 9 0.74 0.77 0.76 397 10 0.81 0.86 0.83 399 11 0.74 0.85 0.79 396 12 0.67 0.49 0.57 393 13 0.78 0.51 0.62 396 14 0.76 0.78 0.77 394 15 0.71 0.81 0.76 398 16 0.73 0.73 0.73 364 17 0.64 0.79 0.71 376 18 0.45 0.69 0.54 310 19 0.61 0.54 0.57 251 avg / total 0.67 0.67 0.67 7532
Naïve Bayes text classification has been used in industry and academia for a long time (introduced by Thomas Bayes between 1701-1761). However, this technique is being studied since the 1950s for text and document categorization. Naive Bayes Classifier (NBC) is generative model which is widely used in Information Retrieval. Many researchers addressed and developed this technique for their applications. We start with the most basic version of NBC which developed by using term-frequency (Bag of Word) fetaure extraction technique by counting number of words in documents
from sklearn.naive_bayes import MultinomialNB from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', MultinomialNB()), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
Output:
precision recall f1-score support 0 0.80 0.52 0.63 319 1 0.81 0.65 0.72 389 2 0.82 0.65 0.73 394 3 0.67 0.78 0.72 392 4 0.86 0.77 0.81 385 5 0.89 0.75 0.82 395 6 0.93 0.69 0.80 390 7 0.85 0.92 0.88 396 8 0.94 0.93 0.93 398 9 0.92 0.90 0.91 397 10 0.89 0.97 0.93 399 11 0.59 0.97 0.74 396 12 0.84 0.60 0.70 393 13 0.92 0.74 0.82 396 14 0.84 0.89 0.87 394 15 0.44 0.98 0.61 398 16 0.64 0.94 0.76 364 17 0.93 0.91 0.92 376 18 0.96 0.42 0.58 310 19 0.97 0.14 0.24 251 avg / total 0.82 0.77 0.77 7532
R In machine learning, the k-nearest neighbors algorithm (kNN) is a non-parametric technique used for classification. This method is used in Natural-language processing (NLP) as a text classification technique in many researches in the past decades.
from sklearn.neighbors import KNeighborsClassifier from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', KNeighborsClassifier()), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
Output:
precision recall f1-score support 0 0.43 0.76 0.55 319 1 0.50 0.61 0.55 389 2 0.56 0.57 0.57 394 3 0.53 0.58 0.56 392 4 0.59 0.56 0.57 385 5 0.69 0.60 0.64 395 6 0.58 0.45 0.51 390 7 0.75 0.69 0.72 396 8 0.84 0.81 0.82 398 9 0.77 0.72 0.74 397 10 0.85 0.84 0.84 399 11 0.76 0.84 0.80 396 12 0.70 0.50 0.58 393 13 0.82 0.49 0.62 396 14 0.79 0.76 0.78 394 15 0.75 0.76 0.76 398 16 0.70 0.73 0.72 364 17 0.62 0.76 0.69 376 18 0.55 0.61 0.58 310 19 0.56 0.49 0.52 251 avg / total 0.67 0.66 0.66 7532
The original version of SVM was introduced by Vapnik and Chervonenkis in 1963. The early 1990s, nonlinear version was addressed by BE. Boser et al.. Original version of SVM was designed for binary classification problem, but Many researchers have worked on multi-class problem using this authoritative technique.
The advantages of support vector machines are based on scikit-learn page:
The disadvantages of support vector machines include:
from sklearn.svm import LinearSVC from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', LinearSVC()), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
output:
precision recall f1-score support 0 0.82 0.80 0.81 319 1 0.76 0.80 0.78 389 2 0.77 0.73 0.75 394 3 0.71 0.76 0.74 392 4 0.84 0.86 0.85 385 5 0.87 0.76 0.81 395 6 0.83 0.91 0.87 390 7 0.92 0.91 0.91 396 8 0.95 0.95 0.95 398 9 0.92 0.95 0.93 397 10 0.96 0.98 0.97 399 11 0.93 0.94 0.93 396 12 0.81 0.79 0.80 393 13 0.90 0.87 0.88 396 14 0.90 0.93 0.92 394 15 0.84 0.93 0.88 398 16 0.75 0.92 0.82 364 17 0.97 0.89 0.93 376 18 0.82 0.62 0.71 310 19 0.75 0.61 0.68 251 avg / total 0.85 0.85 0.85 7532
One of earlier classification algorithm for text and data mining is decision tree. Decision tree classifiers (DTC's) are used successfully in many diverse areas of classification. The structure of this technique includes a hierarchical decomposition of the data space (only train dataset). Decision tree as classification task was introduced by D. Morgan and developed by JR. Quinlan. The main idea is creating trees based on the attributes of the data points, but the challenge is determining which attribute should be in parent level and which one should be in child level. To solve this problem, De Mantaras introduced statistical modeling for feature selection in tree.
from sklearn import tree from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', tree.DecisionTreeClassifier()), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
output:
precision recall f1-score support 0 0.51 0.48 0.49 319 1 0.42 0.42 0.42 389 2 0.51 0.56 0.53 394 3 0.46 0.42 0.44 392 4 0.50 0.56 0.53 385 5 0.50 0.47 0.48 395 6 0.66 0.73 0.69 390 7 0.60 0.59 0.59 396 8 0.66 0.72 0.69 398 9 0.53 0.55 0.54 397 10 0.68 0.66 0.67 399 11 0.73 0.69 0.71 396 12 0.34 0.33 0.33 393 13 0.52 0.42 0.46 396 14 0.65 0.62 0.63 394 15 0.68 0.72 0.70 398 16 0.49 0.62 0.55 364 17 0.78 0.60 0.68 376 18 0.38 0.38 0.38 310 19 0.32 0.32 0.32 251 avg / total 0.55 0.55 0.55 7532
Random forests or random decision forests technique is an ensemble learning method for text classification. This method was introduced by T. Kam Ho in 1995 for first time which used t trees in parallel. This technique was later developed by L. Breiman in 1999 that they found converged for RF as a margin measure.
from sklearn.ensemble import RandomForestClassifier from sklearn.pipeline import Pipeline from sklearn import metrics from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.datasets import fetch_20newsgroups newsgroups_train = fetch_20newsgroups(subset='train') newsgroups_test = fetch_20newsgroups(subset='test') X_train = newsgroups_train.data X_test = newsgroups_test.data y_train = newsgroups_train.target y_test = newsgroups_test.target text_clf = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf', RandomForestClassifier(n_estimators=100)), ]) text_clf.fit(X_train, y_train) predicted = text_clf.predict(X_test) print(metrics.classification_report(y_test, predicted))
output:
precision recall f1-score support 0 0.69 0.63 0.66 319 1 0.56 0.69 0.62 389 2 0.67 0.78 0.72 394 3 0.67 0.67 0.67 392 4 0.71 0.78 0.74 385 5 0.78 0.68 0.73 395 6 0.74 0.92 0.82 390 7 0.81 0.79 0.80 396 8 0.90 0.89 0.90 398 9 0.80 0.89 0.84 397 10 0.90 0.93 0.91 399 11 0.89 0.91 0.90 396 12 0.68 0.49 0.57 393 13 0.83 0.65 0.73 396 14 0.81 0.88 0.84 394 15 0.68 0.91 0.78 398 16 0.67 0.86 0.75 364 17 0.93 0.78 0.85 376 18 0.86 0.48 0.61 310 19 0.79 0.31 0.45 251 avg / total 0.77 0.76 0.75 7532
Conditional Random Field (CRF) is an undirected graphical model as shown in figure. CRFs state the conditional probability of a label sequence Y give a sequence of observation X i.e. P(Y|X). CRFs can incorporate complex features of observation sequence without violating the independence assumption by modeling the conditional probability of the label sequences rather than the joint probability P(X,Y). The concept of clique which is a fully connected subgraph and clique potential are used for computing P(X|Y). Considering one potential function for each clique of the graph, the probability of a variable configuration corresponds to the product of a series of non-negative potential function. The value computed by each potential function is equivalent to the probability of the variables in its corresponding clique taken on a particular configuration.
Example from Here Let’s use CoNLL 2002 data to build a NER system CoNLL2002 corpus is available in NLTK. We use Spanish data.
import nltk import sklearn_crfsuite from sklearn_crfsuite import metrics nltk.corpus.conll2002.fileids() train_sents = list(nltk.corpus.conll2002.iob_sents('esp.train')) test_sents = list(nltk.corpus.conll2002.iob_sents('esp.testb'))
sklearn-crfsuite (and python-crfsuite) supports several feature formats; here we use feature dicts.
def word2features(sent, i): word = sent[i][0] postag = sent[i][1] features = { 'bias': 1.0, 'word.lower()': word.lower(), 'word[-3:]': word[-3:], 'word[-2:]': word[-2:], 'word.isupper()': word.isupper(), 'word.istitle()': word.istitle(), 'word.isdigit()': word.isdigit(), 'postag': postag, 'postag[:2]': postag[:2], } if i > 0: word1 = sent[i-1][0] postag1 = sent[i-1][1] features.update({ '-1:word.lower()': word1.lower(), '-1:word.istitle()': word1.istitle(), '-1:word.isupper()': word1.isupper(), '-1:postag': postag1, '-1:postag[:2]': postag1[:2], }) else: features['BOS'] = True if i < len(sent)-1: word1 = sent[i+1][0] postag1 = sent[i+1][1] features.update({ '+1:word.lower()': word1.lower(), '+1:word.istitle()': word1.istitle(), '+1:word.isupper()': word1.isupper(), '+1:postag': postag1, '+1:postag[:2]': postag1[:2], }) else: features['EOS'] = True return features def sent2features(sent): return [word2features(sent, i) for i in range(len(sent))] def sent2labels(sent): return [label for token, postag, label in sent] def sent2tokens(sent): return [token for token, postag, label in sent] X_train = [sent2features(s) for s in train_sents] y_train = [sent2labels(s) for s in train_sents] X_test = [sent2features(s) for s in test_sents] y_test = [sent2labels(s) for s in test_sents]
To see all possible CRF parameters check its docstring. Here we are useing L-BFGS training algorithm (it is default) with Elastic Net (L1 + L2) regularization.
kk7nc/Text_Classification is a public repo on GitHub by kk7nc. Text Classification Algorithms: A Survey.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2025-04-01. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
Named Entity Recognition (LSTM + CRF) - Tensorflow

A High-Quality Real Time Upscaler for Anime Video
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
A libre lightweight streaming front-end for Android.
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
A MNIST-like fashion product database. Benchmark :point_down:
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/kk7nc/Text_ClassificationTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.