Materials & Methods

Initial dataset

This dataset provides comprehensive coverage of 16S rRNA sequences from 19,931 bacterial and 929 archaeal taxa. Because the dataset distinguishes organisms at a very fine taxonomic level, such as genus and species, it allows researchers to accurately compare and analyze evolutionary relationships between Bacteria and Archaea. The inclusion of RefSeq IDs and taxonomic identifiers facilitates interoperability with NCBI.

When clicking on the 'Download initial dataset' button, a .zip file (11.1MB) will be downloaded containing the initial dataset (.xlsx) and the metadata file (.json).

File specifications

Column description

Filtering dataset

Before the filtersteps were applied a cutoff of 10 was chosen. This was done because of the rule of 10. This rule states that for each parameter you need 10 times as much data.

The dataset has been filtered based on the following criteria:

After the filtering, the dataset was reduced to 20.199 entries, which is 854 fewer than the initial dataset.

Cut-off threshold

K-mer analysis

K-mer analysis has been performed by counting the number of occurrences of each present k-mer in each sequence (total number of sequences: 20915). A length of 8 bases has been chosen, because Wang et al. stated that “sizes of 6 and 8 bases were less accurate” than sizes of 8 and 9 bases, and that a size of 8 can be chosen over a size of 9 to reduce memory requirements given that “sizes of 8 and 9 bases gave nearly identical results” (2007). The k-mer counts have been saved into a pandas DataFrame, converted to a .json file.

Machine learning model exploration

For this project, two machine learning models using two different algorithms were selected and evaluated independently: Random Forest (RF) and Multinomial Naïve Bayes (MNB). The goal was to classify 16S rRNA sequences at the phylum level based on their k-mer representations.

This is an exploratory classification task, as we are working with a biologically complex dataset. K-mer representations can capture both conserved and variable regions, but distinguishing between closely related phyla remains a challenge. We are also investigating which models perform best in identifying taxonomic groups from sequence-derived features. Our dataset is imbalanced and incomplete at deeper taxonomic ranks, making phylum-level classification a practical and informative target. The aim is not to develop a final predictive system, but to evaluate model performance under realistic bioinformatics constraints, such as the imbalanced dataset.

Boosting algorithms were intentionally avoided. While they often achieve high accuracy (Anand, 2024), they are more prone to overfitting, especially in datasets with class imbalance, like ours, where some phyla are much more represented than others. This would hinder the generalizability of the model to unseen data.

Initially, we considered using a Cascading Random Forest approach, which builds a hierarchical classifier that mirrors the taxonomic structure and can be particularly effective for resolving deeper taxonomic levels, such as genus or species (Zhang et al., 2022). This method is capable of uncovering complex relationships, even when some classes have relatively small sample sizes. However, cascading classifiers are especially sensitive to class imbalance, a limitation that becomes more pronounced at lower taxonomic ranks.

Given the characteristics of our dataset, we chose not to pursue this approach. Our dataset is limited in both quality and coverage below the phylum level. The quality is affected by sequencing errors, the presence of ambiguous bases (denoted as 'N'), and incomplete or uncertain taxonomic annotations. Coverage refers to the number of sequences per taxonomic group, which is often insufficient in our dataset. Many phyla are represented by fewer than the commonly recommended minimum of 100 sequences per class, reducing statistical power and increasing the risk of overfitting. These limitations, combined with the sensitivity of cascading classifiers to class imbalance, made a standard Random Forest approach a more suitable choice for our exploratory analysis.

Instead, we opted for a standard Random Forest model. Random Forest is an ensemble method that is well-suited for high-dimensional, structured data such as k-mer frequency tables. It is known for its robustness, even in the presence of noisy or imbalanced data (Qi, 2012). Moreover, Random Forest offers interpretable outputs, such as feature importance scores and out-of-bag error rates, allowing us to understand which k-mers contribute most to the classification. These characteristics make it a strong candidate for exploratory classification tasks. We used GridSearchCV (GridSearchCV, z.d.) to tune hyperparameters and optimize model performance.

Multinomial Naïve Bayes (MNB) is a probabilistic classifier based on Bayes' theorem, typically used in text classification. It is especially efficient with discrete, frequency-based features such as word counts—or in our case, k-mer counts. While MNB assumes feature independence and ignores the order of features—which conflicts with biological realities like motif co-occurrence and sequence context—it has still been shown to perform well in many practical applications (Soria et al., 2011). Moreover, MNB has been previously applied to taxonomic classification using k-mer analysis (e.g., Wang et al., 2007), making it a reasonable baseline model for comparison.

Validation dataset

To construct the validation set, we obtained new sequences from the NCBI nucleotide database. We first searched for 16S rRNA sequences within the Bacteria domain, selecting 21 unique sequences that were not present in our original dataset. Using the same search criteria, we then selected 7 unique 16S rRNA sequences from the Archaea domain to further validate our model. To further challenge the model and assess its ability to generalize, we included a so-called “wildcard” sequence: an 18S rRNA sequence from the Fungi domain, representing a class the models were not trained on. This diverse set of sequences enables a comprehensive evaluation of the model’s classification performance and robustness across different taxonomic groups. This validation set is used to validate the machine learning models and to assess their performance on unseen data.

Download validation dataset (10kB)

When clicking on the 'Download validation dataset' button, a .zip file (10kB) will be downloaded containing the validation sequences (.FASTA) and the metadata file (.json).

File specifications