Explaining similarity-ts: A Code Duplication Detection Tool for the AI Era

As AI-generated code from tools like Claude Code becomes commonplace, there’s a growing trend of similar code accumulating within projects. Such duplicate code reduces maintainability and contributes to technical debt. The tool we’re introducing today, “similarity-ts,” is a high-speed code duplication detector developed to address these challenges¹.

similarity-ts is implemented in Rust and supports multiple programming languages including TypeScript/JavaScript, Python, Rust, Go, and Java. Notably, TypeScript/JavaScript support has reached “Production Ready” status, operating at high speed even in large-scale projects².

Key Features and Functionality

The primary feature of similarity-ts is its adoption of Abstract Syntax Tree (AST)-based structural comparison. This enables detection of structurally similar code even when variable names differ.

Zero Configuration

Using the tool is remarkably simple. Just run the following command in your project’s root directory to start detecting duplicate code¹.

# Installation
cargo install similarity-ts

# Execution
similarity-ts

# Display detailed results
similarity-ts . --print

Multi-language Support

Currently supported languages and their maturity levels²:

Production Ready: TypeScript/JavaScript
Beta: Python, Rust
Experimental: Go, Java, C/C++, C#, Ruby, Elixir

Optimized parsers are used for each language, with oxc-parser for TypeScript/JavaScript and tree-sitter for Python and Rust.

Technical Architecture: The TSED Algorithm

At the core of similarity-ts is the TSED (Tree Similarity of Edit Distance) algorithm. Based on a paper published on arXiv, it determines code similarity by calculating tree structure edit distances¹.

Algorithm Overview

The TSED algorithm builds upon APTED (All Path Tree Edit Distance), applying penalties based on actual code volume to calculate similarity. This detects structural similarities that simple string comparisons would miss³.

Performance Optimizations

Developer mizchi notes that while initially implemented in TypeScript, the computational complexity caused crashes in large projects. After rewriting in Rust, the following optimizations were applied¹:

Bloom filter pre-filtering: Rapidly excludes clearly different function pairs (~5x speedup)
Multi-threaded processing: Parallelizes file reading and analysis (~4x speedup)
Efficient memory management: Optimizations leveraging Rust’s ownership system

These optimizations achieved approximately 50x overall speedup.

Usage Examples and Output

Running similarity-ts produces output like¹:

Similarity: 89.09%, Score: 8.0 points
  src/utils/getUserById.ts:4-12 getUserById
  src/utils/findUserById.ts:8-16 findUserById

This example shows that the getUserById and findUserById functions have 89.09% similarity. The score is calculated based on edit distance, with lower scores indicating higher similarity.

Use Cases and Future Outlook

Primary Use Cases

AI code generation quality control: Detecting and consolidating similar AI-generated code
Refactoring: Identifying and consolidating duplicate code
Code reviews: Checking if new code duplicates existing code
Technical debt visualization: Measuring project-wide duplication

Future Outlook

The developer lists the following planned improvements¹:

Faster type definition literal comparison
Code quality measurement features
More advanced semantic similarity detection

similarity-ts presents a new approach to code quality management in the AI era. It’s a tool that efficiently detects duplicate code developers inadvertently generate, contributing to more maintainable codebases.

For project details and installation instructions, please refer to the GitHub repository. Those interested in the technical details of the algorithm may find the related academic papers helpful.