Benchmark: Filtering duplicates

Script Preparation code:


var array = [];
for (let i = 0; i < 100000; i++) {
    array.push({
        id: Math.floor((Math.random() * 10) + 1),
        name: 'name' + i
    });
}

Tests:

Set + filter
const seen = new Set(); const filtered = array.filter(el => { const duplicate = seen.has(el.id); seen.add(el.id); return !duplicate; });
Reduce
const filtered = array.reduce((acc, current) => { const x = acc.find(item => item.id === current.id); if (!x) { return acc.concat([current]); } else { return acc; } }, []);
Set
const filtered = Array.from(new Set(array.map(a => a.id))) .map(id => { return array.find(a => a.id === id) })

Rendered benchmark preparation results:

Suite status: <idle, ready to run>

Previous results

Test case name	Result
Set + filter
Reduce
Set

Fastest: N/A

Slowest: N/A

Latest run results:

Run details: (Test run date: one year ago)

User agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36

Browser/OS: Chrome 114 on Linux

View result in a separate tab

Test name	Executions per second
Set + filter	527.2 Ops/sec
Reduce	528.9 Ops/sec
Set	493.0 Ops/sec

Autogenerated LLM Summary (model llama3.2:3b, generated one month ago):

LLMs can make mistakes. Check important info.

Let's dive into the world of JavaScript microbenchmarks on MeasureThat.net.

The provided benchmark definition json represents a test case that measures the performance of different approaches to remove duplicates from an array of objects.

**Benchmark Definition**

The script preparation code generates an array of 100,000 objects with random `id` and `name` properties. The purpose is to simulate a scenario where you need to filter out duplicate records.

**Options compared**

There are three options being compared:

1. **Set + Filter**: This approach uses the `Array.prototype.filter()` method in combination with a `Set` data structure to remove duplicates. The idea is to add each element's `id` to the set and then use the filter method to exclude elements that already exist in the set.
2. **Reduce**: This approach uses the `Array.prototype.reduce()` method to build a new array with only unique elements. It iterates through the original array, finds an existing element by its `id`, and either adds it to the accumulator or skips it.
3. **Set**: This approach uses the `Set` data structure to remove duplicates in one step. It maps each element's `id` to a unique value and then uses `Array.from()` to convert the set back into an array.

**Pros and Cons**

*   **Set + Filter**:
    *   Pros: Simple, readable code, and easy to implement.
    *   Cons: May not be as efficient as other approaches due to the overhead of the filter method.
*   **Reduce**:
    *   Pros: Efficient use of the reduce method, which can be optimized for performance.
    *   Cons: Code may look more complex or harder to understand for beginners.
*   **Set**:
    *   Pros: Simple and efficient way to remove duplicates using a set data structure.
    *   Cons: May require additional steps to convert the set back into an array.

**Library**

None of the provided options use any external libraries. However, some browsers may provide built-in functions or methods that can be used in these approaches (e.g., `Set` objects are supported by most modern browsers).

**Special JS feature/syntax**

None of the options explicitly use special JavaScript features or syntax.

**Other alternatives**

There might be other approaches to remove duplicates from an array, such as using `Array.prototype.sort()` and then filtering out consecutive elements. However, these alternatives would likely have different trade-offs in terms of performance, readability, and complexity.

Here is a simple implementation for each approach to better illustrate the differences:

*   **Set + Filter**:
    ```javascript
const seen = new Set();
const filtered = array.filter(el => {
  const duplicate = seen.has(el.id);
  seen.add(el.id);
  return !duplicate;
});
```
*   **Reduce**:
    ```javascript
const filtered = array.reduce((acc, current) => {
  const x = acc.find(item => item.id === current.id);
  if (!x) {
    return acc.concat([current]);
  } else {
    return acc;
  }
}, []);
```
*   **Set**:
    ```javascript
const filtered = Array.from(new Set(array.map(a => a.id)))
.map(id => array.find(a => a.id === id));
```

Related benchmarks:

Filtering duplicates (version: 0)

Comparing performance of: Set + filter vs Reduce vs Set

Created: one year ago by: Guest

Jump to the latest result

Set + filter

Reduce

Set

Suite status: <idle, ready to run>

Experimental features:

Fastest: N/A

Slowest: N/A

Autogenerated LLM Summary (model llama3.2:3b, generated one month ago):