When should you use a set instead of a list?

A set buys O(1) membership and pays for it by losing order and duplicates, and by requiring hashable elements. When you need the speed and the order, dict.fromkeys(seq) deduplicates in one pass and keeps insertion order.

Dicts, sets & hashingConceptualEasy

Step through it

What to watch

  • The set is smaller: duplicates are gone, and so is the order.
  • dict.fromkeys keeps both the order and the O(1) lookup.
  • All three hold the same distinct values.

Say this out loud

"Sets are for membership and uniqueness - O(1) instead of O(n). They lose order and need hashable elements. If I need dedup with order I use dict.fromkeys, since dicts have kept insertion order since 3.7."

Overview

A set stores unique hashable values and supports expected constant-time membership, but does not preserve duplicates. Use a list for order and repeats; use dict.fromkeys when deduplicating must keep insertion order.

In this guide
  1. Run it
  2. Why the set is fast
  3. The requirement: hashable elements
  4. What you gain and what you give up
  5. Deduplicating three ways
  6. The operators that replace loops
  7. Choosing between the containers
  8. Where deduplication goes wrong
  9. Sources and next steps

Run it

The three deduplication idioms and what each does to order, the set operators against their loop equivalents. And the membership timing that motivates all of it.

1Python
Output

Building on the same setup, the next block shows the remaining cases.

2Python
Output

Why the set is fast

A list must scan to know whether it already contains something — O(n) per check, so O(n²) to deduplicate by repeated scanning.

A set hashes the value and looks in one place — O(1) on average, so O(n) overall.

3Python
Output

Both versions preserve order. The first is quadratic and is what people write before they think about it, and on 100,000 items it takes minutes rather than milliseconds.

Keeping the list and a set alongside it is the standard pattern: the list holds the ordered output, the set answers the membership question.

The requirement: hashable elements

Sets and dictionaries need hashable elements, so deduplication by value fails on mutable ones:

4Python
Output

For unhashable items, convert to a hashable canonical form:

Item typeCanonical key
listtuple(x)
setfrozenset(x)
dicttuple(sorted(x.items()))
Nested structurejson.dumps(x, sort_keys=True)
Custom objectA tuple of its identifying fields

The JSON option works and is slow, and it is sensitive to types — 1 and 1.0 serialise differently. Prefer an explicit tuple of the fields that define identity.

What you gain and what you give up

Gain: membership in O(1) rather than O(n), and uniqueness enforced for free.

Give up: order, duplicates, indexing (s[0] is a TypeError), and the ability to hold unhashable elements. You cannot put a list in a set, though you can put a tuple.

Converting costs O(n), so a single membership test on a small list is not worth it. More than a couple of tests, or a large collection, and it is.

Deduplicating three ways

set(seq) — fastest, order destroyed.

sorted(set(seq)) — deduplicated and in sorted order, which is not the same as original order and is often what people accidentally ship.

list(dict.fromkeys(seq)) — deduplicated in first-seen order, one pass, and the one to remember. Dicts have preserved insertion order since 3.7, so this is a guarantee rather than a trick.

The operators that replace loops

a & b intersection, a | b union, a - b difference, a ^ b symmetric difference, a <= b subset. Each replaces a loop with a membership test inside it — which is to say, each replaces an accidental O(n·m) with an O(n + m).

"Which users are in A but not B" is a - b. Writing that as a comprehension over a list is the most common form of the quadratic trap.

Choosing between the containers

 listsetdict
OrderedYesNoYes (insertion)
DuplicatesYesNoKeys unique
inO(n)O(1)O(1) on keys
IndexingYesNoBy key
Elements must be hashableNoYesKeys only
MemoryLowestHigherHighest

The decision is usually straightforward:

Need order and duplicates → list. Need membership testing only → set. Need associated data → dict. Need order without duplicates → dict.fromkeys, or a list plus a set.

And the sets themselves support the operations that make them worth reaching for beyond deduplication:

5Python
Output

Those replace loops. "Which users are in both groups?" is a & b, not a nested loop. And it runs in C at O(min(len(a), len(b))).

Where deduplication goes wrong

Assuming a set is sorted. set([3, 1, 2]) may print as {1, 2, 3} for small integers because of how they hash. Add a string or a larger number and the illusion vanishes. Never rely on it.

Deduplicating floats. 0.1 + 0.2 and 0.3 are different values, so they do not deduplicate. Round to a fixed precision first if that is the intent.

1 == True and 1.0 == 1. These are equal and hash equally, so {1, True, 1.0} is {1}. Surprising and correct.

Case and whitespace. "Alice" and "alice " are distinct. Normalise before deduplicating — strip, casefold, and NFC-normalise Unicode.

Deduplicating objects by identity. Two distinct objects with the same field values are different set members unless __eq__ and __hash__ are defined.

Memory on large inputs. Deduplicating a hundred million items builds a set of them all. For a stream where approximate counting suffices, a Bloom filter or HyperLogLog uses a tiny fraction of the memory.

For the underlying ideas, see sets, hash tables and lists.

Sources and next steps

How the code works

The three deduplication idioms and what each does to order, the set operators against their loop equivalents, and the membership timing that motivates all of it.

How the code works

  1. list(dict.fromkeys(items))Deduplicates in first-seen order, in one pass. The idiom worth memorising, and a language guarantee rather than an implementation detail since 3.7.
  2. sorted(set(items))Deduplicated and reordered. It looks like a tidy version of the previous line and quietly changes the output order, which is a real bug when the order carried meaning.
  3. {[1, 2]}A TypeError: set elements must be hashable for the same reason dictionary keys must be. A tuple works.
  4. set(big_a) & set(big_b)Two O(n) conversions and an O(min) intersection, against a comprehension that scans one list for every element of the other. The timing at the end is that difference.

Change one thing

  • Deduplicate a list of dictionaries. It raises — and the usual fix is to key on something hashable, such as an id or a tuple of fields.
  • Compare a.isdisjoint(b) with not (a & b). The first can stop at the first shared element instead of building the whole intersection.

Where this runs

Real CPython, compiled to WebAssembly and running on your own machine — nothing is uploaded. The first run takes a few seconds while the interpreter downloads; after that it is immediate. Need more room, or want to paste your own attempt? Use the Python compiler.

Check yourself

0 of 3

Answer without scrolling back up.

  1. Which deduplicates a list while preserving the original order?

  2. What can a list hold that a set cannot?

  3. 'Which items are in A but not in B' is best written as:

Cheat sheet

When should you use a set instead of a list?

Switch from a list to a set and watch membership checks drop from O(n) to O(1). Deduplicate a sequence while preserving order with dict.fromkeys.

INTERVIEW · vizlearn.in/interview/sets-versus-lists-and-deduplication.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

Teaching a course with this? Individual teachers can use it freely; for a whole department or programme, see institutional licences.