Counting distinct values, comparing two arrays, and rebuilding the original from what unique returned.
Overview
unique
Finding distinct values is one of the first things you do with categorical or integer data. How many different ids are there? What are the distinct labels? Are there values in one array that are missing from another? NumPy's set operations answer these questions on arrays directly, without converting to Python sets.
np.unique returns the distinct values from an array, sorted. The sorting is not optional. That matters because it means the output order does not match the input order. If you need the values in the order they first appeared, you can use the return_index parameter and sort by those indices, but the default behaviour is sorted output.
Two things about unique catch people. First, it flattens multi-dimensional arrays before finding unique values. A 2-D array of shape (3, 4) gives a 1-D array of unique values, not unique rows. Finding unique rows requires axis=0, which was added later and is less well known. Second, floating-point values that differ by tiny rounding errors are treated as distinct. Rounding before calling unique avoids spurious duplicates.
The return parameters give you extra information without a second pass. return_counts gives the frequency of each unique value. return_index gives the position of the first occurrence. return_inverse gives an array that reconstructs the original from the unique values. These are cheaper than calling unique once and then searching for each value separately.
The set operations -- intersect1d, union1d, setdiff1d, and setxor1d -- work on sorted unique arrays. They answer questions like "which values are in both arrays", "which values are in the first but not the second", and "which values are in exactly one of the two". Each returns a sorted 1-D array.
in1d and its modern replacement isin test membership. Given two arrays, isin returns a boolean array showing which elements of the first are present in the second. That boolean array can be used directly as a mask for filtering.
This module covers unique with its return parameters, the set operations, and isin. The examples show common use cases: deduplication, frequency counting, filtering by membership, and comparing two datasets for overlap.
np.unique(a) returns the distinct values, sorted, as a 1-D array.
Two things about that are worth stating by name, because both catch people.
It's always sorted. If you wanted first-appearance order, unique alone doesn't give it. You need return_index and then a sort of those indices.
It always flattens, unless you pass axis. A 2-D input gives back a flat list of each distinct value in the whole array.
Worth knowing
np.unique always returns sorted distinct values, and flattens the input unless you pass axis.
return_counts=True gives a frequency table in one call — values and counts, aligned.
return_inverse maps every element to its position in the unique array, so vals[inv] rebuilds the original. That is label encoding.
axis=0 makes unique operate on whole rows rather than individual values.
intersect1d, union1d, setdiff1d and setxor1d all de-duplicate and sort.
np.isin returns a mask of the original shape, which is what you need for filtering rather than reducing.
Unique Values and Set Operations
Counting distinct values, checking arrays, and rebuilding the first.
unique returns sorted distinct values
Always sorted, always 1-D unless you ask otherwise.
example_01.pyNumPy
import numpy as np
a = np.array([3, 1, 4, 1, 5, 9, 2, 6, 5, 3])
print("input :", a)
print("unique:", np.unique(a), "<- sorted, duplicates gone")
print()
print("it flattens a 2-D input by default:")
b = np.array([[1, 2], [2, 3]])
print(" ", np.unique(b))
print()
print("strings work too, sorted alphabetically:")
print(" ", np.unique(np.array(["pear", "fig", "pear", "apple"])))
Output
return_counts is the frequency table
One call gives values and how often each appeared.
example_02.pyNumPy
import numpy as np
votes = np.array(["red", "blue", "red", "green", "red", "blue"])
vals, counts = np.unique(votes, return_counts=True)
for v, c in zip(vals.tolist(), counts.tolist()):
print(" %-6s %d" % (v, c))
print()
print("the winner, without a loop:")
print(" ", vals[counts.argmax()], "with", int(counts.max()))
print()
print("counts sum back to the input length:", int(counts.sum()), "==", len(votes))
Output
return_index and return_inverse
Where each unique value first appeared, and how to rebuild the original.
example_03.pyNumPy
import numpy as np
a = np.array([30, 10, 20, 10, 30])
vals, first, inv = np.unique(a, return_index=True, return_inverse=True)
print("values :", vals)
print("first :", first, "<- index in a where each value first occurs")
print("inverse:", inv, "<- which unique value each element of a is")
print()
print("inverse rebuilds the original exactly:")
print(" ", vals[inv], " matches:", np.array_equal(vals[inv], a))
print()
print("This is label encoding in one line - categories to integers.")
Output
Unique rows, not unique values
axis=0 treats each row as a single item.
example_04.pyNumPy
import numpy as np
rows = np.array([[1, 2],
[3, 4],
[1, 2],
[5, 6]])
print("input:"); print(rows)
print()
print("np.unique(rows) ->", np.unique(rows), "<- all values, flattened")
print()
print("np.unique(rows, axis=0) -> duplicate ROWS removed:")
print(np.unique(rows, axis=0))
print()
uniq, cnt = np.unique(rows, axis=0, return_counts=True)
print("with counts:")
for r, c in zip(uniq.tolist(), cnt.tolist()):
print(" %s seen %d time(s)" % (r, c))
Output
The four set operations
Intersection, union, difference and symmetric difference - all returning sorted unique results.
example_05.pyNumPy
import numpy as np
a = np.array([1, 2, 3, 4, 4])
b = np.array([3, 4, 5])
print("a =", a, " b =", b)
print()
print("intersect1d :", np.intersect1d(a, b), "in both")
print("union1d :", np.union1d(a, b), "in either")
print("setdiff1d :", np.setdiff1d(a, b), "in a only")
print("setxor1d :", np.setxor1d(a, b), "in exactly one")
print()
print("All of them de-duplicate: the repeated 4 in a appears once.")
Output
isin keeps the shape - and that is the difference
Set operations reduce; isin gives a mask you can index with.
example_06.pyNumPy
import numpy as np
ids = np.array([[101, 102], [103, 104]])
wanted = np.array([102, 104, 999])
mask = np.isin(ids, wanted)
print("ids:"); print(ids)
print("mask (same shape):"); print(mask)
print()
print("select with it:", ids[mask])
print("invert it :", ids[~mask])
print()
print("intersect1d would have given a flat sorted list instead:")
print(" ", np.intersect1d(ids, wanted))
print()
print("isin is the one to use when you need to filter rows by a key.")
Output
The optional returns
Three flags turn unique from a de-duplicator into a thing a lot more handy.
**return_counts** gives how many times each value occurred, aligned with the values. This is a rate table. Mixed with argmax it gives the mode in two lines with no loop.
**return_index** gives, for each unique value, the index in the input where it first appeared. handy when you want the *first* record for each key rather than just the key.
**return_inverse** gives, for each element of the input. This Unique value it matches to. vals[inv] rebuilds the input just.
That last one is worth dwelling on. It's label encoding. An array of groups turns an array of small ints plus a lookup table. Machine learning pipelines do this constantly. np.unique(..., return_inverse=True) is the whole setup.
The returns come back in a fixed order. Values, index, inverse, counts. Even so of which flags you set. Unpack carefully when you enable more than one.
Unique rows
np.unique(rows, axis=0) treats each row as an item and drops copy rows.
This is what you want for de-duplicating records. It composes with return_counts to find how many times each distinct row appeared. A group-by, for the case where the whole row is the key.
Rows are checked elementwise, and the result is sorted lexicographically.
Set operations
Four functions, all following the same rules: they treat the inputs as sets, de-copy, and return sorted results.
intersect1d — in both. union1d — in either. setdiff1d — in the first only. setxor1d — in just one.
The 1d in the names is a warning. They flatten their inputs. If you pass 2-D arrays you get a flat answer.
intersect1d accepts assume_unique=True, which skips the inner de-duplication and is usefully faster on large arrays you already know are unique.
isin is the different one
The set functions reduce. They answer "what is in both?" and hand you a smaller array, with the first spots gone.
np.isin(a, b)preserves shape. It returns a boolean mask the same shape as a, saying whether each element appears in b.
That gap decides which one you need. If the question is "which values do these two arrays share?", use intersect1d. If the question is "which rows of my table have an id in this list?", you need the mask. You're going to index with it. And intersect1d has thrown away just the data you need.
~mask inverts it, which is how you exclude a list rather than include one.
Performance
These are all sort-based, so roughly O(n log n). That is fast. For repeated membership tests against the same fixed set, a Python set or a sorted array with searchsorted can beat rebuilding the compare each time.
For counting, np.bincount is faster than unique(return_counts=True) when the values are small non-minus ints. It can index right instead of sorting.
The mistakes
Expecting first-appearance order.unique sorts.
Passing 2-D to a set function and being surprised by a flat result. They all flatten.
**Reaching for intersect1d when filtering.** It discards spots. isin keeps them.
Unpacking the returns in the wrong order. Values, index, inverse, counts. Always in that order, whichever subset you asked for.
bincount, the fast path for integers
When the values are small non-minus ints, np.bincount counts them far faster than unique(return_counts=True). It indexes an output array right instead of sorting.
np.bincount(a) returns an array of length a.max() + 1. Here Spot i holds the count of value i. Values that never appear get zero. This Is either convenient or a nuisance based on whether the value space is dense.
minlength forces a min output size. This Matters when you're counting class labels and a class might be absent from a given batch. Without it, the output length varies with the data and downstream code breaks.
The weights arg turns it into a group-sum. np.bincount(group_ids, weights=values) sums the values belonging to each group in a single pass. This Is the fastest group-by NumPy offers and a truly handy thing to know.
The constraint is that the values must be non-minus ints. Memory is scaled to the largest one. Counting values around a million allocates a million-element array even so of how few distinct values there are.
unique maps any keys — strings included — to dense ints. bincount aggregates over those ints. Four lines, no loop. It handles any number of groups.
For aggregations that aren't sums, sorting by the key and using np.split at the boundaries where the key changes gives a list of per-group arrays:
order = np.argsort(keys)
sorted_keys = keys[order]
cuts = np.flatnonzero(sorted_keys[1:] != sorted_keys[:-1]) + 1
groups = np.split(values[order], cuts)
That is more machinery than pandas' groupby. It's the right amount when adding pandas to a project for one aggregation would be the larger cost.
histogram, for continuous values
unique is for discrete values. Its smooth counterpart is np.histogram. This Counts how many values fall into each of a set of bins.
It returns counts and edges. The edges array is one longer than the counts. N bins have n+1 boundaries. A small detail that causes a lot of off-by-one confusion when plotting.
bins accepts a count, an clear array of edges, or one of a few auto rules like "auto" and "fd" that choose a bin width from the data.
np.histogram2d and np.histogramdd handle two and more dims.
The bins are half-open — [a, b) — except the last. This Includes its right edge so that the max value is counted one spot. That skew is deliberate and at times surprising.
assume_unique, and when to use it
intersect1d, setdiff1d and isin accept assume_unique=True.
Setting it skips the inner de-duplication. This Is a real saving on large arrays. Setting it wrongly gives wrong answers, not an error. Copies are handled as though they weren't there. Results can come out with surprise repeats.
Use it only when uniqueness is sure structurally. The array came from unique, or it's a main key, or you've just checked. "It should be unique" isn't the same thing.
isin also has a kind arg that chooses between a sort-based and a table-based setup. The table-based version is much faster for ints over a small range and uses memory scaled to that range. The auto choice is often right and worth replacing only after measuring.
Ordering, and how to get first-appearance order
each one of these functions returns sorted output. At times sorted isn't what you want.
For first-appearance order, use return_index and sort the indices:
The indices say where each unique value first appeared. Sorting by them restores the first meet order.
This comes up more often than it looks. Group codes that should follow the data's order, deduplication that should keep the first case, log analysis where the sequence carries meaning.
Summary of the choices
Distinct values, any dtype: np.unique.
Counts of small non-negative integers: np.bincount, and use minlength.
Counts of anything else: np.unique(return_counts=True).
Group aggregation: unique(return_inverse=True) feeding bincount(weights=...).
Continuous data: np.histogram.
Comparing two arrays as sets: the 1d family, remembering that they flatten.
Filtering by membership: np.isin, because it keeps the shape and the others do not.
Uniqueness on floats
Applying unique to floating-point data often disappoints. Values that are conceptually equal differ in the last bits.
np.unique([0.1 + 0.2, 0.3]) returns two values, not one.
There's no handling arg, and adding one would be ill-defined. "close enough" isn't transitive. A tolerant unique has no unambiguous answer.
The hands-on approaches are to round first, np.unique(np.round(a, 6)). This Is exact and cheap but sensitive to values sitting near a rounding boundary. Or to sort and group with an clear handling if the semantics matter.
For anything more careful, this is a clustering problem rather than a uniqueness problem. Treating it as one avoids pretending the answer is exact.
Set operations on rows
The 1d family flattens, so checking two tables row-wise needs a other approach.
np.unique(rows, axis=0) handles de-duplication. For intersection and gap between two sets of rows, the usual trick is to view each row as a single opaque item. Either through a structured dtype, or by converting rows to tuples and using Python sets when the arrays are small enough.
For larger data this is the point where pandas' merge is the right tool. Reaching for it isn't a defeat. NumPy is on purpose a numeric array library rather than a relational one.
Performance notes
np.unique sorts, so it's O(n log n) and allocates.
np.bincount is O(n) for small non-minus ints. Allocates scaled to the largest value rather than the number of distinct ones. For values up to a few million that is a good trade. For sparse large values it isn't.
np.isin builds a sorted structure or a lookup table based on kind. It isn't free either. But it's far better than a loop of checks. Better than again and again calling intersect1d.
For repeated membership tests against a fixed small set, a Python set can beat all of them. The per-call cost of NumPy dominates when the query is a single value.
The summary
np.unique — sorted distinct values. return_counts for rates, return_inverse for label encoding, return_index for first case, axis=0 for rows.
np.bincount — fast counting for small non-minus ints, with minlength to fix the output size and weights to turn it into a group-sum.
np.histogram — the smooth analogue, returning counts and one more edge than counts.
intersect1d / union1d / setdiff1d / setxor1d. Set checks, all sorting, de-duplicating and flattening.
np.isin — membership as a mask of the first shape. This Is the one to use for filtering.
The recurring choice is between the functions that reduce and the one that preserves shape. If the answer is going to index a thing, you want the mask.
A closing note on Unique Values and Set Operations
The functions in this module are simple on their own and powerful in mix. unique with return_inverse feeding bincount with weights is four lines that replace a loop, a dict and a fair amount of care about missing keys.
The one gap worth carrying away is between reducing and preserving shape. Almost each mistake in this area is reaching for a set function. Which discards spots. When the question was really about filtering. This Needs them. When the answer is going to index a thing, isin is the function you want.
Questions people ask about Unique Values and Set Operations
What does np.unique return? The sorted distinct values. Add return_counts=True for counts and return_inverse=True for the mapping back to the original.
How do I count occurrences of each value? Use np.unique(a, return_counts=True). For small non-negative integers np.bincount is faster.
How do I find common elements of two arrays? Use np.intersect1d. There are also union1d, setdiff1d and setxor1d for the other set operations.
What is the quick way to test membership? np.isin(a, values) returns a boolean array the shape of a. It is the vectorised version of the in operator.
Does np.unique keep the original order? No, it sorts. To keep the first-seen order, use return_index=True and sort those indices.
Can I find unique rows? Yes, np.unique(a, axis=0) treats each row as one item.
Where this fits
Counting distinct values is often the first thing anyone does with a new dataset. The last thing anyone remembers to do before trusting a result.
np.unique(col, return_counts=True) on each cat column at load time takes seconds and routinely surfaces the problems that would else appear much later. A group with a trailing space, an identifier that isn't unique, a class that appears twice in the training set and never in the test set.
Check yourself
0 of 4
Answer without scrolling back up.
What order does `np.unique` return values in?
Always sorted, and it flattens the input unless you pass axis. For first-appearance order you need return_index and a sort of those indices.
What does `return_inverse` give you?
That is label encoding in one call: categories become small integers plus a lookup table.
You have a table of ids and a list of wanted ids, and need the matching rows. Which function?
isin returns a mask of the original shape, which you can index with. intersect1d reduces and throws away the positions you need.
What does `np.unique(rows, axis=0)` do?
Combined with return_counts it is a group-by for the case where the whole row is the key.
Cheat sheet
Unique Values and Set Operations
Find unique values, counts and set intersections in a NumPy array. Rebuild the original from the unique values and an inverse index.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.