Dfyjkt

For example:

df1 = pd.DataFrame(np.repeat(np.arange(1,7),3), columns=['A'])

df1.A.value_counts(sort=False)
1 3
2 3
3 3
4 3
5 3
6 3
Name: A, dtype: int64

df2 = pd.DataFrame(np.repeat(np.arange(1,7),100), columns=['A'])

df2.A.value_counts(sort=False)
1 100
2 100
3 100
4 100
5 100
6 100
Name: A, dtype: int64

In the above examples the value_counts works perfectly and give the required result. whereas when coming to larger dataframes it is giving a different output. Here the A values are already sorted and counts are also same, but the order of index that is A changed after value_counts. Why is it doing correctly for small counts but not for large counts:

df3 = pd.DataFrame(np.repeat(np.arange(1,7),1000), columns=['A'])

df3.A.value_counts(sort=False)
4 1000
1 1000
5 1000
2 1000
6 1000
3 1000
Name: A, dtype: int64

Here I can do df3.A.value_counts(sort=False).sort_index() or df3.A.value_counts(sort=False).reindex(df.A.unique()). I want to know the reason why it is behaving differently for different counts?

Using:

Numpy version :1.15.2
Pandas version :0.23.4

This is actually a known problem.

If you browse through the source code -

1. 
   C:ProgramDataAnaconda3Libsite-packagespandascorealgorithims.py line 581 is the original implementation


2. It calls _value_counts_arraylike for int64 values when bins=None
   


3. This function makes a call - keys, counts = htable.value_count_int64(values, dropna)
   

If you then look at the htable implementation you will conclude that the keys are in an arbitrary order, subject to how the hashtable works.

Its not a guarantee of ANY kind of ordering. Typically this routine sorts by biggest values, and that is almost always what you want.

I guess they can change this to have sort=False mean original ordering. I don't know if this would actually break anything (and done internally this isn't very costly as the uniques are already known).

The order is changed from pandas/hashtable.pyx.build_count_table_object(). Resizing of the pymap moves the entries by hashing values.

Here is the full discussion