/Python/Using pandas stack and subset to return a dataframe object of highly correated pairs

Using pandas stack and subset to return a dataframe object of highly correated pairs

# save absolute value of correlation matrix as a data frame
# converts all values to absolute value
# stacks the row:column pairs into a multindex
# reset the index to set the multindex to seperate columns
# sort values. 0 is the column automatically generated by the stacking

df= old_df.corr().abs().stack().reset_index().sort_values(0, ascending=False)

# zip the variable name columns (Which were only named level_0 and level_1 by default) 
# in a new column named "pairs".
df['pairs'] = list(zip(df.level_0, df.level_1))

# set index to pairs
df.set_index(['pairs'], inplace = True)

#d rop level columns
df.drop(columns=['level_1', 'level_0'], inplace = True)

# rename correlation column as cc rather than 0
df.columns = ['cc']

# drop duplicates. This could be dangerous if you have variables perfectly correlated with variables other than themselves.
# for the sake of exercise, kept it in.
df.drop_duplicates(inplace=True)

# subset to include desired range
df[(df.cc>.75) & (df.cc <1)]

Similar codes

2 votes274 views1 answer
Tf.reduce_sum()
Python
-2 votes269 views1 answer
Jwt authentication python flask
Python
0 vote277 views1 answer
`distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
Python
0 vote261 views2 answers
Problem 1 dot product python
Python
0 vote292 views1 answer
How to work with django ornm __in
Python Django
1 vote282 views1 answer
Python print unicode character
Python
0 vote251 views2 answers
Python thousands separators
Python
0 vote245 views1 answer
Create new column pandas and order sequence
Python
0 vote264 views1 answer
Multiple args for pandas apply
Python
3 votes223 views1 answer
Pylint no name in module cv2
Python