/Python/Using pandas stack and subset to return a dataframe object of highly correated pairs

Using pandas stack and subset to return a dataframe object of highly correated pairs

# save absolute value of correlation matrix as a data frame
# converts all values to absolute value
# stacks the row:column pairs into a multindex
# reset the index to set the multindex to seperate columns
# sort values. 0 is the column automatically generated by the stacking

df= old_df.corr().abs().stack().reset_index().sort_values(0, ascending=False)

# zip the variable name columns (Which were only named level_0 and level_1 by default) 
# in a new column named "pairs".
df['pairs'] = list(zip(df.level_0, df.level_1))

# set index to pairs
df.set_index(['pairs'], inplace = True)

#d rop level columns
df.drop(columns=['level_1', 'level_0'], inplace = True)

# rename correlation column as cc rather than 0
df.columns = ['cc']

# drop duplicates. This could be dangerous if you have variables perfectly correlated with variables other than themselves.
# for the sake of exercise, kept it in.
df.drop_duplicates(inplace=True)

# subset to include desired range
df[(df.cc>.75) & (df.cc <1)]

Similar codes

0 vote292 views1 answer
Django model meta ordering multiple ordering
Python
2 votes254 views2 answers
How to check if user is using main file or importing the file and using in python
Python
89 votes245 views6 answers
How to use for in python
Python
-4 votes312 views1 answer
Hello world python 2
Python
0 vote242 views1 answer
Get number of zero is a row pandas
Python
0 vote262 views1 answer
"status": 400, "message": "The string was not recognized as a valid DateTime
Python
-1 vote246 views1 answer
Python unicode point to utf8 string
Python
13 votes394 views1 answer
Flask app starter
Python Flask
4 votes265 views1 answer
How to delete every row in excel using openpyxl
Python
56 votes313 views2 answers
Convert date string to date time string python
Python