Avoiding the for loop using dataframes in python
I have two dataframes in Python named GroupedCode (70000 rows and 3 columns) and ICD9 (11500 rows and 27 columns). My goal is to find every element in Code9 column of GroupedCode that is present in the ICD9CMCode column of ICD9 and every time that I find a match, append the value of the TotalDiag column of the ICD9 into a list called Freq.
I came up with a For loop to do this but it takes a good amount of time to complete. I was wondering if there is a better way to speed up the for loop or even better avoid it.
Here is my for loop:
Freq =
for code in GroupedCode.Code9:
if (len(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)]) == 0):
Freq.append(0)
else:
Freq.append(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)].values)
python-3.x pandas dataframe
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
add a comment |
I have two dataframes in Python named GroupedCode (70000 rows and 3 columns) and ICD9 (11500 rows and 27 columns). My goal is to find every element in Code9 column of GroupedCode that is present in the ICD9CMCode column of ICD9 and every time that I find a match, append the value of the TotalDiag column of the ICD9 into a list called Freq.
I came up with a For loop to do this but it takes a good amount of time to complete. I was wondering if there is a better way to speed up the for loop or even better avoid it.
Here is my for loop:
Freq =
for code in GroupedCode.Code9:
if (len(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)]) == 0):
Freq.append(0)
else:
Freq.append(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)].values)
python-3.x pandas dataframe
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
add a comment |
I have two dataframes in Python named GroupedCode (70000 rows and 3 columns) and ICD9 (11500 rows and 27 columns). My goal is to find every element in Code9 column of GroupedCode that is present in the ICD9CMCode column of ICD9 and every time that I find a match, append the value of the TotalDiag column of the ICD9 into a list called Freq.
I came up with a For loop to do this but it takes a good amount of time to complete. I was wondering if there is a better way to speed up the for loop or even better avoid it.
Here is my for loop:
Freq =
for code in GroupedCode.Code9:
if (len(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)]) == 0):
Freq.append(0)
else:
Freq.append(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)].values)
python-3.x pandas dataframe
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
I have two dataframes in Python named GroupedCode (70000 rows and 3 columns) and ICD9 (11500 rows and 27 columns). My goal is to find every element in Code9 column of GroupedCode that is present in the ICD9CMCode column of ICD9 and every time that I find a match, append the value of the TotalDiag column of the ICD9 into a list called Freq.
I came up with a For loop to do this but it takes a good amount of time to complete. I was wondering if there is a better way to speed up the for loop or even better avoid it.
Here is my for loop:
Freq =
for code in GroupedCode.Code9:
if (len(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)]) == 0):
Freq.append(0)
else:
Freq.append(ICD9.TotalDiag[ICD9['ICD9CMCode'].str.match(code)].values)
python-3.x pandas dataframe
python-3.x pandas dataframe
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
edited Nov 11 '18 at 17:27
Aqueous Carlos
293213
293213
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
asked Nov 10 '18 at 23:14
Mahmoud ZeydabadinezhadMahmoud Zeydabadinezhad
1
1
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Consider merging the two data frames to retain matches between each other then downcasting pandas Series to a list. Currently, you are storing numpy arrays (not single values) or 0 to a list.
merged_df = pd.merge(GroupedCode, ICD9, left_on='ICD9CMCode', right_on='Code9')
Freq = merged_df['TotalDiag'].tolist()
Even consider unique() for unique values in case of multiple inner join matches.
Freq = merged_df['TotalDiag'].unique().tolist()
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53244347%2favoiding-the-for-loop-using-dataframes-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Consider merging the two data frames to retain matches between each other then downcasting pandas Series to a list. Currently, you are storing numpy arrays (not single values) or 0 to a list.
merged_df = pd.merge(GroupedCode, ICD9, left_on='ICD9CMCode', right_on='Code9')
Freq = merged_df['TotalDiag'].tolist()
Even consider unique() for unique values in case of multiple inner join matches.
Freq = merged_df['TotalDiag'].unique().tolist()
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
add a comment |
Consider merging the two data frames to retain matches between each other then downcasting pandas Series to a list. Currently, you are storing numpy arrays (not single values) or 0 to a list.
merged_df = pd.merge(GroupedCode, ICD9, left_on='ICD9CMCode', right_on='Code9')
Freq = merged_df['TotalDiag'].tolist()
Even consider unique() for unique values in case of multiple inner join matches.
Freq = merged_df['TotalDiag'].unique().tolist()
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
add a comment |
Consider merging the two data frames to retain matches between each other then downcasting pandas Series to a list. Currently, you are storing numpy arrays (not single values) or 0 to a list.
merged_df = pd.merge(GroupedCode, ICD9, left_on='ICD9CMCode', right_on='Code9')
Freq = merged_df['TotalDiag'].tolist()
Even consider unique() for unique values in case of multiple inner join matches.
Freq = merged_df['TotalDiag'].unique().tolist()
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
Consider merging the two data frames to retain matches between each other then downcasting pandas Series to a list. Currently, you are storing numpy arrays (not single values) or 0 to a list.
merged_df = pd.merge(GroupedCode, ICD9, left_on='ICD9CMCode', right_on='Code9')
Freq = merged_df['TotalDiag'].tolist()
Even consider unique() for unique values in case of multiple inner join matches.
Freq = merged_df['TotalDiag'].unique().tolist()
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
answered Nov 10 '18 at 23:45
ParfaitParfait
50.3k84269
50.3k84269
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53244347%2favoiding-the-for-loop-using-dataframes-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
