Disable the on-demand ngen.exe on lightweight windows servers (“.NET Framework NGEN v4.0.30319 64”)
We're deploying .Net4-based asp.net apis to a large number of windows servers, and in order to keep aws costs low, we run them as small as possible.
The problem we're seeing is that after the vm is provisioned and code is deployed, we're seeing high cpu usage for a few minutes. Most of this cpu usage comes from the process "ngentask.exe". It's not related to app startup, we're typically seeing this happening a few minutes after the deploy continues and other cpu-heavy processes calm down.
It's also worth mentioning that as part of our deploy process, we "touch" the app by invoking a web request to a predefined "health" endpoint in the app, which is simply an empty MVC controller in most of our apps - we do this both to verify that the app is actually running, and to make sure is "warmed up" and ready to serve traffic.
Looking at the ngen.log, the timestamps match perfectly with cpu usage:
11/13/2018 14:20:13.275 [732]: Task scheduler requested starting 32-bit task
11/13/2018 14:20:13.371 [732]: Task scheduler requested starting 64-bit task
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFrameworkv4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:380
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.512 [3104]: NGen Task starting, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.560 [3104]: Attempting to acquire task lock.
Like I wrote, this can happen as much as 10 minutes after deploy, and by that time the app is already serving requests. This leads me to believe that these tasks are actually not needed - if they were, the app wouldn't function in the 10 minutes between deploy and the completion of these processes.
So my question is: Am I safe to disable this task? These are Windows 2016 Core servers only running a single asp.net web app, so there's no "random" gui apps getting installed or anything.
c# asp.net .net windows-server-2016 ngen
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
add a comment |
We're deploying .Net4-based asp.net apis to a large number of windows servers, and in order to keep aws costs low, we run them as small as possible.
The problem we're seeing is that after the vm is provisioned and code is deployed, we're seeing high cpu usage for a few minutes. Most of this cpu usage comes from the process "ngentask.exe". It's not related to app startup, we're typically seeing this happening a few minutes after the deploy continues and other cpu-heavy processes calm down.
It's also worth mentioning that as part of our deploy process, we "touch" the app by invoking a web request to a predefined "health" endpoint in the app, which is simply an empty MVC controller in most of our apps - we do this both to verify that the app is actually running, and to make sure is "warmed up" and ready to serve traffic.
Looking at the ngen.log, the timestamps match perfectly with cpu usage:
11/13/2018 14:20:13.275 [732]: Task scheduler requested starting 32-bit task
11/13/2018 14:20:13.371 [732]: Task scheduler requested starting 64-bit task
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFrameworkv4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:380
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.512 [3104]: NGen Task starting, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.560 [3104]: Attempting to acquire task lock.
Like I wrote, this can happen as much as 10 minutes after deploy, and by that time the app is already serving requests. This leads me to believe that these tasks are actually not needed - if they were, the app wouldn't function in the 10 minutes between deploy and the completion of these processes.
So my question is: Am I safe to disable this task? These are Windows 2016 Core servers only running a single asp.net web app, so there's no "random" gui apps getting installed or anything.
c# asp.net .net windows-server-2016 ngen
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29
add a comment |
We're deploying .Net4-based asp.net apis to a large number of windows servers, and in order to keep aws costs low, we run them as small as possible.
The problem we're seeing is that after the vm is provisioned and code is deployed, we're seeing high cpu usage for a few minutes. Most of this cpu usage comes from the process "ngentask.exe". It's not related to app startup, we're typically seeing this happening a few minutes after the deploy continues and other cpu-heavy processes calm down.
It's also worth mentioning that as part of our deploy process, we "touch" the app by invoking a web request to a predefined "health" endpoint in the app, which is simply an empty MVC controller in most of our apps - we do this both to verify that the app is actually running, and to make sure is "warmed up" and ready to serve traffic.
Looking at the ngen.log, the timestamps match perfectly with cpu usage:
11/13/2018 14:20:13.275 [732]: Task scheduler requested starting 32-bit task
11/13/2018 14:20:13.371 [732]: Task scheduler requested starting 64-bit task
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFrameworkv4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:380
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.512 [3104]: NGen Task starting, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.560 [3104]: Attempting to acquire task lock.
Like I wrote, this can happen as much as 10 minutes after deploy, and by that time the app is already serving requests. This leads me to believe that these tasks are actually not needed - if they were, the app wouldn't function in the 10 minutes between deploy and the completion of these processes.
So my question is: Am I safe to disable this task? These are Windows 2016 Core servers only running a single asp.net web app, so there's no "random" gui apps getting installed or anything.
c# asp.net .net windows-server-2016 ngen
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
We're deploying .Net4-based asp.net apis to a large number of windows servers, and in order to keep aws costs low, we run them as small as possible.
The problem we're seeing is that after the vm is provisioned and code is deployed, we're seeing high cpu usage for a few minutes. Most of this cpu usage comes from the process "ngentask.exe". It's not related to app startup, we're typically seeing this happening a few minutes after the deploy continues and other cpu-heavy processes calm down.
It's also worth mentioning that as part of our deploy process, we "touch" the app by invoking a web request to a predefined "health" endpoint in the app, which is simply an empty MVC controller in most of our apps - we do this both to verify that the app is actually running, and to make sure is "warmed up" and ready to serve traffic.
Looking at the ngen.log, the timestamps match perfectly with cpu usage:
11/13/2018 14:20:13.275 [732]: Task scheduler requested starting 32-bit task
11/13/2018 14:20:13.371 [732]: Task scheduler requested starting 64-bit task
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFrameworkv4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:380
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.512 [3104]: NGen Task starting, command line: "C:WindowsMicrosoft.NETFramework64v4.0.30319NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.560 [3104]: Attempting to acquire task lock.
Like I wrote, this can happen as much as 10 minutes after deploy, and by that time the app is already serving requests. This leads me to believe that these tasks are actually not needed - if they were, the app wouldn't function in the 10 minutes between deploy and the completion of these processes.
So my question is: Am I safe to disable this task? These are Windows 2016 Core servers only running a single asp.net web app, so there's no "random" gui apps getting installed or anything.
c# asp.net .net windows-server-2016 ngen
c# asp.net .net windows-server-2016 ngen
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
asked Nov 13 '18 at 19:26
TrondhTrondh
2,22411418
2,22411418
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29
add a comment |
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29
add a comment |
1 Answer
1
active
oldest
votes
Instead of trying to disable ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).
Below one-liner finds the newest version of ngen.exe and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.
& ([String] $ngenExe = ((Get-ChildItem -Path C:WindowsMicrosoft.NETFramework64v*ngen.exe) | sort -Descending | Select -First 1)) update /force
See: https://docs.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53288184%2fdisable-the-on-demand-ngen-exe-on-lightweight-windows-servers-net-framework-n%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Instead of trying to disable ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).
Below one-liner finds the newest version of ngen.exe and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.
& ([String] $ngenExe = ((Get-ChildItem -Path C:WindowsMicrosoft.NETFramework64v*ngen.exe) | sort -Descending | Select -First 1)) update /force
See: https://docs.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
add a comment |
Instead of trying to disable ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).
Below one-liner finds the newest version of ngen.exe and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.
& ([String] $ngenExe = ((Get-ChildItem -Path C:WindowsMicrosoft.NETFramework64v*ngen.exe) | sort -Descending | Select -First 1)) update /force
See: https://docs.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
add a comment |
Instead of trying to disable ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).
Below one-liner finds the newest version of ngen.exe and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.
& ([String] $ngenExe = ((Get-ChildItem -Path C:WindowsMicrosoft.NETFramework64v*ngen.exe) | sort -Descending | Select -First 1)) update /force
See: https://docs.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
Instead of trying to disable ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).
Below one-liner finds the newest version of ngen.exe and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.
& ([String] $ngenExe = ((Get-ChildItem -Path C:WindowsMicrosoft.NETFramework64v*ngen.exe) | sort -Descending | Select -First 1)) update /force
See: https://docs.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
answered Nov 14 '18 at 19:24
MisterSmithMisterSmith
977611
977611
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
add a comment |
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
awesome, thanks. We do run ngen.exe when generating vm images, but doing it again during deployment (when the node is taken out of load balancing anyway) is probably a really good idea. Merci!
– Trondh
Nov 14 '18 at 21:28
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53288184%2fdisable-the-on-demand-ngen-exe-on-lightweight-windows-servers-net-framework-n%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You can disable this task but if you get .NET runtime updates you might not get an NGenned .NET Framework. If you have many restarts of your web server you will have in the long run less CPU costs because with every start of your application you will need more CPU to JIT compile your code over and over again. If disabling that task really cuts down your bill depends on how often you update the code and how often you restart your web server.
– Alois Kraus
Nov 13 '18 at 20:35
We don't install updates on these machines, so updates would be implemented by updating the "base image".
– Trondh
Nov 14 '18 at 7:30
Could you explain "you will need more CPU to JIT compile your code over"? The app is already working before ngen kicks in, so is it really needed?
– Trondh
Nov 14 '18 at 7:31
If you start your app 10 times your code will be JIT compiled 10 times. If you have NGenned your code you will have zero JIT costs. That is the reason NGen exists in the first place.
– Alois Kraus
Nov 14 '18 at 8:49
a gotcha. Thanks for clarifying.
– Trondh
Nov 14 '18 at 21:29