Read and list large number of files from multiple folders in c#
up vote
0
down vote
favorite
I want to read large number of files which spans across multiple folders in a network drive.
I have approximately 20 folders, each folder has 50 sub folders and each sub folder has approximately 400 json files. I need to read one keyword in json file and if then if the data is good then I need to parse the file. From all these files I may need to parse only 300 files a day, but I need to go through each file to find if it's good or not.
I have tried reading files fro each folder using,
Directory.EnumerateFiles
But it takes a long time. What is the bes way to handle this situation?
Is there a better way to some powershell script or Pearl etc to traverse through all folders and get paths of all good files in a list as text file and I can read only those good files?
My program is in c#
c# powershell-v2.0 system.io.file system.io.directory
asked Nov 8 at 18:10
blue
361529
add a comment |
up vote
0
down vote
favorite
I want to read large number of files which spans across multiple folders in a network drive.
I have approximately 20 folders, each folder has 50 sub folders and each sub folder has approximately 400 json files. I need to read one keyword in json file and if then if the data is good then I need to parse the file. From all these files I may need to parse only 300 files a day, but I need to go through each file to find if it's good or not.
I have tried reading files fro each folder using,
Directory.EnumerateFiles
But it takes a long time. What is the bes way to handle this situation?
Is there a better way to some powershell script or Pearl etc to traverse through all folders and get paths of all good files in a list as text file and I can read only those good files?
My program is in c#
c# powershell-v2.0 system.io.file system.io.directory
asked Nov 8 at 18:10
blue
361529
1
Directory.EnumerateFilesreturns the file names that it finds. How is that slow? What code are you using to read the file contents?
– gunr2171
Nov 8 at 18:13
1
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
1
There is no faster way of obtaining file names from the file system thanDirectory.EnumerateFiles(and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.
– gunr2171
Nov 8 at 19:10
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I want to read large number of files which spans across multiple folders in a network drive.
I have approximately 20 folders, each folder has 50 sub folders and each sub folder has approximately 400 json files. I need to read one keyword in json file and if then if the data is good then I need to parse the file. From all these files I may need to parse only 300 files a day, but I need to go through each file to find if it's good or not.
I have tried reading files fro each folder using,
Directory.EnumerateFiles
But it takes a long time. What is the bes way to handle this situation?
Is there a better way to some powershell script or Pearl etc to traverse through all folders and get paths of all good files in a list as text file and I can read only those good files?
My program is in c#
c# powershell-v2.0 system.io.file system.io.directory
asked Nov 8 at 18:10
blue
361529
I want to read large number of files which spans across multiple folders in a network drive.
I have approximately 20 folders, each folder has 50 sub folders and each sub folder has approximately 400 json files. I need to read one keyword in json file and if then if the data is good then I need to parse the file. From all these files I may need to parse only 300 files a day, but I need to go through each file to find if it's good or not.
I have tried reading files fro each folder using,
Directory.EnumerateFiles
But it takes a long time. What is the bes way to handle this situation?
Is there a better way to some powershell script or Pearl etc to traverse through all folders and get paths of all good files in a list as text file and I can read only those good files?
My program is in c#
c# powershell-v2.0 system.io.file system.io.directory
c# powershell-v2.0 system.io.file system.io.directory
asked Nov 8 at 18:10
blue
361529
asked Nov 8 at 18:10
blue
361529
asked Nov 8 at 18:10
blue
361529
asked Nov 8 at 18:10
blue
361529
asked Nov 8 at 18:10
blue
361529
361529
1
Directory.EnumerateFilesreturns the file names that it finds. How is that slow? What code are you using to read the file contents?
– gunr2171
Nov 8 at 18:13
1
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
1
There is no faster way of obtaining file names from the file system thanDirectory.EnumerateFiles(and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.
– gunr2171
Nov 8 at 19:10
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16
add a comment |
1
Directory.EnumerateFilesreturns the file names that it finds. How is that slow? What code are you using to read the file contents?
– gunr2171
Nov 8 at 18:13
1
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
1
There is no faster way of obtaining file names from the file system thanDirectory.EnumerateFiles(and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.
– gunr2171
Nov 8 at 19:10
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16
1
1
Directory.EnumerateFiles returns the file names that it finds. How is that slow? What code are you using to read the file contents?– gunr2171
Nov 8 at 18:13
Directory.EnumerateFiles returns the file names that it finds. How is that slow? What code are you using to read the file contents?– gunr2171
Nov 8 at 18:13
1
1
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
1
1
There is no faster way of obtaining file names from the file system than
Directory.EnumerateFiles (and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.– gunr2171
Nov 8 at 19:10
There is no faster way of obtaining file names from the file system than
Directory.EnumerateFiles (and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.– gunr2171
Nov 8 at 19:10
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53213749%2fread-and-list-large-number-of-files-from-multiple-folders-in-c-sharp%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Directory.EnumerateFilesreturns the file names that it finds. How is that slow? What code are you using to read the file contents?– gunr2171
Nov 8 at 18:13
1
Possible duplicate of Reading a large number of files quickly
– gunr2171
Nov 8 at 18:14
Files are on a network share and one parent folder has around 700K files in it including ~ 3000 sub directories. If I do EnumerateFiles in the main directory then it takes very long time.
– blue
Nov 8 at 19:05
1
There is no faster way of obtaining file names from the file system than
Directory.EnumerateFiles(and variations). Trying to read that many folders/files across a network will always be slow, regardless of the programming language you use. The better way to fix this is to think like an engineer and see how you can optimize the layout of your folders so you don't need do do a complete file scan.– gunr2171
Nov 8 at 19:10
I can't change the file structure here, it exists in the system for many years and it can't be changed now. Only option for me to read the files in a better way.
– blue
Nov 8 at 19:16