how can i read file fastest way in c++ ? to send data to 'Vertex Buffer Object










0















i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question
























  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

    – S.M.
    Nov 12 '18 at 3:31












  • Have you looked into using a memory mapped file?

    – user4942583
    Nov 12 '18 at 3:46












  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

    – o11c
    Nov 12 '18 at 4:10











  • thanks for answers guys i think i need to do another method like said wagner

    – Emre Kaya
    Nov 12 '18 at 4:19















0















i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question
























  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

    – S.M.
    Nov 12 '18 at 3:31












  • Have you looked into using a memory mapped file?

    – user4942583
    Nov 12 '18 at 3:46












  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

    – o11c
    Nov 12 '18 at 4:10











  • thanks for answers guys i think i need to do another method like said wagner

    – Emre Kaya
    Nov 12 '18 at 4:19













0












0








0








i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.










share|improve this question
















i have 4-5 gigabyte ".obj" files like:



Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.



files contents these :




v 16.418303 40.112064 0.078153



vt 0.445198 0.462720



vn 0.264392 0.654428 0.708394



f 27/72/32 38/73/43 23/74/13




they are classic texts



and i am reading each them like this:



char text[10000][60];
ifstream in(filename);
if(!in.is_open())

return false;

while(!in.eof())

in.getline(text[i],60);
i++;



after read one folder i am sending datas to 'VBO'.



all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.



how can i make it faster ?



if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.







c++






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 12 '18 at 4:02









eyllanesc

78.6k103256




78.6k103256










asked Nov 12 '18 at 3:05









Emre KayaEmre Kaya

164




164












  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

    – S.M.
    Nov 12 '18 at 3:31












  • Have you looked into using a memory mapped file?

    – user4942583
    Nov 12 '18 at 3:46












  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

    – o11c
    Nov 12 '18 at 4:10











  • thanks for answers guys i think i need to do another method like said wagner

    – Emre Kaya
    Nov 12 '18 at 4:19

















  • So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

    – S.M.
    Nov 12 '18 at 3:31












  • Have you looked into using a memory mapped file?

    – user4942583
    Nov 12 '18 at 3:46












  • If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

    – o11c
    Nov 12 '18 at 4:10











  • thanks for answers guys i think i need to do another method like said wagner

    – Emre Kaya
    Nov 12 '18 at 4:19
















So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

– S.M.
Nov 12 '18 at 3:31






So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.

– S.M.
Nov 12 '18 at 3:31














Have you looked into using a memory mapped file?

– user4942583
Nov 12 '18 at 3:46






Have you looked into using a memory mapped file?

– user4942583
Nov 12 '18 at 3:46














If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

– o11c
Nov 12 '18 at 4:10





If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.

– o11c
Nov 12 '18 at 4:10













thanks for answers guys i think i need to do another method like said wagner

– Emre Kaya
Nov 12 '18 at 4:19





thanks for answers guys i think i need to do another method like said wagner

– Emre Kaya
Nov 12 '18 at 4:19












1 Answer
1






active

oldest

votes


















0














There is no "precise final answer to your question", once it depends on a couple variables here.



C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



Google for it and learn a little more.



About reading binary data in C++ is easy:



std::ifstream file("somedata.obj", ios::binary);
if (file)

char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!



But again, this is not enough. You need to wisely interpret the data you just read.






share|improve this answer






















    Your Answer






    StackExchange.ifUsing("editor", function ()
    StackExchange.using("externalEditor", function ()
    StackExchange.using("snippets", function ()
    StackExchange.snippets.init();
    );
    );
    , "code-snippets");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "1"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    There is no "precise final answer to your question", once it depends on a couple variables here.



    C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



    Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



    Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



    Google for it and learn a little more.



    About reading binary data in C++ is easy:



    std::ifstream file("somedata.obj", ios::binary);
    if (file)

    char buffer[1024];
    file.read(buffer, sizeof(buffer));
    // buffer has you binary data... you can interpret it!



    But again, this is not enough. You need to wisely interpret the data you just read.






    share|improve this answer



























      0














      There is no "precise final answer to your question", once it depends on a couple variables here.



      C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



      Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



      Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



      Google for it and learn a little more.



      About reading binary data in C++ is easy:



      std::ifstream file("somedata.obj", ios::binary);
      if (file)

      char buffer[1024];
      file.read(buffer, sizeof(buffer));
      // buffer has you binary data... you can interpret it!



      But again, this is not enough. You need to wisely interpret the data you just read.






      share|improve this answer

























        0












        0








        0







        There is no "precise final answer to your question", once it depends on a couple variables here.



        C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



        Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



        Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



        Google for it and learn a little more.



        About reading binary data in C++ is easy:



        std::ifstream file("somedata.obj", ios::binary);
        if (file)

        char buffer[1024];
        file.read(buffer, sizeof(buffer));
        // buffer has you binary data... you can interpret it!



        But again, this is not enough. You need to wisely interpret the data you just read.






        share|improve this answer













        There is no "precise final answer to your question", once it depends on a couple variables here.



        C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).



        Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.



        Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)



        Google for it and learn a little more.



        About reading binary data in C++ is easy:



        std::ifstream file("somedata.obj", ios::binary);
        if (file)

        char buffer[1024];
        file.read(buffer, sizeof(buffer));
        // buffer has you binary data... you can interpret it!



        But again, this is not enough. You need to wisely interpret the data you just read.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 12 '18 at 3:50









        Wagner PatriotaWagner Patriota

        4,0511940




        4,0511940





























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            𛂒𛀶,𛀽𛀑𛂀𛃧𛂓𛀙𛃆𛃑𛃷𛂟𛁡𛀢𛀟𛁤𛂽𛁕𛁪𛂟𛂯,𛁞𛂧𛀴𛁄𛁠𛁼𛂿𛀤 𛂘,𛁺𛂾𛃭𛃭𛃵𛀺,𛂣𛃍𛂖𛃶 𛀸𛃀𛂖𛁶𛁏𛁚 𛂢𛂞 𛁰𛂆𛀔,𛁸𛀽𛁓𛃋𛂇𛃧𛀧𛃣𛂐𛃇,𛂂𛃻𛃲𛁬𛃞𛀧𛃃𛀅 𛂭𛁠𛁡𛃇𛀷𛃓𛁥,𛁙𛁘𛁞𛃸𛁸𛃣𛁜,𛂛,𛃿,𛁯𛂘𛂌𛃛𛁱𛃌𛂈𛂇 𛁊𛃲,𛀕𛃴𛀜 𛀶𛂆𛀶𛃟𛂉𛀣,𛂐𛁞𛁾 𛁷𛂑𛁳𛂯𛀬𛃅,𛃶𛁼

            Crossroads (UK TV series)

            ữḛḳṊẴ ẋ,Ẩṙ,ỹḛẪẠứụỿṞṦ,Ṉẍừ,ứ Ị,Ḵ,ṏ ṇỪḎḰṰọửḊ ṾḨḮữẑỶṑỗḮṣṉẃ Ữẩụ,ṓ,ḹẕḪḫỞṿḭ ỒṱṨẁṋṜ ḅẈ ṉ ứṀḱṑỒḵ,ḏ,ḊḖỹẊ Ẻḷổ,ṥ ẔḲẪụḣể Ṱ ḭỏựẶ Ồ Ṩ,ẂḿṡḾồ ỗṗṡịṞẤḵṽẃ ṸḒẄẘ,ủẞẵṦṟầṓế