TFRecord only selecting a small subset of features
I have a tfrecord file which should have ~8000 image patches in it, all 128x128x4. I've written an input function about the same as what I found on the tensorflow site here:
https://www.tensorflow.org/guide/datasets#randomly_shuffling_input_data
My code to read the tfrecord is:
def tfrecord_input_fn(fileName,
numEpochs=None,
shuffle=None,
batch_size=None):
dataset = tf.data.TFRecordDataset(fileName, compression_type='GZIP')
dataset = dataset.map(lambda example_proto: tf.parse_single_example(example_proto, featuresDict))
if shuffle == True:
dataset = dataset.shuffle(buffer_size=(batch_size * 10))
dataset = dataset.batch(batch_size)
dataset = dataset.repeat(numEpochs)
iterator = dataset.make_one_shot_iterator()
features = iterator.get_next()
return features
When I train it on the network that I've written everything seems to work... however I've noticed that when I'm running it in a session, it only appears to use a very small subset of the input images. I'm using it like this:
with tf.Session() as sess:
for ep in range(num_epochs):
train_img_lab = tfrecord_input_fn(fileName=datapath, numEpochs=epoch_num, shuffle=True, train_batch_size=train_batch_size)
a = sess.run(train_img_lab)
image, label = create_np_arrray_from_tensor(a, train_batch_size)
_, c = sess.run([train_op, train_loss], feed_dict= trainimg_placeholder: image, trainlabels_placeholder: label})
When I wasn't seeing the outputs from the model that I expected I wrote something to save the 128X128 images to my disk. That's when I realized that somehow I was using the same 10 or 15 images of the ~8000 total each batch of each epoch. I suspect this means I'm either not understanding something in my tfrecord_input function (maybe the batch or repeat functions?) or I'm not calling it at the correct point in the session. Could someone help me understand the steps I'm taking, and the steps I need to take, to actually use all the images in my training data?
Thank you!
python tensorflow
add a comment |
I have a tfrecord file which should have ~8000 image patches in it, all 128x128x4. I've written an input function about the same as what I found on the tensorflow site here:
https://www.tensorflow.org/guide/datasets#randomly_shuffling_input_data
My code to read the tfrecord is:
def tfrecord_input_fn(fileName,
numEpochs=None,
shuffle=None,
batch_size=None):
dataset = tf.data.TFRecordDataset(fileName, compression_type='GZIP')
dataset = dataset.map(lambda example_proto: tf.parse_single_example(example_proto, featuresDict))
if shuffle == True:
dataset = dataset.shuffle(buffer_size=(batch_size * 10))
dataset = dataset.batch(batch_size)
dataset = dataset.repeat(numEpochs)
iterator = dataset.make_one_shot_iterator()
features = iterator.get_next()
return features
When I train it on the network that I've written everything seems to work... however I've noticed that when I'm running it in a session, it only appears to use a very small subset of the input images. I'm using it like this:
with tf.Session() as sess:
for ep in range(num_epochs):
train_img_lab = tfrecord_input_fn(fileName=datapath, numEpochs=epoch_num, shuffle=True, train_batch_size=train_batch_size)
a = sess.run(train_img_lab)
image, label = create_np_arrray_from_tensor(a, train_batch_size)
_, c = sess.run([train_op, train_loss], feed_dict= trainimg_placeholder: image, trainlabels_placeholder: label})
When I wasn't seeing the outputs from the model that I expected I wrote something to save the 128X128 images to my disk. That's when I realized that somehow I was using the same 10 or 15 images of the ~8000 total each batch of each epoch. I suspect this means I'm either not understanding something in my tfrecord_input function (maybe the batch or repeat functions?) or I'm not calling it at the correct point in the session. Could someone help me understand the steps I'm taking, and the steps I need to take, to actually use all the images in my training data?
Thank you!
python tensorflow
add a comment |
I have a tfrecord file which should have ~8000 image patches in it, all 128x128x4. I've written an input function about the same as what I found on the tensorflow site here:
https://www.tensorflow.org/guide/datasets#randomly_shuffling_input_data
My code to read the tfrecord is:
def tfrecord_input_fn(fileName,
numEpochs=None,
shuffle=None,
batch_size=None):
dataset = tf.data.TFRecordDataset(fileName, compression_type='GZIP')
dataset = dataset.map(lambda example_proto: tf.parse_single_example(example_proto, featuresDict))
if shuffle == True:
dataset = dataset.shuffle(buffer_size=(batch_size * 10))
dataset = dataset.batch(batch_size)
dataset = dataset.repeat(numEpochs)
iterator = dataset.make_one_shot_iterator()
features = iterator.get_next()
return features
When I train it on the network that I've written everything seems to work... however I've noticed that when I'm running it in a session, it only appears to use a very small subset of the input images. I'm using it like this:
with tf.Session() as sess:
for ep in range(num_epochs):
train_img_lab = tfrecord_input_fn(fileName=datapath, numEpochs=epoch_num, shuffle=True, train_batch_size=train_batch_size)
a = sess.run(train_img_lab)
image, label = create_np_arrray_from_tensor(a, train_batch_size)
_, c = sess.run([train_op, train_loss], feed_dict= trainimg_placeholder: image, trainlabels_placeholder: label})
When I wasn't seeing the outputs from the model that I expected I wrote something to save the 128X128 images to my disk. That's when I realized that somehow I was using the same 10 or 15 images of the ~8000 total each batch of each epoch. I suspect this means I'm either not understanding something in my tfrecord_input function (maybe the batch or repeat functions?) or I'm not calling it at the correct point in the session. Could someone help me understand the steps I'm taking, and the steps I need to take, to actually use all the images in my training data?
Thank you!
python tensorflow
I have a tfrecord file which should have ~8000 image patches in it, all 128x128x4. I've written an input function about the same as what I found on the tensorflow site here:
https://www.tensorflow.org/guide/datasets#randomly_shuffling_input_data
My code to read the tfrecord is:
def tfrecord_input_fn(fileName,
numEpochs=None,
shuffle=None,
batch_size=None):
dataset = tf.data.TFRecordDataset(fileName, compression_type='GZIP')
dataset = dataset.map(lambda example_proto: tf.parse_single_example(example_proto, featuresDict))
if shuffle == True:
dataset = dataset.shuffle(buffer_size=(batch_size * 10))
dataset = dataset.batch(batch_size)
dataset = dataset.repeat(numEpochs)
iterator = dataset.make_one_shot_iterator()
features = iterator.get_next()
return features
When I train it on the network that I've written everything seems to work... however I've noticed that when I'm running it in a session, it only appears to use a very small subset of the input images. I'm using it like this:
with tf.Session() as sess:
for ep in range(num_epochs):
train_img_lab = tfrecord_input_fn(fileName=datapath, numEpochs=epoch_num, shuffle=True, train_batch_size=train_batch_size)
a = sess.run(train_img_lab)
image, label = create_np_arrray_from_tensor(a, train_batch_size)
_, c = sess.run([train_op, train_loss], feed_dict= trainimg_placeholder: image, trainlabels_placeholder: label})
When I wasn't seeing the outputs from the model that I expected I wrote something to save the 128X128 images to my disk. That's when I realized that somehow I was using the same 10 or 15 images of the ~8000 total each batch of each epoch. I suspect this means I'm either not understanding something in my tfrecord_input function (maybe the batch or repeat functions?) or I'm not calling it at the correct point in the session. Could someone help me understand the steps I'm taking, and the steps I need to take, to actually use all the images in my training data?
Thank you!
python tensorflow
python tensorflow
asked Nov 11 '18 at 19:25
ncuuncuu
11
11
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252356%2ftfrecord-only-selecting-a-small-subset-of-features%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252356%2ftfrecord-only-selecting-a-small-subset-of-features%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown