Dfyjkt

I'm running faster R-CNN in matlab 2018b on a Windows 10. I face an exception CUDA_ERROR_ILLEGAL_ADDRESS when I increase the number of my training items or when I increase the MaxEpoch.

Below are the information of my gpuDevice

 CUDADevice with properties:

 Name: 'GeForce GTX 1050'
 Index: 1
 ComputeCapability: '6.1'
 SupportsDouble: 1
 DriverVersion: 9.2000
 ToolkitVersion: 9.1000
 MaxThreadsPerBlock: 1024
 MaxShmemPerBlock: 49152
 MaxThreadBlockSize: [1024 1024 64]
 MaxGridSize: [2.1475e+09 65535 65535]
 SIMDWidth: 32
 TotalMemory: 4.2950e+09
 AvailableMemory: 3.4635e+09
 MultiprocessorCount: 5
 ClockRateKHz: 1493000
 ComputeMode: 'Default'
 GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
 CanMapHostMemory: 1
 DeviceSupported: 1
 DeviceSelected: 1

And this is my code

latest_index =0;

for i=1:6

load (strcat('newDataset', int2str(i), '.mat')); 
len =length(vehicleDataset.imageFilename);

for j=1:len

filename = vehicleDataset.imageFilenamej;
latest_index=latest_index+1;
fulldata.imageFilenamelatest_index = filename;
fulldata.vehiclelatest_index = vehicleDataset.vehiclej;

end
end 

trainingDataTable = table(fulldata.imageFilename', fulldata.vehicle');
trainingDataTable.Properties.VariableNames = 'imageFilename','vehicle';

data.trainingDataTable = trainingDataTable;
trainingDataTable(1:4,:)


% Split data into a training and test set.
idx = floor(0.6 * height(trainingDataTable));

trainingData = trainingDataTable(1:idx,:);
testData = trainingDataTable(idx:end,:);

% Create image input layer.
inputLayer = imageInputLayer([32 32 3]);

% Define the convolutional layer parameters.
filterSize = [3 3];

numFilters = 64;

% Create the middle layers.
middleLayers = [

convolution2dLayer(filterSize, numFilters, 'Padding', 1) 
reluLayer()
convolution2dLayer(filterSize, numFilters, 'Padding', 1) 
reluLayer() 
maxPooling2dLayer(3, 'Stride',2)
];

finalLayers = [

fullyConnectedLayer(128)
% Add a ReLU non-linearity.
reluLayer()

fullyConnectedLayer(width(trainingDataTable))
% Add the softmax loss layer and classification layer. 
softmaxLayer()
classificationLayer()
];

layers = [
inputLayer
middleLayers
finalLayers
];

% Options for step 1.
 optionsStage1 = trainingOptions('sgdm', ...
 'MaxEpochs', 2, ...
 'MiniBatchSize', 1, ...
 'InitialLearnRate', 1e-3, ...
 'CheckpointPath', tempdir);

% Options for step 2.
optionsStage2 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);

% Options for step 3.
optionsStage3 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ... 
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);

% Options for step 4.
optionsStage4 = trainingOptions('sgdm', ...
'MaxEpochs', 2, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);

 options = [
 optionsStage1
 optionsStage2
 optionsStage3
 optionsStage4
 ];

 doTrainingAndEval = true;

if doTrainingAndEval
 % Set random seed to ensure example training reproducibility.
 rng(0);

 % Train Faster R-CNN detector. Select a BoxPyramidScale of 1.2 to allow
 % for finer resolution for multiscale object detection.
 detector = trainFasterRCNNObjectDetector(trainingData, layers, options, ...
 'NegativeOverlapRange', [0 0.3], ...
 'PositiveOverlapRange', [0.6 1], ...
 'BoxPyramidScale', 1.2);

 data.detector= detector;
else

% Load pretrained detector for the example.
detector = data.detector;

end

save mix_data data

if doTrainingAndEval

% Run detector on each image in the test set and collect results.

resultsStruct = struct();

for i = 1:height(testData)
 % Read the image.
 I = imread(testData.imageFilenamei);
 % Run the detector.
 [bboxes, scores, labels] = detect(detector, I);

 % Collect the results.
 resultsStruct(i).Boxes = bboxes;
 resultsStruct(i).Scores = scores;
 resultsStruct(i).Labels = labels;
end

% Convert the results into a table.
results = struct2table(resultsStruct);

data.results = results;

save mix_data data

else

% Load results from disk.
results = data.results;
end

% Extract expected bounding box locations from test data.
expectedResults = testData(:, 2:end);

% Evaluate the object detector using Average Precision metric.
[ap, recall, precision] = evaluateDetectionPrecision(results, expectedResults);

% Plot precision/recall curve
figure
plot(recall,precision)

xlabel('Recall')
ylabel('Precision')

grid on
title(sprintf('Average Precision = %.2f', ap))

First it prints the warning multiple time and throws the below exception

Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In trainFasterRCNNObjectDetector (line 320)
In rcnn_trail (line 184)

Error using -
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

Error in vision.internal.cnn.layer.SmoothL1Loss/backwardLoss (line 156)
idx = (X > -one) & (X < one);
Error in nnet.internal.cnn.DAGNetwork/computeGradientsForTraining/efficientBackProp (line 585)
dLossdX = thisLayer.backwardLoss( ...

Error in nnet.internal.cnn.DAGNetwork>@()efficientBackProp(i) (line 661)
@() efficientBackProp(i), ...

Error in nnet.internal.cnn.util.executeWithStagedGPUOOMRecovery (line 11)
[ varargout1:nOutputs ] = computeFun();

Error in nnet.internal.cnn.DAGNetwork>iExecuteWithStagedGPUOOMRecovery (line 1195)
[varargout1:nargout] = nnet.internal.cnn.util.executeWithStagedGPUOOMRecovery(varargin:);

Error in nnet.internal.cnn.DAGNetwork/computeGradientsForTraining (line 660)
theseGradients = iExecuteWithStagedGPUOOMRecovery( ...

Error in nnet.internal.cnn.Trainer/computeGradients (line 184)
[gradients, predictions, states] = net.computeGradientsForTraining(X, Y,
needsStatefulTraining, propagateState);

Error in nnet.internal.cnn.Trainer/train (line 85)
[gradients, predictions, states] = this.computeGradients(net, X, response,
needsStatefulTraining, propagateState);

Error in vision.internal.cnn.trainNetwork (line 47)
trainedNet = trainer.train(trainedNet, trainingDispatcher);

Error in fastRCNNObjectDetector.train (line 190)
[network, info] = vision.internal.cnn.trainNetwork(ds, lgraph, opts, mapping,
checkpointSaver);

Error in trainFasterRCNNObjectDetector (line 410)
[stage2Detector, fastRCNN, ~, info(2)] = fastRCNNObjectDetector.train(trainingData, fastRCNN,
options(2), iStageTwoParams(params), checkpointSaver);

Error in rcnn_trail (line 184)
detector = trainFasterRCNNObjectDetector(trainingData, layers, options, ...

After talking to Matlab support, apparently my GPU is not the "right" GPU for deep learning and Neural Network.

However, I found that the issue was that Windows changed the GPU during the run, to fix this I went to INVIDIA Control Panel > Programs settings >
1. Select Mathworks Matlab
2. Preferred graphic processor choose your GPU card