Scikit-Learn DBSCAN clustering yielding no clusters
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params="V": covariance)
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 '18 at 17:24
IanIan
1,99632145
add a comment |
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params="V": covariance)
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 '18 at 17:24
IanIan
1,99632145
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23
add a comment |
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params="V": covariance)
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 '18 at 17:24
IanIan
1,99632145
I have a data set with a dozen dimensions (columns) and about 200 observations (rows). This dataset has been normalized using quantile_transform_normalize. (Edit: I tried running the clustering without normalization, but still no luck, so I don't believe this is the cause.) Now I want to cluster the data into several clusters. Until now I had been using KMeans, but I have read that it may not be accurate in higher dimensions and doesn't handle outliers well, so I wanted to compare to DBSCAN to see if I get a different result.
However, when I try to cluster the data with DBSCAN using the Mahalanobis distance metric, every item is clustered into -1. According to the documentation:
Noisy samples are given the label -1.
I'm not really sure what this means, but I was getting some OK clusters with KMeans so I know there is something there to cluster -- it's not just random.
Here is the code I am using for clustering:
covariance = np.cov(data.values.astype("float32"), rowvar=False)
clusterer = sklearn.cluster.DBSCAN(min_samples=6, metric="mahalanobis", metric_params="V": covariance)
clusterer.fit(data)
And that's all. I know for certain that data is a numeric Pandas DataFrame as I have inspected it in the debugger.
What could be causing this issue?
python machine-learning scikit-learn cluster-analysis dbscan
python machine-learning scikit-learn cluster-analysis dbscan
asked Nov 12 '18 at 17:24
IanIan
1,99632145
asked Nov 12 '18 at 17:24
IanIan
1,99632145
edited Nov 12 '18 at 17:33
Ian
asked Nov 12 '18 at 17:24
IanIan
1,99632145
asked Nov 12 '18 at 17:24
IanIan
1,99632145
asked Nov 12 '18 at 17:24
IanIan
1,99632145
1,99632145
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23
add a comment |
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23
add a comment |
1 Answer
1
active
oldest
votes
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53267148%2fscikit-learn-dbscan-clustering-yielding-no-clusters%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
add a comment |
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
add a comment |
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
You need to choose the parameter eps, too.
DBSCAN results depend on this parameter very much. You can find some methods for estimating it in literature.
IMHO, sklearn should not provide a default for this parameter, because it rarely ever works (on normalized toy data it is usually okay, but that's about it).
200 instances probably is too small to reliably measure density, in particular with a dozen variables.
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
answered Nov 13 '18 at 8:50
Anony-MousseAnony-Mousse
58.5k797162
58.5k797162
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
add a comment |
How is the defaultepsdetermined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.
– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
How is the default
eps determined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.– Ian
Nov 14 '18 at 15:49
How is the default
eps determined? I was a little suspicious of it but I couldn't find any documentation, so I figured it must be an acceptable default.– Ian
Nov 14 '18 at 15:49
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
Someone at some point chose a default value. There is no data dependent process. Which is why it most often does not work. File a bug that you find the default value misleading.
– Anony-Mousse
Nov 15 '18 at 20:20
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53267148%2fscikit-learn-dbscan-clustering-yielding-no-clusters%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Maybe your data is just not "dense" enough for DBScan. Have you tried to use another metric and to adjust the min_samples value?
– ixeption
Nov 12 '18 at 23:23