How to rate quality of a (scraped) sentence?

I am running a scrape and process routine in Python3 - but some of the sentences I get are garbage - I would like to reject these but cant figure out how to do it.

I am using POS tagging and chunking with NLTK but that doesnt seem to help me identify non-valid sentences. The number of NNs, VBs etc. doesnt seem to be any different in a garbage "sentence" than a good one.

I guess I am just looking for a simple method to score the grammar of a sentence and reject ones with too many "errors". I tried to use grammar_check but AWS Lambda doesnt like running it. I immediately get "connection refused" error as soon as I initialise it. (NLTK also needs to be 'spoofed' in order to run on AWS Lambda but I found how to do that).

EXAMPLES:
GOOD:Manchester united boss jose mourinho has told his players to 'grow up' in order to stop conceding early on in games following their comeback wins over newcastle bournemouth and juventus

GARBAGE: [latest results brought to you by played 42 draws 8 etihad stadium manchester old trafford manchester etihad stadium manchester old trafford manchester etihad stadium manchester no content available city return to training after manchester derby win external link city draw fc basel in the ucl round of 16 external link report: united 1-2 city external link city win thrilling derby to move 11 point

Thanks for any advice.

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

Could you give some examples of 'good' and 'garbage' sentences?

– CIAndrews
Nov 11 '18 at 7:58

Thanks for the response. I edited my original question to include examples.

– Gingmeister
Nov 12 '18 at 6:19

Unfortunately, a very brief sample is probably only going to bring very specific suggestions. This particular example seems to exhibit a significant amount of repetition; is this true for all the "garbage"?

– tripleee
Nov 12 '18 at 6:35

Based upon the sample, you can have a look at the order of the words + POS tags. E.g. 5 NNs in a row would be rather uncommon in a normal sentence

– CIAndrews
Nov 12 '18 at 7:39

The text scraped from each page seems to random stuff as well as the paragraph text, which is likely to change with each website. And the websites will also change.You make a good point about sequential nouns but that may not be true of other "garbage". I was looking for a simple grammar check that would hopefully score "garbage" so badly on a grammar checker that could be rejected.

– Gingmeister
Nov 12 '18 at 15:54

add a comment |

I am running a scrape and process routine in Python3 - but some of the sentences I get are garbage - I would like to reject these but cant figure out how to do it.

Thanks for any advice.

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

Could you give some examples of 'good' and 'garbage' sentences?

– CIAndrews
Nov 11 '18 at 7:58

Thanks for the response. I edited my original question to include examples.

– Gingmeister
Nov 12 '18 at 6:19

Unfortunately, a very brief sample is probably only going to bring very specific suggestions. This particular example seems to exhibit a significant amount of repetition; is this true for all the "garbage"?

– tripleee
Nov 12 '18 at 6:35

Based upon the sample, you can have a look at the order of the words + POS tags. E.g. 5 NNs in a row would be rather uncommon in a normal sentence

– CIAndrews
Nov 12 '18 at 7:39

The text scraped from each page seems to random stuff as well as the paragraph text, which is likely to change with each website. And the websites will also change.You make a good point about sequential nouns but that may not be true of other "garbage". I was looking for a simple grammar check that would hopefully score "garbage" so badly on a grammar checker that could be rejected.

– Gingmeister
Nov 12 '18 at 15:54

add a comment |

I am running a scrape and process routine in Python3 - but some of the sentences I get are garbage - I would like to reject these but cant figure out how to do it.

Thanks for any advice.

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

I am running a scrape and process routine in Python3 - but some of the sentences I get are garbage - I would like to reject these but cant figure out how to do it.

Thanks for any advice.

python-3.x parsing nlp nltk

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

edited Nov 12 '18 at 6:23

asked Nov 11 '18 at 7:05

Gingmeister

537

asked Nov 11 '18 at 7:05

Gingmeister

537

asked Nov 11 '18 at 7:05

Gingmeister

537

Could you give some examples of 'good' and 'garbage' sentences?

– CIAndrews
Nov 11 '18 at 7:58

Thanks for the response. I edited my original question to include examples.

– Gingmeister
Nov 12 '18 at 6:19

Unfortunately, a very brief sample is probably only going to bring very specific suggestions. This particular example seems to exhibit a significant amount of repetition; is this true for all the "garbage"?

– tripleee
Nov 12 '18 at 6:35

Based upon the sample, you can have a look at the order of the words + POS tags. E.g. 5 NNs in a row would be rather uncommon in a normal sentence

– CIAndrews
Nov 12 '18 at 7:39

The text scraped from each page seems to random stuff as well as the paragraph text, which is likely to change with each website. And the websites will also change.You make a good point about sequential nouns but that may not be true of other "garbage". I was looking for a simple grammar check that would hopefully score "garbage" so badly on a grammar checker that could be rejected.

– Gingmeister
Nov 12 '18 at 15:54

add a comment |

Could you give some examples of 'good' and 'garbage' sentences?

– CIAndrews
Nov 11 '18 at 7:58

Thanks for the response. I edited my original question to include examples.

– Gingmeister
Nov 12 '18 at 6:19

Unfortunately, a very brief sample is probably only going to bring very specific suggestions. This particular example seems to exhibit a significant amount of repetition; is this true for all the "garbage"?

– tripleee
Nov 12 '18 at 6:35

Based upon the sample, you can have a look at the order of the words + POS tags. E.g. 5 NNs in a row would be rather uncommon in a normal sentence

– CIAndrews
Nov 12 '18 at 7:39

The text scraped from each page seems to random stuff as well as the paragraph text, which is likely to change with each website. And the websites will also change.You make a good point about sequential nouns but that may not be true of other "garbage". I was looking for a simple grammar check that would hopefully score "garbage" so badly on a grammar checker that could be rejected.

– Gingmeister
Nov 12 '18 at 15:54

Could you give some examples of 'good' and 'garbage' sentences?

– CIAndrews
Nov 11 '18 at 7:58

Thanks for the response. I edited my original question to include examples.

– Gingmeister
Nov 12 '18 at 6:19

Unfortunately, a very brief sample is probably only going to bring very specific suggestions. This particular example seems to exhibit a significant amount of repetition; is this true for all the "garbage"?

– tripleee
Nov 12 '18 at 6:35

Based upon the sample, you can have a look at the order of the words + POS tags. E.g. 5 NNs in a row would be rather uncommon in a normal sentence

– CIAndrews
Nov 12 '18 at 7:39

The text scraped from each page seems to random stuff as well as the paragraph text, which is likely to change with each website. And the websites will also change.You make a good point about sequential nouns but that may not be true of other "garbage". I was looking for a simple grammar check that would hopefully score "garbage" so badly on a grammar checker that could be rejected.

– Gingmeister
Nov 12 '18 at 15:54

add a comment |

1 Answer
1

active

oldest

votes

I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.

The method I am attempting to work out in more detail amounts to articulating a number of constraints on well-formed sentences. There is no way you can enumerate all the possible kinds of noise in a corpus, but you can remove certain kinds of noise with certain kinds of filters, many of them ideally simple to understand and implement.

For example:

Discard samples with a very low entropy

Discard samples with characters or character sequences outside of the normal repertoire of English

Discard samples with many repeated words

Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.

A prototype of this method was published in the LREC 2016 proceedings (helpfully, the proceedings are published under a lenient CC BY-NC 4.0 license): abstract but the submissions were restricted to a maximum of four pages, so the article is by necessity a very brief overview. My actual materials and scripts are on Github: https://github.com/rcv2/rcv2r1; but the corpus I used is not redistributable, so there's a piece missing.

Appendix A outlines a brief catalog of proposed constraints.

answered Nov 16 '18 at 14:43

robin eriksson

262

add a comment |

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246564%2fhow-to-rate-quality-of-a-scraped-sentence%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.

For example:

Discard samples with a very low entropy

Discard samples with characters or character sequences outside of the normal repertoire of English

Discard samples with many repeated words

Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.

Appendix A outlines a brief catalog of proposed constraints.

answered Nov 16 '18 at 14:43

robin eriksson

262

add a comment |

I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.

For example:

Discard samples with a very low entropy

Discard samples with characters or character sequences outside of the normal repertoire of English

Discard samples with many repeated words

Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.

Appendix A outlines a brief catalog of proposed constraints.

answered Nov 16 '18 at 14:43

robin eriksson

262

add a comment |

I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.

For example:

Discard samples with a very low entropy

Discard samples with characters or character sequences outside of the normal repertoire of English

Discard samples with many repeated words

Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.

Appendix A outlines a brief catalog of proposed constraints.

answered Nov 16 '18 at 14:43

robin eriksson

262

I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.

For example:

Discard samples with a very low entropy

Discard samples with characters or character sequences outside of the normal repertoire of English

Discard samples with many repeated words

Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.

Appendix A outlines a brief catalog of proposed constraints.

answered Nov 16 '18 at 14:43

robin eriksson

262

answered Nov 16 '18 at 14:43

robin eriksson

262

answered Nov 16 '18 at 14:43

robin eriksson

262

answered Nov 16 '18 at 14:43

robin eriksson

262

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Dfyjkt