8000 Rita.untokenize incorrectly handling apostrophe's at end of word · Issue #477 · dhowe/RiTaV1 · GitHub
[go: up one dir, main page]

Skip to content

Navigation Menu

This repository was archived by the owner on Dec 9, 2025. It is now read-only.
This repository was archived by the owner on Dec 9, 2025. It is now read-only.

Rita.untokenize incorrectly handling apostrophe's at end of word #477

Open

Description

@dhowe

please see my fix (in RiTaJS b1e70577a8fc5c86) and apply the same fix to the failing test in RiTa-java

Activity

  1. dhowe commented on Jul 5, 2017

    @dhowe
    OwnerAuthor

    Here is problematic case in our untokenize tests:

    a. The student said 'learning is fun'
    b. We should consider the students' learning

    In each case, when we get to the single-quote, we don't know whether it is the start of a new quotation or an apostrophe at the end of the previous word

    @cqx931 any thoughts?

  2. cqx931 commented on Jul 6, 2017

    @cqx931
    Collaborator

    When I test RiTa.tokenize(), student’s stays as one token while 'students'' becomes two.
    But if student’s is regarded as one token, shouldn't 'students'' also stays as one? If this is ok, then it would be an issue for tokenize(), rather than untokenize().

    If we do think 'students'' should be broke into two tokens, then are words ending in 's' the only case here? (I can't come up with more cases... ) If it is the only case we could check whether the token before is ending in 's' to differentiate case a and b.

  3. dhowe commented on Jul 6, 2017

    @dhowe
    OwnerAuthor

    This is a good question. I think punctuation that is internal to a word (e.g., student's) should be a single token, while punctuation at end of word (e.g., students' or students" or students, or students:) should be two. One reason is that we don't want the 's' character in "student's" to be a separate token... though we also need to consider the SPLIT_CONTRACTIONS flag.

    In any case @cqx931, can you check/fix the known-issues test for tokenize() in RiTaJS? then lets discuss

  4. added a commit that references this issue on Jul 7, 2017
  5. cqx931 commented on Jul 7, 2017

    @cqx931
    Collaborator

    First fixed the issue in tokenize():
    dhowe/ritajs@cfa2e90

  6. cqx931 commented on Jul 8, 2017

    @cqx931
    Collaborator

    Please check: https://github.com/dhowe/RiTaJS/pull/89
    Fixed the rest issues with untokenize() and added other curly quotes to tokenize().

  7. dhowe commented on Jul 8, 2017

    @dhowe
    OwnerAuthor

    Excellent.

    We still have this failing case:

          var expected = "The student said 'learning is fun'";
          var input = ["The", "student", "said", "'", "learning", "is", "fun", "'"];
          var output = RiTa.untokenize(input);
          deepEqual(output, expected);
    

    But lets leave that alone for now.

    Next, please 1) update the RiTa java tests to match those in JS, then 2) update RiTa.tokenize()/untokenize() functions in java to match the behavior of RiTaJS... you may want to simply rewrite those functions as ports from JS, rather than trying to adapt what's there now

  8. cqx931 commented on Jul 9, 2017

    @cqx931
    Collaborator

    Updated Java and fixed the new failing case.

    Please check https://github.com/dhowe/RiTa/pull/480
    and https://github.com/dhowe/RiTaJS/pull/90

  9. cqx931 commented on Aug 12, 2017

    @cqx931
    Collaborator

    Can this be closed? @dhowe

  10. dhowe commented on Aug 12, 2017

    @dhowe
    OwnerAuthor

    Do we have this case in KnownISsues for both platforms?

         var expected = "The student said 'learning is fun'";
         var input = ["The", "student", "said", "'", "learning", "is", "fun", "'"];
         var output = RiTa.untokenize(input);
         deepEqual(output, expected);
    
  11. cqx931 commented on Aug 12, 2017

    @cqx931
    Collaborator

    This test case is already solved for both platforms (in RiTaTest).

  12. dhowe commented on Aug 14, 2017

    @dhowe
    OwnerAuthor

    So we have both of these arrays as inputs to untokenize() tests (which do pass) 8000 :

    1. ["We", "should", "consider", "the", "students", "'", "learning" ]
    2. ["The", "student", "said", "'", "learning", "is", "fun", "'"]

    Pls explain how the algorithm groups the first single-quote in 1 with the previous word (students'), but does the opposite in 2, and groups it with the subsequent word ('learning)?

  13. dhowe commented on Nov 24, 2017

    @dhowe
    OwnerAuthor
  14. cqx931 commented on Jun 17, 2018

    @cqx931
    Collaborator

    What do you mean by grouping here? Isn't both in case 1 and case 2, that the single quote is considered as a single token?

  15. dhowe commented on Jun 17, 2018

    @dhowe
    OwnerAuthor

    Yes, but we are talking about RiTa.untokenize here, so the output is a single string.

  16. dhowe commented on Sep 24, 2019

    @dhowe
    OwnerAuthor

    @kennyviperhk Please verify that both of the tests below pass (in both RiTa and RiTaJS) for RiTa.untokenize:

        expected = "We should consider the students' learning";
        input = ["We", "should", "consider", "the", "students", "'", "learning" ]
        var output = RiTa.untokenize(input);
        equal(output, expected);
    
        expected = "The student said 'learning is fun'";
        input = ["The", "student", "said", "'", "learning", "is", "fun", "'"]
        var output = RiTa.untokenize(input);
        equal(output, expected);
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    0