djdb/search.py
def _parse_query_string(query_str):
    """Convert a query string into a sequence of annotated terms.

    Our query language is very simple:
      (1) "foo bar" means "find all entities whose text contains both
          the terms "foo" and "bar".
      (2) "-foo" means "exclude all entities whose text contains
          the term "foo".
      (3) "foo*" means "find all entities whose text contains a term starting
          with the prefix "foo".
      (4) "label:blah" means "find all entities whose given field (e.g., label)
          contains the given term.
      (5) "label:blah*" means "find all entities whose given field (e.g., label)
          starts with the prefix "blah".
      (6) "year:1970-1979" means "find all entities whose given field (e.g., year)
          falls within the range "1970" to "1979".

    We automatically filter out query terms that are stop words, as
    well as prefix-query terms that are also the prefix of a stop
    word.

    Returns:
      A sequence of tuples of rules, of the form
        (logic, flavor, arg, field, end)
      all of which are strings except for field and end.

      If logic == IS_REQUIRED, text must match the rule to be returned.
      If logic == IS_FORBIDDEN, text must not match the rule to be returned.
      If flavor == IS_TERM, arg is a term string.  
      If flavor == IS_PREFIX, arg is a term prefix string.
      If flavor == IS_RANGE, arg is a range of values.
      
      field (can be None) is the name of a field to search on.
      end (can be None) is the last term in a series of terms
    """
    query_str_parts = query_str.split()
    query = []
    for qp in query_str_parts:
        is_required = True
        is_prefix = False
        if qp.startswith("-"):
            is_required = False
            qp = qp.lstrip("-")
        if qp.endswith("*"):
            is_prefix = True
            qp = qp.rstrip("*")

        # Check if a field:value is given.
        field_qp = qp.split(':')
        if len(field_qp) == 2:
            if is_prefix:
                flavor = IS_PREFIX
            else:
                flavor = IS_TERM
            if is_required:
                logic = IS_REQUIRED
            else:
                logic = IS_FORBIDDEN
            end = None
            field, qp = field_qp

            if not is_prefix or not _is_stop_word_prefix(qp):
                # Check if a range is given.
                start_end = qp.split('-')
                if len(start_end) == 2:
                    qp, end = start_end

                if not _is_stop_word(qp):
                    query.append((logic, flavor, scrub(qp), field, end))

        else:
            subparts = scrub(qp).split()
            for i, subp in enumerate(subparts):
                flavor = IS_TERM
                logic = IS_REQUIRED
                if i == 0 and not is_required:
                    logic = IS_FORBIDDEN
                if i == len(subparts)-1 and is_prefix:
                    # We need to filter out any prefix which might match a
                    # stop word.  Otherwise a prefix search like "mott
                    # th*" would fail to match "Mott the Hoople".
                    if _is_stop_word_prefix(subp):
                        continue
                    flavor = IS_PREFIX                
                # Skip stop words when building a query 
                # because no stop words exist in the index.
                if _is_stop_word(subp):
                    continue
                # Skip parts where the search term or prefix is empty.
                if not subp:
                    continue
                query.append((logic, flavor, subp, None, None))

    return query
Convert a query string into a sequence of annotated terms.

    Our query language is very simple:
      (1) "foo bar" means "find all entities whose text contains both
          the terms "foo" and "bar".
      (2) "-foo" means "exclude all entities whose text contains
          the term "foo".
      (3) "foo*" means "find all entities whose text contains a term starting
          with the prefix "foo".
      (4) "label:blah" means "find all entities whose given field (e.g., label)
          contains the given term.
      (5) "label:blah*" means "find all entities whose given field (e.g., label)
          starts with the prefix "blah".
      (6) "year:1970-1979" means "find all entities whose given field (e.g., year)
          falls within the range "1970" to "1979".

    We automatically filter out query terms that are stop words, as
    well as prefix-query terms that are also the prefix of a stop
    word.

    Returns:
      A sequence of tuples of rules, of the form
        (logic, flavor, arg, field, end)
      all of which are strings except for field and end.

      If logic == IS_REQUIRED, text must match the rule to be returned.
      If logic == IS_FORBIDDEN, text must not match the rule to be returned.
      If flavor == IS_TERM, arg is a term string.  
      If flavor == IS_PREFIX, arg is a term prefix string.
      If flavor == IS_RANGE, arg is a range of values.
      
      field (can be None) is the name of a field to search on.
      end (can be None) is the last term in a series of terms
    
Convert a query string into a sequence of annotated terms.

We automatically filter out query terms that are stop words, as
well as prefix-query terms that are also the prefix of a stop
word.
Convert a query string into a sequence of annotated terms . We automatically filter out query terms that are stop words as well as prefix - query terms that are also the prefix of a stop word .
def _parse_query_string(query_str):
    query_str_parts = query_str.split()
    query = []
    for qp in query_str_parts:
        is_required = True
        is_prefix = False
        if qp.startswith("-"):
            is_required = False
            qp = qp.lstrip("-")
        if qp.endswith("*"):
            is_prefix = True
            qp = qp.rstrip("*")
        field_qp = qp.split(':')
        if len(field_qp) == 2:
            if is_prefix:
                flavor = IS_PREFIX
            else:
                flavor = IS_TERM
            if is_required:
                logic = IS_REQUIRED
            else:
                logic = IS_FORBIDDEN
            end = None
            field, qp = field_qp
            if not is_prefix or not _is_stop_word_prefix(qp):
                start_end = qp.split('-')
                if len(start_end) == 2:
                    qp, end = start_end
                if not _is_stop_word(qp):
                    query.append((logic, flavor, scrub(qp), field, end))
        else:
            subparts = scrub(qp).split()
            for i, subp in enumerate(subparts):
                flavor = IS_TERM
                logic = IS_REQUIRED
                if i == 0 and not is_required:
                    logic = IS_FORBIDDEN
                if i == len(subparts)-1 and is_prefix:
                    if _is_stop_word_prefix(subp):
                        continue
                    flavor = IS_PREFIX                
                if _is_stop_word(subp):
                    continue
                if not subp:
                    continue
                query.append((logic, flavor, subp, None, None))
    return query
def _parse_query_string ( query_str ) : query_str_parts = query_str . split ( ) query = [ ] for qp in query_str_parts : is_required = True is_prefix = False if qp . startswith ( "-" ) : is_required = False qp = qp . lstrip ( "-" ) if qp . endswith ( "*" ) : is_prefix = True qp = qp . rstrip ( "*" ) field_qp = qp . split ( ':' ) if len ( field_qp ) == 2 : if is_prefix : flavor = IS_PREFIX else : flavor = IS_TERM if is_required : logic = IS_REQUIRED else : logic = IS_FORBIDDEN end = None field , qp = field_qp if not is_prefix or not _is_stop_word_prefix ( qp ) : start_end = qp . split ( '-' ) if len ( start_end ) == 2 : qp , end = start_end if not _is_stop_word ( qp ) : query . append ( ( logic , flavor , scrub ( qp ) , field , end ) ) else : subparts = scrub ( qp ) . split ( ) for i , subp in enumerate ( subparts ) : flavor = IS_TERM logic = IS_REQUIRED if i == 0 and not is_required : logic = IS_FORBIDDEN if i == len ( subparts ) - 1 and is_prefix : if _is_stop_word_prefix ( subp ) : continue flavor = IS_PREFIX if _is_stop_word ( subp ) : continue if not subp : continue query . append ( ( logic , flavor , subp , None , None ) ) return query
Convert a query string into a sequence of annotated terms.
Convert a query string into a sequence of annotated terms .
"""Convert a query string into a sequence of annotated terms.

    Our query language is very simple:
      (1) "foo bar" means "find all entities whose text contains both
          the terms "foo" and "bar".
      (2) "-foo" means "exclude all entities whose text contains
          the term "foo".
      (3) "foo*" means "find all entities whose text contains a term starting
          with the prefix "foo".
      (4) "label:blah" means "find all entities whose given field (e.g., label)
          contains the given term.
      (5) "label:blah*" means "find all entities whose given field (e.g., label)
          starts with the prefix "blah".
      (6) "year:1970-1979" means "find all entities whose given field (e.g., year)
          falls within the range "1970" to "1979".

    We automatically filter out query terms that are stop words, as
    well as prefix-query terms that are also the prefix of a stop
    word.

    Returns:
      A sequence of tuples of rules, of the form
        (logic, flavor, arg, field, end)
      all of which are strings except for field and end.

      If logic == IS_REQUIRED, text must match the rule to be returned.
      If logic == IS_FORBIDDEN, text must not match the rule to be returned.
      If flavor == IS_TERM, arg is a term string.  
      If flavor == IS_PREFIX, arg is a term prefix string.
      If flavor == IS_RANGE, arg is a range of values.
      
      field (can be None) is the name of a field to search on.
      end (can be None) is the last term in a series of terms
    """ # Check if a field:value is given. # Check if a range is given. # We need to filter out any prefix which might match a # stop word.  Otherwise a prefix search like "mott # th*" would fail to match "Mott the Hoople". # Skip stop words when building a query  # because no stop words exist in the index. # Skip parts where the search term or prefix is empty.
A sequence of tuples of rules, of the form
(logic, flavor, arg, field, end)
all of which are strings except for field and end.

If logic == IS_REQUIRED, text must match the rule to be returned.
If logic == IS_FORBIDDEN, text must not match the rule to be returned.
If flavor == IS_TERM, arg is a term string.
If flavor == IS_PREFIX, arg is a term prefix string.
If flavor == IS_RANGE, arg is a range of values.

field (can be None) is the name of a field to search on.
end (can be None) is the last term in a series of terms
A sequence of tuples of rules of the form ( logic flavor arg field end ) all of which are strings except for field and end . If logic == IS_REQUIRED text must match the rule to be returned . If logic == IS_FORBIDDEN text must not match the rule to be returned . If flavor == IS_TERM arg is a term string . If flavor == IS_PREFIX arg is a term prefix string . If flavor == IS_RANGE arg is a range of values . field ( can be None ) is the name of a field to search on . end ( can be None ) is the last term in a series of terms
query_str