Kouhei Sutou
null+****@clear*****
Mon Nov 5 12:44:45 JST 2012
Kouhei Sutou 2012-11-05 12:44:45 +0900 (Mon, 05 Nov 2012) New Revision: 5d4b645a2858cef8fa530a39800b91cf8d2fd106 https://github.com/groonga/groonga/commit/5d4b645a2858cef8fa530a39800b91cf8d2fd106 Log: tokenizer: support 1 token with normalization 2 or more tokens with normalization works before. But 1 token with normalization is supported from now. Modified files: lib/ii.c lib/token.c Modified: lib/ii.c (+2 -2) =================================================================== --- lib/ii.c 2012-11-05 11:58:19 +0900 (4710b74) +++ lib/ii.c 2012-11-05 12:44:45 +0900 (c5a8f15) @@ -5443,8 +5443,8 @@ token_info_build(grn_ctx *ctx, grn_obj *lexicon, grn_ii *ii, const char *string, ti = token_info_open(ctx, lexicon, ii, key, size, token->pos, ef & EX_SUFFIX); break; case grn_token_done : - ti = token_info_open(ctx, lexicon, ii, (char *)token->orig, - token->orig_blen, 0, ef); + ti = token_info_open(ctx, lexicon, ii, (const char *)token->curr, + token->curr_size, 0, ef); /* key = _grn_table_key(ctx, lexicon, tid, &size); ti = token_info_open(ctx, lexicon, ii, token->curr, token->curr_size, token->pos, ef); Modified: lib/token.c (+2 -0) =================================================================== --- lib/token.c 2012-11-05 11:58:19 +0900 (4fce9bb) +++ lib/token.c 2012-11-05 12:44:45 +0900 (6d0bb6c) @@ -506,6 +506,8 @@ grn_token_next(grn_ctx *ctx, grn_token *token) } } else { token->status = grn_token_done; + token->curr = token->orig; + token->curr_size = token->orig_blen; } if (token->mode == grn_token_add) { switch (table->header.type) { -------------- next part -------------- HTML����������������������������...Download