Yoji Shidara
null+****@clear*****
Thu Feb 27 18:00:54 JST 2014
Yoji Shidara 2014-02-27 18:00:54 +0900 (Thu, 27 Feb 2014) New Revision: ee8e59489a6f700f6239f1eedec0eb4e48036dbb https://github.com/droonga/droonga.org/commit/ee8e59489a6f700f6239f1eedec0eb4e48036dbb Message: ja: Update translation Modified files: ja/tutorial/plugin-development/adapter/index.md ja/tutorial/plugin-development/handler/index.md ja/tutorial/plugin-development/index.md Modified: ja/tutorial/plugin-development/adapter/index.md (+8 -4) =================================================================== --- ja/tutorial/plugin-development/adapter/index.md 2014-02-27 17:45:07 +0900 (8e24be0) +++ ja/tutorial/plugin-development/adapter/index.md 2014-02-27 18:00:54 +0900 (3c15cf9) @@ -19,7 +19,7 @@ layout: ja Learning steps to develop a Droonga plugin by yourself. -This page focuses on the adaption phase for Droonga plugins. +This page focuses at the adaption phase for Droonga plugins. At the last, wraps up them to make a small practical plugin named `store-search`, for the adaption phase. ## 前提条件 @@ -29,7 +29,7 @@ At the last, wraps up them to make a small practical plugin named `store-search` ## Adaption for incoming messages -First, let's study basics with a simple logger plugin named `sample-logger` affects on the adaption phase. +First, let's study basics with a simple logger plugin named `sample-logger` affects at the adaption phase. We sometime need to modify incoming requests from outside to Droonga Engine. We can use a plugin for this purpose. @@ -72,7 +72,9 @@ require "droonga/plugin" module Droonga module Plugins module SampleLoggerPlugin - Plugin.registry.register("sample-logger", self) + extend Plugin + + registry.register("sample-logger", self) class Adapter < Droonga::Adapter # You'll put codes to modify messages here. @@ -86,6 +88,8 @@ This plugin does nothing except registering itself to Droonga. * The `sample-logger` is the name of the plugin itself. You'll use it in your `catalog.json`, to activate the plugin. * As the example above, you must define your plugin as a module. + * Behaviors at the adaption phase is defined a class called *adapter*. + An adapter class must be defined as a subclass of the `Droonga::Adapter`, under the namespace of the plugin module. ### Activate the plugin with `catalog.json` @@ -104,7 +108,7 @@ catalog.json: (snip) ~~~ -Note: you must place `"sample-logger"` before `"search"`, because the `sample-logger` plugin depends on the `search`. Droonga Engine applies plugins on the adaption phase in the order defined in the `catalog.json`, so you must resolve plugin dependencies by your hand (for now). +Note: you must place `"sample-logger"` before `"search"`, because the `sample-logger` plugin depends on the `search`. Droonga Engine applies plugins at the adaption phase in the order defined in the `catalog.json`, so you must resolve plugin dependencies by your hand (for now). ### Run Modified: ja/tutorial/plugin-development/handler/index.md (+398 -75) =================================================================== --- ja/tutorial/plugin-development/handler/index.md 2014-02-27 17:45:07 +0900 (8ba6fe0) +++ ja/tutorial/plugin-development/handler/index.md 2014-02-27 18:00:54 +0900 (2af5cfb) @@ -1,5 +1,5 @@ --- -title: "Plugin: Handle requests" +title: "Plugin: Handle requests on all partitions" layout: en --- @@ -12,63 +12,163 @@ layout: en {% endcomment %} -!!! WORK IN PROGRESS !!! - * TOC {:toc} ## チュートリアルのゴール -This tutorial aims to help you to learn how to develop plugins -which extends operations in handle phrase. +This tutorial aims to help you to learn how to develop plugins which do something dispersively for/in each partition, around the handling phase. ## 前提条件 -* You must complete [Modify requests and responses tutorial][adapter]. +* You must complete the [tutorial for the adaption phase][adapter]. + +## Handling of incoming messages + +When an incoming message is transferred from the adaption phase, the Droonga Engine enters into the *processing phase*. + +In the processing phase, the Droonga Engine processes incoming messages step by step. +One *step* is constructed from some sub phases: *planning phase*, *distribution phase*, *handling phase*, and *collection phase*. + + * At the *planning phase*, the Droonga Engine generates multiple sub steps to process an incoming message. + In simple cases, you don't have to write codes for this phase, then there is just one sub step to handle the message. + * At the *distribution phase*, the Droonga Engine distributes the message to multiple partitions. + (It is completely done by the Droonga Engine itself, so this phase is not pluggable.) + * At the *handling phase*, *each partition simply processes only one distributed message as its input, and returns a result.* + This is the time that actual storage accesses happen. + Actually, some commands (`search`, `add`, `create_table` and so on) access to the storage at the time. + * At the *collection phase*, the Droonga Engine collects results from partitions to one unified result. + There are some useful generic collectors, so you don't have to write codes for this phase in most cases. + +After all steps are finished, the Droonga Engine transfers the result to the post adaption phase. + +A class to define operations at the handling phase is called *handler*. +Put simply, adding of a new handler means adding a new command. + + + -## Handling phase -The handling phase is the phase that the actual storage access is happen. -As Droonga is a distributed system, handler phase is done in multiple partitions. -Here, in this tutorial, we are going to replace the handling phase of `search` command for explanation. This breaks the `search` command. So this is not useful in practice, but it will help you to learn how Droonga works. +## Design a read-only command `countRecords` + +Here, in this tutorial, we are going to add a new custom `countRecords` command. +At first, let's design it. + +The command reports the number of records about a specified table, for each partition. +So it will help you to know how records are distributed in the cluster. +Nothing is changed by the command, so it is a *read-only command*. + +The request must have the name of one table, like: + +~~~json +{ + "dataset" : "Starbucks", + "type" : "countRecords", + "body" : { + "table": "Store" + } +} +~~~ + +Create a JSON file `count-records.json` with the content above. +We'll use it for testing. + +The response must have number of records in the table, for each partition. +They can be appear in an array, like: + +~~~json +{ + "inReplyTo": "(message id)", + "statusCode": 200, + "type": "countRecords.result", + "body": [10, 10] +} +~~~ -In practice, we need to *extend* Droonga. In this case, we need to add a new command which does not conflict with the existing commands. To do so, you need to learn not only how to handle messages but also how to distribute messages to handlers and collect messages from them. Proceed to [Distribute requests and collect responses][] after this tutorial completed. +If there are 2 partitions and 20 records are stored evenly, the array will have two elements like above. +It means that a partition has 10 records and another one also has 10 records. -TODO fix the link to "Distribute requests and collect responses" tutorial +We're going to create a plugin to accept such requests and return such responses. -## Directory Structure -The directory structure for plugins are in same rule as explained in [Modify requests and responses tutorial][adapter]. +### Directory structure -Now let's create `sample-logger` plugin again. This will act almost same as [Modify requests and responses tutorial][adapter] version, except the phase in which the plugin works. We need to put `sample-logger.rb` to `lib/droonga/plugins/sample-logger.rb`. The directory tree will be like this: +The directory structure for plugins are in same rule as explained in the [tutorial for the adaption phase][adapter]. +Now let's create the `count-records` plugin, as the file `count-records.rb`. The directory tree will be: ~~~ lib └── droonga └── plugins - └── sample-logger.rb + └── count-records.rb ~~~ -## Create a plugin +Then, create a skelton of a plugin as follows: -Create a plugin as follows: +lib/droonga/plugins/count-records.rb: -lib/droonga/plugins/sample-logger.rb: +~~~ruby +require "droonga/plugin" + +module Droonga + module Plugins + module CountRecordsPlugin + Plugin.registry.register("count-records", self) + end + end +end +~~~ + +### Define a "step" for the command + +Define a "step" for the new `countRecords` command, in your plugin. Like: + +lib/droonga/plugins/count-records.rb: ~~~ruby require "droonga/plugin" module Droonga module Plugins - module SampleLoggerPlugin - Plugin.registry.register("sample-logger", self) + module CountRecordsPlugin + Plugin.registry.register("count-records", self) - class Handler < Droonga::Handler - message.type = "search" + define_single_step do |step| + step.name = "countRecords" + end + end + end +end +~~~ + +The `step.name` equals to the name of the command itself. +Currently we just define the name of the command. +That's all. - def handle(message, messenger) - $log.info "Droonga::Plugins::SampleLoggerPlugin", :message => message +### Define the handling logic + +The command has no handler, so it does nothing yet. +Let's define the behavior. + +lib/droonga/plugins/count-records.rb: + +~~~ruby +require "droonga/plugin" + +module Droonga + module Plugins + module CountRecordsPlugin + Plugin.registry.register("count-records", self) + + define_single_step do |step| + step.name = "countRecords" + step.handler = :Handler + end + + class Handler < Droonga::Handler + def handle(message) + [0] end end end @@ -76,78 +176,270 @@ module Droonga end ~~~ -## Activate the plugin with `catalog.json` +The class `Handler` is a handler class for our new command. + + * It must inherit a builtin-class `Droonga::Handler`. + * It implements the logic to handle requests. + Its instance method `#handle` actually handles requests. + +Currently the handler does nothing and returns an array of a number. +The returned value is used to construct the response body. + +The handler is bound to the step with the configuration `step.handler`. +Because we define the class `Handler` after `define_single_step`, we specify the handler class with a symbol `:Handler`. +If you define the handler class before `define_single_step`, then you can write as `step.handler = Handler` simply. +Moreover, a class path string like `"OtherPlugin::Handler"` is also available. -Update catalog.json to activate this plugin. Add `"sample-logger"` to `"plugins"`. +Then, we also have to bind a collector to the step, with the configuration `step.collector`. + +lib/droonga/plugins/count-records.rb: + +~~~ruby +(snip) + define_single_step do |step| + step.name = "countRecords" + step.handler = :Handler + step.collector = SumCollector + end +(snip) +~~~ + +The `SumCollector` is one of built-in collectors. +It merges results retuned from handler instances for each partition to one result. + + +### Activate the plugin with `catalog.json` + +Update catalog.json to activate this plugin. +Add `"count-records"` to `"plugins"`. ~~~ (snip) "datasets": { "Starbucks": { (snip) - "plugins": ["sample-logger", "groonga", "crud", "search"], + "plugins": ["count-records", "groonga", "crud", "search"], (snip) ~~~ -## Run +### Run and test -Let's get Droonga started. Note that you need to specify ./lib directory in RUBYLIB environment variable in order to make ruby possible to find your plugin. +Let's get Droonga started. +Note that you need to specify ./lib directory in RUBYLIB environment variable in order to make ruby possible to find your plugin. # kill $(cat fluentd.pid) # RUBYLIB=./lib fluentd --config fluentd.conf --log fluentd.log --daemon fluentd.pid -## Test +Then, send a message for the `countRecords` command to the Droonga Engine. + +~~~ +# droonga-request --tag starbucks count-records.json +Elapsed time: 0.01494 +[ + "droonga.message", + 1392621168, + { + "inReplyTo": "1392621168.0119512", + "statusCode": 200, + "type": "countRecords.result", + "body": [0, 0, 0] + } +] +~~~ + +You'll get a response message like above. +Look at these points: + + * The `type` of the response becomes `countRecords.result`. + It is automatically named by the Droonga Engine. + * The format of the `body` is same to the returned value of the handler's `handle` method. + +There are 3 elements in the array. Why? + + * Remember that we have configured the `Starbucks` dataset to use 3 partitions (and each has 2 replicas) in the `catalog.json` of [the basic tutorial][basic]. + * Because it is a read-only command, an incoming message is distributed only to paritions, not to replicas. + So there are only 3 results, not 6. + (TODO: I have to add a figure to indicate active nodes: [000, 001, 010, 011, 020, 021] => [000, 011, 020]) + * The `SumCollector` collects them. + Those 3 results are joined to just one array by the collector. + +As the result, just one array with 3 elements appears in the response message. + +### Read-only access to the storage + +Now, each instance of the handler class always returns `[0]` as its result. +Let's implement codes to count up the number of records from the actual storage. + +lib/droonga/plugins/count-records.rb: -Send a search request to Droonga Engine. Use `search-columbus.json` same as of [Modify requests and responses tutorial][adapter]. +~~~ruby +(snip) + class Handler < Droonga::Handler + def handle(message) + table_name = message["body"]["table"] + table = @context[table_name] + count = table.size + [count] + end + end +(snip) +~~~ + +The instance variable `@context` is an instance of `Groonga::Context` for the storage of the partition. +See the [class reference of Rroonga][Groonga::Context]. +You can use any feature of Rroonga via `@context`. +For now, we simply access to the table itself by its name and read the value of its `size` method - it returns the number of records. + +Then, test it. +Restart the Droonga Engine and send the request again. ~~~ -# droonga-request --tag starbucks search-columbus.json +# kill $(cat fluentd.pid) +# RUBYLIB=./lib fluentd --config fluentd.conf --log fluentd.log --daemon fluentd.pid +# droonga-request --tag starbucks count-records.json +Elapsed time: 0.01494 +[ + "droonga.message", + 1392621168, + { + "inReplyTo": "1392621168.0119512", + "statusCode": 200, + "type": "countRecords.result", + "body": [12, 12, 11] + } +] ~~~ -You will see no output for `droonga-request` execution because out `sample-logger` plugin traps the `add` request. +Because there are totally 35 records, they are stored evenly like above. + + + + + + + +## Design a read-write command `deleteStores` + +Next, let's add another new custom command `deleteStores`. -Instead, you will see something like these lines in `fluentd.log`: +The command deletes records of the `Store` table, from the storage. +Because it modifies something in existing storage, it is a *read-write command*. +The request must have the condition to select records to be deleted, like: + +~~~json +{ + "dataset" : "Starbucks", + "type" : "deleteStores", + "body" : { + "keyword": "Broardway" + } +} ~~~ -2014-02-17 16:25:23 +0900 [info]: Droonga::Plugins::SampleLoggerPlugin message=#<Droonga::HandlerMessage:0x007f9a7f0987a8 @raw={"dataset"=>"Starbucks", "type"=>"search", "body"=>{"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.011", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores"=>["localhost:24224/star bucks"]} }, "replyTo"=>{"type"=>"search.result", "to"=>"127.0.0.1:50410/droonga"}, "id"=>"1392621923.903868", "date"=>"2014-02-17 16:25:23 +0900", "appliedAdapters"=>["Droonga::Plugins::Error::Adapter"]}, @body={"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.011", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores" =>["loca lhost:24224/starbucks"]}}, @task={"route"=>"localhost:24224/starbucks.011", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, @step={"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1} }}}, "ty pe"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}> -2014-02-17 16:25:23 +0900 [info]: Droonga::Plugins::SampleLoggerPlugin message=#<Droonga::HandlerMessage:0x007f9a7f060970 @raw={"dataset"=>"Starbucks", "type"=>"search", "body"=>{"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.020", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores"=>["localhost:24224/star bucks"]} }, "replyTo"=>{"type"=>"search.result", "to"=>"127.0.0.1:50410/droonga"}, "id"=>"1392621923.903868", "date"=>"2014-02-17 16:25:23 +0900", "appliedAdapters"=>["Droonga::Plugins::Error::Adapter"]}, @body={"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.020", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores" =>["loca lhost:24224/starbucks"]}}, @task={"route"=>"localhost:24224/starbucks.020", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, @step={"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1} }}}, "ty pe"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}> -2014-02-17 16:25:23 +0900 [info]: Droonga::Plugins::SampleLoggerPlugin message=#<Droonga::HandlerMessage:0x007f9a7f069c50 @raw={"dataset"=>"Starbucks", "type"=>"search", "body"=>{"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores"=>["localhost:24224/star bucks"]} }, "replyTo"=>{"type"=>"search.result", "to"=>"127.0.0.1:50410/droonga"}, "id"=>"1392621923.903868", "date"=>"2014-02-17 16:25:23 +0900", "appliedAdapters"=>["Droonga::Plugins::Error::Adapter"]}, @body={"id"=>"localhost:24224/starbucks.#0", "task"=>{"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "stores" =>["loca lhost:24224/starbucks"]}}, @task={"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1}}}}, "type"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}, "n_of_inputs"=>0, "values"=>{}}, @step={"command"=>"search", "dataset"=>"Starbucks", "body"=>{"queries"=>{"stores"=>{"source"=>"Store", "condition"=>{"query"=>"Columbus", "matchTo"=>"_key"}, "output"=>{"elements"=>["startTime", "elapsedTime", "count", "attributes", "records"], "attributes"=>["_key"], "limit"=>-1} }}}, "ty pe"=>"broadcast", "outputs"=>["errors", "stores"], "replica"=>"random", "routes"=>["localhost:24224/starbucks.001", "localhost:24224/starbucks.011", "localhost:24224/starbucks.020"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#0"], "stores"=>["localhost:24224/starbucks.#0"]}}> + +Any record including the given keyword `"Broadway"` in its `"key"` is deleted from the storage of all partitions. + +Create a JSON file `delete-stores-broadway.json` with the content above. +We'll use it for testing. + +The response must have a boolean value to indicate "success" or "fail", like: + +~~~json +{ + "inReplyTo": "(message id)", + "statusCode": 200, + "type": "deleteStores.result", + "body": true +} ~~~ -Note that three lines are shown for only one request. What is happening? +If the request is successfully processed, the `body` becomes `true`. Otherwise `false`. +The `body` is just one boolean value, because we don't have to receive multiple results from partitions. -Remember that we have configured `Starbucks` dataset to use three partitions (and each has two replicas) in `catalog.json` of [the basic tutorial][basic]. -The `search` request is dispatched to three partitions and passed into handling phase for each partition. That is because we saw three lines for one request. +### Directory Structure -The messages shown is in internal format, which is transformed from the request you've sent. -You can see your search request is distributed to partitions `localhost:24224/starbucks.001`, `localhost:24224/starbucks.011` and `localhost:24224/starbucks.020` from `"routes"`. +Now let's create the `delete-stores` plugin, as the file `delete-stores.rb`. The directory tree will be: -In `search` case, it is enough to use one replica per one partition because replicas for a partition are expected to have the exactly same contents. -So the planner ordered distributor to choose one replica randomly. +~~~ +lib +└── droonga + └── plugins + └── delete-stores.rb +~~~ -## Trap "add" command +Then, create a skelton of a plugin as follows: -We have seen how distributed search is done from the view point of handling phase so far. -How about `"add"` command? +lib/droonga/plugins/delete-stores.rb: -Update `smaple-logger` plugin to trap `"add"` message instead of `"search"`. +~~~ruby +require "droonga/plugin" + +module Droonga + module Plugins + module DeleteStoresPlugin + Plugin.registry.register("delete-stores", self) + end + end +end +~~~ -lib/droonga/plugins/sample-logger.rb: +### Define a "step" for the command + +Define a "step" for the new `deleteStores` command, in your plugin. Like: + +lib/droonga/plugins/delete-stores.rb: + +~~~ruby +require "droonga/plugin" + +module Droonga + module Plugins + module DeleteStoresPlugin + Plugin.registry.register("delete-stores", self) + + define_single_step do |step| + step.name = "deleteStores" + step.write = true + end + end + end +end ~~~ + +Look at a new configuration `step.write`. +Because this command modifies the storage, we must indicate it clearly. + +### Define the handling logic + +Let's define the handler. + +lib/droonga/plugins/delete-stores.rb: + +~~~ruby require "droonga/plugin" module Droonga module Plugins - module SampleLoggerPlugin - Plugin.registry.register("sample-logger", self) + module DeleteStoresPlugin + Plugin.registry.register("delete-stores", self) + + define_single_step do |step| + step.name = "deleteStores" + step.write = true + step.handler = :Handler + step.collector = AndCollector + end class Handler < Droonga::Handler - message.type = "add" # This was "search" in the previous version. - - def handle(message, messenger) - $log.info "Droonga::Plugins::SampleLoggerPlugin", :message => message + def handle(message) + keyword = message["body"]["keyword"] + table = @context["Store"] + table.delete do |record| + record.key @ keyword + end + true end end end @@ -155,41 +447,70 @@ module Droonga end ~~~ -Restart `fluentd`: +The handler finds and deletes existing records which have the given keyword in its "key", by the [API of Rroonga][Groonga::Table_delete]. + +And, the `AndCollector` is bound to the step by the configuration `step.collector`. +It is is also one of built-in collectors, and merges boolean values retuned from handler instances for each partition and replica, to one boolean value. -~~~ -# kill $(cat fluentd.pid) -# RUBYLIB=./lib fluentd --config fluentd.conf --log fluentd.log --daemon fluentd.pid -~~~ -Let's send a request to Droonga Engine. -Here, we use the first line of `stores.json`. -add-store.json: +### Activate the plugin with `catalog.json` + +Update catalog.json to activate this plugin. +Add `"delete-stores"` to `"plugins"`. ~~~ -{"dataset":"Starbucks","type":"add","body":{"table":"Store","key":"1st Avenue & 75th St. - New York NY (W)","values":{"location":"40.770262,-73.954798"}}} +(snip) + "datasets": { + "Starbucks": { + (snip) + "plugins": ["delete-stores", "count-records", "groonga", "crud", "search"], +(snip) ~~~ -Send it to the engine: +### Run and test + +Restart the Droonga Engine and send the request. ~~~ -# droonga-request --tag starbucks add-store.json +# kill $(cat fluentd.pid) +# RUBYLIB=./lib fluentd --config fluentd.conf --log fluentd.log --daemon fluentd.pid +# droonga-request --tag starbucks count-records.json +Elapsed time: 0.01494 +[ + "droonga.message", + 1392621168, + { + "inReplyTo": "1392621168.0119512", + "statusCode": 200, + "type": "deleteStores.result", + "body": true + } +] ~~~ -You will see no output for `droonga-request` execution because out `sample-logger` plugin traps the `add` request. - -Instead, you will see results like this in `fluentd.log`: +Because results from partitions are unified to just one boolean value, the response's `body` is a `true`. +As the verification, send the request of `countRecords` command. ~~~ -2014-02-17 16:29:18 +0900 [info]: Droonga::Plugins::SampleLoggerPlugin message=#<Droonga::HandlerMessage:0x007f7f6a66c4c0 @raw={"dataset"=>"Starbucks", "type"=>"add", "body"=>{"id"=>"localhost:24224/starbucks.#2", "task"=>{"route"=>"localhost:24224/starbucks.000", "step"=>{"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "success"=>["localhost:24224/starbucks"]}}, "replyTo"=>{"type"=>"add.result", "to"=>"127.0.0.1:50480/droonga"}, "id" =>"13926 22158.374441", "date"=>"2014-02-17 16:29:18 +0900", "appliedAdapters"=>["Droonga::Plugins::CRUD::Adapter", "Droonga::Plugins::Error::Adapter"]}, @body={"id"=>"localhost:24224/starbucks.#2", "task"=>{"route"=>"localhost:24224/starbucks.000", "step"=>{"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "success"=>["localhost:24224/starbucks"]}}, @task={"route"=>"localhost:24224/starbucks.000", "step"=>{"command"=>"add", "dataset"=>"Starbuck s", "bod y"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, @step={"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:2 4224/sta rbucks.#2"]}}> -2014-02-17 16:29:18 +0900 [info]: Droonga::Plugins::SampleLoggerPlugin message=#<Droonga::HandlerMessage:0x007f7f6a65ff40 @raw={"dataset"=>"Starbucks", "type"=>"add", "body"=>{"id"=>"localhost:24224/starbucks.#2", "task"=>{"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "success"=>["localhost:24224/starbucks"]}}, "replyTo"=>{"type"=>"add.result", "to"=>"127.0.0.1:50480/droonga"}, "id" =>"13926 22158.374441", "date"=>"2014-02-17 16:29:18 +0900", "appliedAdapters"=>["Droonga::Plugins::CRUD::Adapter", "Droonga::Plugins::Error::Adapter"]}, @body={"id"=>"localhost:24224/starbucks.#2", "task"=>{"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, "descendants"=>{"errors"=>["localhost:24224/starbucks"], "success"=>["localhost:24224/starbucks"]}}, @task={"route"=>"localhost:24224/starbucks.001", "step"=>{"command"=>"add", "dataset"=>"Starbuck s", "bod y"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:24224/starbucks.#2"]}}, "n_of_inputs"=>0, "values"=>{}}, @step={"command"=>"add", "dataset"=>"Starbucks", "body"=>{"table"=>"Store", "key"=>"1st Avenue & 75th St. - New York NY (W)", "values"=>{"location"=>"40.770262,-73.954798"}}, "key"=>"1st Avenue & 75th St. - New York NY (W)", "type"=>"scatter", "outputs"=>["errors", "success"], "replica"=>"all", "post"=>true, "routes"=>["localhost:24224/starbucks.000", "localhost:24224/starbucks.001"], "n_of_expects"=>0, "descendants"=>{"errors"=>["localhost:24224/starbucks.#2"], "success"=>["localhost:2 4224/sta rbucks.#2"]}}> +# droonga-request --tag starbucks count-records.json +Elapsed time: 0.01494 +[ + "droonga.message", + 1392621168, + { + "inReplyTo": "1392621168.0119512", + "statusCode": 200, + "type": "countRecords.result", + "body": [8, 8, 7] + } +] ~~~ -In `add` case, two log lines are shown for one request. This is because we have configured to have two replicas for each partition. +Note, the number of records are smaller than the previous result. +This means that 4 or some records are deleted from each partitions. + -In order to be consistent, `add` command must reach all of the replicas of the partition, but not the other partitions. -As a consequence, `localhost:24224/starbucks.000` and `localhost:24224/starbucks.001` are chosen. ## まとめ @@ -199,3 +520,5 @@ We have learned how to create plugins work in handling phrase. [adapter]: ../adapter [basic]: ../basic + [Groonga::Context]: http://ranguba.org/rroonga/en/Groonga/Context.html + [Groonga::Table_delete]: http://ranguba.org/rroonga/en/Groonga/Table.html#delete-instance_method Modified: ja/tutorial/plugin-development/index.md (+4 -3) =================================================================== --- ja/tutorial/plugin-development/index.md 2014-02-27 17:45:07 +0900 (9188324) +++ ja/tutorial/plugin-development/index.md 2014-02-27 18:00:54 +0900 (ff4a952) @@ -66,9 +66,10 @@ Following this tutorial, you will learn how to write plugins. This will be the f 詳細は以下のサブチュートリアルを参照してください: - 1. [リクエストとレスポンスの加工][adapter] - 2. [リクエストの処理][handler] (作成中) - 3. リクエストの分散とレスポンスの回収 (作成中) + 1. [Modify requests and responses][adapter] + 2. [Handle requests on all partitions][handler] + 3. Handle requests only on a specific partition (under construction) + 4. Distribute requests and collect responses (under construction) [basic tutorial]: ../basic/ -------------- next part -------------- HTML����������������������������...Download