Kouhei Sutou
null+****@clear*****
Mon Oct 7 18:40:21 JST 2013
Kouhei Sutou 2013-10-07 18:40:21 +0900 (Mon, 07 Oct 2013) New Revision: ac458a72561845abaeeb6712fe1643db9eb7439b https://github.com/groonga/groonga/commit/ac458a72561845abaeeb6712fe1643db9eb7439b Message: doc: describe about tuning parameters TODO: * Update file list * Translate Added files: doc/source/reference/tuning.txt Modified files: doc/source/reference.txt Modified: doc/source/reference.txt (+1 -0) =================================================================== --- doc/source/reference.txt 2013-10-07 11:56:57 +0900 (3ebaf3e) +++ doc/source/reference.txt 2013-10-07 18:40:21 +0900 (e868d87) @@ -21,4 +21,5 @@ reference/function reference/indexing reference/log + reference/tuning reference/api Added: doc/source/reference/tuning.txt (+176 -0) 100644 =================================================================== --- /dev/null +++ doc/source/reference/tuning.txt 2013-10-07 18:40:21 +0900 (df3ad6f) @@ -0,0 +1,176 @@ +.. -*- rst -*- + +.. highlightlang:: none + +Tuning +====== + +Summary +------- + +There are some tuning parameters for improving groonga performance or +handling a large database. + +Parameters +---------- + +This section describes parameters. + +.. _tuning-max-n-open-files: + +The max number of open files per process +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This parameter is for handling a large database. + +Groonga creates one or more files per table and colum. If your +database has many tables and columns, groonga process needs to open +many files. + +System limits the max number of open files per process. So you need to +relax the limitation. + +Here is an expression that compute how many files are opened by +groonga:: + + 3 (for DB) + + N tables + + N columns (except index clumns) + + (N index columns * 2) + + X (the number of plugins etc.) + +Here is an example schema:: + + table_create Entries TABLE_HASH_KEY ShortText + column_create Entries content COLUMN_SCALAR Text + column_create Entries n_likes COLUMN_SCALAR UInt32 + table_create Terms TABLE_PAT_KEY|KEY_NORMALIZE ShortText --default_tokenizer TokenBigram + column_create Terms entries_key_index COLUMN_INDEX|WITH_POSITION Entries _key + column_create Terms entries_content_index COLUMN_INDEX|WITH_POSITION Entries content + +This example opens at least 11 files:: + + 3 + + 2 (Entries and Terms) + + 2 (Entries.content and Entries.n_likes) + + 4 (Terms.entries_key_index and Terms.entries_content_index) + + X = 11 + X + +.. _tuning-memory-usage: + +Memory usage +^^^^^^^^^^^^ + +This parameter is for handling a large database. + +Groonga maps database files onto memory and accesses to them. Groonga +doesn't maps unnecessary files onto memory until they are nneded. + +If you access to all data in database, all database files are mapped +onto memory. If total size of your database files is 6GiB, your +groonga process uses 6GiB memory. + +Normally, your all database files aren't mapped onto memry. But is may +be occurred. It is an example case that you dump your database. + +You must have memory and swap that is larger than database. + +.. _tuning-linux: + +Linux +----- + +This section describes how to configure parameters on Linux. + +.. _tuning-linux-nofile: + +nofile +^^^^^^ + +You can relax the :ref:`tuning-max-n-open-files` parameter by creating +a configuration file ``/etc/security/limits.d/groonga.conf`` that has +the following content:: + + ${USER} soft nofile ${MAX_VALUE} + ${USER} hard nofile ${MAX_VALUE} + +If you run ``groonga`` process by ``groonga`` user and your groonga +process needs to open less than 10000 files, use the following +configuration:: + + groonga soft nofile 10000 + groonga hard nofile 10000 + +The configuration is applied after your groonga service is restarted +or re-login as your groonga user. + +.. _tuning-linux-overcommit-memory: + +vm.overcommit_memory +^^^^^^^^^^^^^^^^^^^^ + +This is :ref:`tuning-memory-usage` related parameter. You can handle a +database that is larger than your memory and swap by setting +``vm.overcommit_memory`` kernel parameter to 1. 1 means that groonga +can always map database files onto memory. It is no problem until +groonga touch mapped database files that their size is larger than +memory and swap. Groonga recommends the configuration. + +See `Linux kernel documentaion about overcommit +<https://www.kernel.org/doc/Documentation/vm/overcommit-accounting>`_ +about ``vm.overcommit_memory`` parameter details. + +You can set the configuration by putting a configuration file +``/etc/sysctl.d/groonga.conf`` that has the following content:: + + vm.overcommit_memory = 1 + +The configuration can be applied by restarting your system or run the +following command:: + + % sudo sysctl -p + + +.. _tuning-linux-max-map-count: + +vm.max_map_count +^^^^^^^^^^^^^^^^ + +This is :ref:`tuning-memory-usage` related parameter. You can handle a +16GiB or more larger size database by increasing ``vm.max_map_count`` +kernel parameter. The parameter limits the max number of memory maps. + +The default value of the kernel parameter may be 65530 or 65536. +Groonga maps 256KiB memory chunk at one time. If a database is larger +than 16GiB, groonga reaches the limitation. (``256KiB * 65536 = 16GiB``) + +You needs to increase the value of the kernel parameter to handle +16GiB or more larger size database. For example, you can handle almost +32GiB size database by ``65536 * 2 = 131072``. You can set the +configuration by putting a configuration file +``/etc/sysctl.d/groonga.conf`` that has the following content:: + + vm.max_map_count = 131072 + +Note that your real confiugration file will be the following because +you already have ``vm.overcommit_memory`` configuration:: + + vm.overcommit_memory = 1 + vm.max_map_count = 131072 + +The configuration can be applied by restarting your system or run the +following command:: + + % sudo sysctl -p + +FreeBSD +------- + +This section describes how to configure parameters on FreeBSD. + +.. _tuning-freebsd-maxfilesperproc: + +kern.maxfileperproc +^^^^^^^^^^^^^^^^^^^ + +TODO -------------- next part -------------- HTML����������������������������...Download