Usage

This document describes how to use ukt with Kyoto Tycoon.

Features

This section describes basic features and APIs of the KyotoTycoon client. For simplicity, we’ll use the EmbeddedServer, which sets up the database server in a subprocess and makes it easy to develop.

>>> from ukt import EmbeddedServer
>>> server = EmbeddedServer()
>>> server.run()  # Starts "ktserver" in a subprocess.
True
>>> client = server.client  # Get a client for use with our embedded server.

As you would expect for a key/value database, the client implements get(), set() and remove():

>>> client.set('k1', 'v1')
1
>>> client.get('k1')
'v1'
>>> client.remove('k1')
1

It is not an error to try to get or delete a key that doesn’t exist:

>>> client.get('not-here')  # Returns None
>>> client.remove('not-here')
0

To check whether a key exists we can use exists():

>>> client.set('k1', 'v1')
>>> client.exists('k1')
True
>>> client.exists('not-here')
False

In addition, there are also efficient methods for bulk operations: get_bulk(), set_bulk() and remove_bulk():

>>> client.set_bulk({'k1': 'v1', 'k2': 'v2', 'k3': 'v3'})
3
>>> client.get_bulk(['k1', 'k2', 'k3', 'not-here'])
{'k1': 'v1', 'k2': 'v2', 'k3': 'v3'}
>>> client.remove_bulk(['k1', 'k2', 'k3', 'not-here'])
3

The client library also supports a dict-like interface:

>>> client['k1'] = 'v1'
>>> print(client['k1'])
v1
>>> del client['k1']
>>> client.update({'k1': 'v1', 'k2': 'v2', 'k3': 'v3'})
3
>>> client.pop('k1')
'v1'
>>> client.pop('k1')  # Returns None
>>> 'k1' in client
False
>>> len(client)
2

To remove all records, you can use the clear() method:

>>> client.clear()
True

Serialization

By default the client will assume that keys and values should be encoded as UTF-8 byte-strings and decoded to unicode upon retrieval. You can set the serializer parameter when creating your client to use a different value serialization. ukt provides the following:

  • KT_BINARY - default, treat values as unicode and serialize as UTF-8.
  • KT_JSON - use JSON to serialize values.
  • KT_MSGPACK - use msgpack to serialize values.
  • KT_PICKLE - use pickle to serialize values.
  • KT_NONE - no serialization, values must be bytestrings.

For example, to use the pickle serializer:

>>> from ukt import KT_PICKLE, KyotoTycoon
>>> client = KyotoTycoon(serializer=KT_PICKLE)
>>> client.set('k1', {'this': 'is', 'a': ['python object']})
1
>>> client.get('k1')
{'this': 'is', 'a': ['python object']}

Kyoto Tycoon

The Kyoto Tycoon section continues from the previous section, and assumes that you are running an EmbeddedServer and accessing it through it’s client property.

Database filenames

Kyoto Tycoon determines the database type by looking at the filename of the database(s) specified when ktserver is executed. Additionally, for in-memory databases, you use special symbols instead of filenames.

  • hash_table.kch - on-disk hash table (“kch”).
  • btree.kct - on-disk b-tree (“kct”).
  • dirhash.kcd - directory hash (“kcd”).
  • dirtree.kcf - directory b-tree (“kcf”).
  • * - cache-hash, in-memory hash-table with LRU deletion.
  • % - cache-tree, in-memory b-tree (ordered cache).
  • : - stash db, in-memory database with lower memory usage.
  • - - prototype hash, simple in-memory hash using std::unordered_map.
  • + - prototype tree, simple in-memory hash using std::map (ordered).

Generally:

  • For unordered collections, use either the cache-hash (*) or the file-hash (.kch).
  • For ordered collections or indexes, use either the cache-tree (%) or the file b-tree (.kct).
  • I avoid the prototype hash and btree as the entire data-structure is locked during writes (as opposed to an individual record or page).

For more information about the above database types, their algorithmic complexity, and the unit of locking, see kyotocabinet db chart.

Key Expiration

Kyoto Tycoon servers feature a built-in expiration mechanism, allowing you to use it as a cache. Whenever setting a value or otherwise writing to the database, you can also specify an expiration time (in seconds):

>>> client.set('k1', 'v1', expire_time=5)
>>> client.get('k1')
'v1'
>>> time.sleep(5)
>>> client.get('k1')  # Returns None

Expiration time can be specified in the following ways:

  • Integers less than 6 months (in seconds) are treated as relative timestamps.
  • Integers larger are treated as unix timestamps.
  • datetime.datetime objects specify the expire time.
  • datetime.timedelta objects specify a the time relative to now.

Multiple Databases

Kyoto Tycoon can also be used as the front-end for multiple databases. For example, to start ktserver with an in-memory hash-table and an in-memory b-tree, you would run:

$ ktserver \* \%

By default, the KyotoTycoon client assumes you are working with the first database (starting from zero, our hash-table would be 0 and the b-tree would be 1).

The client can be initialized to use a different database by default:

>>> client = KyotoTycoon(default_db=1)

To change the default database at run-time, you can call the set_database() method:

>>> client = KyotoTycoon()
>>> client.set_database(1)

Lastly, to perform a one-off operation against a specific database, all methods accept a db parameter which you can use to specify the database:

>>> client.set('k1', 'v1', db=1)
>>> client.get('k1', db=0)  # Returns None
>>> client.get('k1', db=1)
'v1'

Similarly, if a tuple is passed into the dictionary APIs, it is assumed that the key consists of (key, db) and the value of (value, expire):

>>> client['k1', 1] = 'v1'  # Set k1=v1 in db1.
>>> client['k1', 1]
'v1'
>>> client['k2'] = ('v2', 10)  # Set k2=v2 in default db with 10s expiration.
>>> client['k2', 0] = ('v2', 20)  # Set k2=v2 in db0 with 20s expiration.
>>> del client['k1', 1]  # Delete 'k1' in db1.

Lua Scripts

Kyoto Tycoon can be scripted using lua. To run a Lua script from the client, you can use the script() method. In Kyoto Tycoon, a script may receive arbitrary key/value-pairs as parameters, and may return arbitrary key/value pairs:

>>> client.script('myfunction', {'key': 'some-key', 'data': 'etc'})
{'data': 'returned', 'by': 'user-script'}

To simplify script execution, you can also use the lua() helper, which provides a slightly more Pythonic API:

>>> lua = client.lua
>>> lua.myfunction(key='some-key', data='etc')
{'data': 'returned', 'by': 'user-script'}
>>> lua.another_function(key='another-key')
{}

Learn more about scripting Kyoto Tycoon by reading the lua doc.