Overall Table of Contents

KInterbasDB Usage Guide

(Last updated 2006.09.08 at 16:35 UTC)

Contents
Introduction (Propaganda)

The Firebird relational database engine has a large feature set, conforms closely to SQL standards, and is flexible enough to operate either as a standalone server or as an embedded library on diverse platforms. In spite of this versatility, the database is easy to use--almost self-managing.

The Python programming language supports numerous paradigms, is suitable for constructing both small and large programs, and integrates well with native C and C++ libraries. Despite the versatility of the language, well written Python code achieves an exceptional lucidity that has led some to call the language "executable pseudocode".

These two top-flight software tools intersect in a library named KInterbasDB. KInterbasDB implements Python's standard Database API 2.0, but also extends far beyond, to cover almost all of Firebird's extensive native client API. KInterbasDB strives to deliver the power of Firebird into the hands of the Python programmer without compromising the qualities of either tool.

This Usage Guide is not a tutorial on Python, SQL, or Firebird; rather, it is a topical presentation of KInterbasDB's feature set, with example code to demonstrate basic usage patterns. This guide is meant to be consumed in conjunction with the Python Database API Specification and the Firebird documentation, especially the professional, seven-volume manual for Firebird's commercial ancestor, Interbase®.

The table of contents presents a structural overview of this document.



Python Database API 2.0 Compliance

Incompatibilities

Unsupported Optional Features

Nominally Supported Optional Features

Extensions and Caveats

KInterbasDB offers a large feature set beyond the minimal requirements of the Python DB API. Most of these extensions are documented in the section of this document entitled Native Database Engine Features and Extensions Beyond the Python DB API.

This section attempts to document only those features that overlap with the DB API, or are too insignificant to warrant their own subsection elsewhere.

Tutorial

This brief tutorial aims to get the reader started by demonstrating elementary usage of KInterbasDB. It is not a comprehensive Python Database API tutorial, nor is it comprehensive in its coverage of anything else.

The numerous advanced features of KInterbasDB are covered in another section of this document, which is not in a tutorial format, though it is replete with examples.

Connecting to a Database

Example 1

A database connection is typically established with code such as this: 

import kinterbasdb

# The server is named 'bison'; the database file is at '/temp/test.db'.
con = kinterbasdb.connect(dsn='bison:/temp/test.db', user='sysdba', password='pass')

# Or, equivalently:
con = kinterbasdb.connect(
    host='bison', database='/temp/test.db',
    user='sysdba', password='pass'
  )

Example 2

Suppose we want to connect to an Interbase® 5.5 server, specifying UNICODE_FSS as the character set of the connection: 

import kinterbasdb

con = kinterbasdb.connect(
    dsn='bison:/temp/test.db',
    user='sysdba', password='pass',
    dialect=1, # necessary for Interbase® < 6.0
    charset='UNICODE_FSS' # specify a character set for the connection
  )


Executing SQL Statements

For this section, suppose we have a table defined and populated by the following SQL code: 

create table languages
(
  name               varchar(20),
  year_released      integer
);

insert into languages (name, year_released) values ('C',        1972);
insert into languages (name, year_released) values ('Python',   1991);

Example 1

This example shows the simplest way to print the entire contents of the languages table: 

import kinterbasdb

con = kinterbasdb.connect(dsn='/temp/test.db', user='sysdba', password='masterkey')

# Create a Cursor object that operates in the context of Connection con:
cur = con.cursor()

# Execute the SELECT statement:
cur.execute("select * from languages order by year_released")

# Retrieve all rows as a sequence and print that sequence:
print cur.fetchall()

Sample output: 

[('C', 1972), ('Python', 1991)]

Example 2

Here's another trivial example that demonstrates various ways of fetching a single row at a time from a SELECT-cursor: 

import kinterbasdb

con = kinterbasdb.connect(dsn='/temp/test.db', user='sysdba', password='masterkey')

cur = con.cursor()
SELECT = "select name, year_released from languages order by year_released"

# 1. Iterate over the rows available from the cursor, unpacking the
# resulting sequences to yield their elements (name, year_released):
cur.execute(SELECT)
for (name, year_released) in cur:
    print '%s has been publicly available since %d.' % (name, year_released)

# 2. Equivalently:
cur.execute(SELECT)
for row in cur:
    print '%s has been publicly available since %d.' % (row[0], row[1])

# 3. Using mapping-iteration rather than sequence-iteration:
cur.execute(SELECT)
for row in cur.itermap():
    print '%(name)s has been publicly available since %(year_released)d.' % row

Sample output: 

C has been publicly available since 1972.
Python has been publicly available since 1991.
C has been publicly available since 1972.
Python has been publicly available since 1991.
C has been publicly available since 1972.
Python has been publicly available since 1991.

Example 3

The following program is a simplistic table printer (applied in this example to languages): 

import kinterbasdb as k

TABLE_NAME = 'languages'
SELECT = 'select * from %s order by year_released' % TABLE_NAME

con = k.connect(dsn='/temp/test.db', user='sysdba', password='masterkey')

cur = con.cursor()
cur.execute(SELECT)

# Print a header.
for fieldDesc in cur.description:
    print fieldDesc[k.DESCRIPTION_NAME].ljust(fieldDesc[k.DESCRIPTION_DISPLAY_SIZE]) ,
print # Finish the header with a newline.
print '-' * 78

# For each row, print the value of each field left-justified within
# the maximum possible width of that field.
fieldIndices = range(len(cur.description))
for row in cur:
    for fieldIndex in fieldIndices:
        fieldValue = str(row[fieldIndex])
        fieldMaxWidth = cur.description[fieldIndex][k.DESCRIPTION_DISPLAY_SIZE]

        print fieldValue.ljust(fieldMaxWidth) ,

    print # Finish the row with a newline.

Sample output: 

NAME                 YEAR_RELEASED
------------------------------------------------------------------------------
C                    1972
Python               1991

Example 4

Let's insert more languages: 

import kinterbasdb

con = kinterbasdb.connect(dsn='/temp/test.db', user='sysdba', password='masterkey')

cur = con.cursor()

newLanguages = [
    ('Lisp',  1958),
    ('Dylan', 1995),
  ]

cur.executemany("insert into languages (name, year_released) values (?, ?)",
    newLanguages
  )

# The changes will not be saved unless the transaction is committed explicitly:
con.commit()

Note the use of a parameterized SQL statement above. When dealing with repetitive statements, this is much faster and less error-prone than assembling each SQL statement manually. (You can read more about parameterized SQL statements in the section on Prepared Statements.)

After running Example 4, the table printer from Example 3 would print: 

NAME                 YEAR_RELEASED
------------------------------------------------------------------------------
Lisp                 1958
C                    1972
Python               1991
Dylan                1995

Calling Stored Procedures

Interbase® and Firebird support stored procedures written in a proprietary procedural SQL language. IB/FB stored procedures can have input parameters and/or output parameters. Some databases support input/output parameters, where the same parameter is used for both input and output; IB/FB does not support this.

It is important to distinguish between procedures that return a result set and procedures that populate and return their output parameters exactly once. Conceptually, the latter "return their output parameters" like a Python function, whereas the former "yield result rows" like a Python generator.

IB/FB's server-side procedural SQL syntax makes no such distinction, but client-side SQL code (and C API code) must. A result set is retrieved from a stored procedure by SELECTing from the procedure, whereas output parameters are retrieved with an EXECUTE PROCEDURE statement.

To retrieve a result set from a stored procedure with KInterbasDB, use code such as this: 

cur.execute("select output1, output2 from the_proc(?, ?)", (input1, input2))

# Ordinary fetch code here, such as:
for row in cur:
    ... # process row

con.commit() # If the procedure had any side effects, commit them.

To execute a stored procedure and access its output parameters, use code such as this: 

cur.callproc("the_proc", (input1, input2))

# If there are output parameters, retrieve them as though they were the
# first row of a result set.  For example:
outputParams = cur.fetchone()

con.commit() # If the procedure had any side effects, commit them.

This latter is not very elegant; it would be preferable to access the procedure's output parameters as the return value of Cursor.callproc. The Python DB API specification requires the current behavior, however.




Native Database Engine Features and Extensions Beyond the Python DB API

Programmatic Database Creation and Deletion

The Firebird engine stores a database in a fairly straightforward manner: as a single file or, if desired, as a segmented group of files.

The engine supports dynamic database creation via the SQL statement CREATE DATABASE, which is documented on page 49 of the Interbase® 6 Language Reference.

The engine also supports dropping (deleting) databases dynamically, but dropping is a more complicated operation than creating, for several reasons: an existing database may be in use by users other than the one who requests the deletion, it may have supporting objects such as temporary sort files, and it may even have dependent shadow databases. Although the database engine recognizes a DROP DATABASE SQL statement, support for that statement is limited to the isql command-line administration utility. However, the engine supports the deletion of databases via an API call, which KInterbasDB exposes to Python (see below).

KInterbasDB supports dynamic database creation and deletion via the module-level function create_database and the method Connection.drop_database. These are documented below, then demonstrated by a brief example.

create_database  (function; member of kinterbasdb)

Creates a database according to the supplied CREATE DATABASE SQL statement. Returns an open connection to the newly created database.

Arguments:

  • sql - string containing the CREATE DATABASE statement.

    Note that this statement may need to include a username and password (see the IB 6 Language Reference for syntax).

  • dialect (optional) - the SQL dialect under which to execute the statement (defaults to 3).
drop_database  (method; member of kinterbasdb.Connection)

Deletes the database to which the connection is attached.

This method performs the database deletion in a responsible fashion. Specifically, it:

  • raises an OperationalError instead of deleting the database if there are other active connections to the database
  • deletes supporting files and logs in addition to the primary database file(s)

This method has no arguments.

Example program: 

import kinterbasdb

con = kinterbasdb.create_database(
    "create database '/temp/db.db' user 'sysdba' password 'pass'"
  )
con.drop_database()

Database Event Notification

What are database events?

The database engine features a distributed, interprocess communication mechanism based on messages called database events. Chapter 11 of the Interbase® 6 API Guide describes database events this way:

[A database event is] a message passed from a trigger or stored procedure to an application to announce the occurrence of a specified condition or action, usually a database change such as an insertion, modification, or deletion of a record.
The Interbase® [and Firebird] event mechanism enables applications to respond to actions and database changes made by other, concurrently running applications without the need for those applications to communicate directly with one another, and without incurring the expense of CPU time required for periodic polling to determine if an event has occurred.

Why use database events?

Anything that can be accomplished with database events can also be implemented using other techniques, so why bother with events? Since you've chosen to write database-centric programs in Python rather than assembly language, you probably already know the answer to this question, but let's illustrate.

A typical application for database events is the handling of administrative messages. Suppose you have an administrative message database with a messages table, into which various applications insert timestamped status reports. It may be desirable to react to these messages in diverse ways, depending on the status they indicate: to ignore them, to initiate the update of dependent databases upon their arrival, to forward them by e-mail to a remote administrator, or even to set off an alarm so that on-site administrators will know a problem has occurred.

It is undesirable to tightly couple the program whose status is being reported (the message producer) to the program that handles the status reports (the message handler). There are obvious losses of flexibility in doing so. For example, the message producer may run on a separate machine from the administrative message database and may lack access rights to the downstream reporting facilities (e.g., network access to the SMTP server, in the case of forwarded e-mail notifications). Additionally, the actions required to handle status reports may themselves be time-consuming and error-prone, as in accessing a remote network to transmit e-mail.

In the absence of database event support, the message handler would probably be implemented via polling. Polling is simply the repetition of a check for a condition at a specified interval. In this case, the message handler would check in an infinite loop to see whether the most recent record in the messages table was more recent than the last message it had handled. If so, it would handle the fresh message(s); if not, it would go to sleep for a specified interval, then loop.

The polling-based implementation of the message handler is fundamentally flawed. Polling is a form of busy-wait; the check for new messages is performed at the specified interval, regardless of the actual activity level of the message producers. If the polling interval is lengthy, messages might not be handled within a reasonable time period after their arrival; if the polling interval is brief, the message handler program (and there may be many such programs) will waste a large amount of CPU time on unnecessary checks.

The database server is necessarily aware of the exact moment when a new message arrives. Why not let the message handler program request that the database server send it a notification when a new message arrives? The message handler can then efficiently sleep until the moment its services are needed. Under this event-based scheme, the message handler becomes aware of new messages at the instant they arrive, yet it does not waste CPU time checking in vain for new messages when there are none available.

How does the database engine expose events to SQL (in the server process) and C (in the client process)?

  1. Server Process ("An event just occurred!")

    Recall from Chapter 11 of the Interbase® 6 API Guide that

    [A database event is] a message passed from a trigger or stored procedure to an application to announce the occurrence of a specified condition or action, usually a database change such as an insertion, modification, or deletion of a record.

    To notify any interested listeners that a specific event has occurred, issue the POST_EVENT statement (see page 176 of the Interbase® 6 Language Reference). The POST_EVENT statement has one parameter: the name of the event to post.

    In the preceding example of the administrative message database, POST_EVENT might be used from an after insert trigger on the messages table, like this: 

    create trigger trig_messages_handle_insert
  for messages
    after insert
as
begin
  POST_EVENT 'new_message';
end

    Note that the physical notification of the client process does not occur until the transaction in which the POST_EVENT took place is actually committed. Therefore, multiple events may conceptually occur before the client process is physically informed of even one occurrence.

    Furthermore, the database engine makes no guarantee that clients will be informed of events in the same groupings in which they conceptually occurred. If, within a single transaction, an event named event_a is posted once and an event named event_b is posted once, the client may receive those posts in separate "batches", despite the fact that they occurred in the same conceptual unit (a single transaction). This also applies to multiple occurrences of the same event within a single conceptual unit: the physical notifications may arrive at the client separately.

  2. Client Process ("Send me a message when an event occurs.")

    Note: If you don't care about the gory details of event notification, skip to the section that describes KInterbasDB's Python-level event handling API.

    The Interbase®/Firebird C client library offers two forms of event notification.

    The first form is synchronous notification, by way of the function isc_wait_for_event. This form is admirably simple for a C programmer to use, but is inappropriate as a basis for KInterbasDB's event support, chiefly because it's not sophisticated enough to serve as the basis for a comfortable Python-level API.

    The other form of event notification offered by the database client library is asynchronous, by way of the functions isc_que_events (note that the name of that function is misspelled), isc_cancel_events, and others.

    The details are as nasty as they are numerous, but the essence of using asynchronous notification from C is as follows:

    1. Call isc_event_block to create a formatted binary buffer that will tell the server which events the client wants to listen for.
    2. Call isc_que_events (passing the buffer created in the previous step) to inform the server that the client is ready to receive event notifications, and provide a callback that will be asynchronously invoked when one or more of the registered events occurs.
    3. [The thread that called isc_que_events to initiate event listening must now do something else.]
    4. When the callback is invoked (the database client library starts a thread dedicated to this purpose), it can use the isc_event_counts function to determine how many times each of the registered events has occurred since the last call to isc_event_counts (if any).
    5. [The callback thread should now "do its thing", which may include communicating with the thread that called isc_que_events.]
    6. When the callback thread is finished handling an event notification, it must call isc_que_events again in order to receive future notifications. Future notifications will invoke the callback again, effectively "looping" the callback thread back to Step 4.

How does KInterbasDB expose database events to the Python programmer?

The KInterbasDB database event API is comprised of the following: the method Connection.event_conduit and the class EventConduit.

event_conduit  (method; member of kinterbasdb.Connection)

Creates a conduit (an instance of EventConduit) through which database event notifications will flow into the Python program.

event_conduit is a method of Connection rather than a module-level function or a class constructor because the database engine deals with events in the context of a particular database (after all, POST_EVENT must be issued by a stored procedure or a trigger).

Arguments:

  • event_names - a sequence of string event names

    The EventConduit.wait method will block until the occurrence of at least one of the events named by the strings in event_names.

    KInterbasDB's own event-related code is capable of operating with up to 2147483647 events per conduit. However, it has been observed that the Firebird client library experiences catastrophic problems (including memory corruption) on some platforms with anything beyond about 100 events per conduit. These limitations are dependent on both the Firebird version and the platform.

EventConduit:

__init__  (method; member of kinterbasdb.EventConduit)

The EventConduit class is not designed to be instantiated directly by the Python programmer. Instead, use the Connection.event_conduit method to create EventConduit instances.

wait  (method; member of kinterbasdb.EventConduit)

Blocks the calling thread until at least one of the events occurs, or the specified timeout (if any) expires.

If one or more event notifications has arrived since the last call to wait, this method will retrieve a notification from the head of the EventConduit's internal queue and return immediately.

The names of the relevant events were supplied to the Connection.event_conduit method during the creation of this EventConduit. In the code snippet below, the relevant events are named event_a and event_b: 

conduit = connection.event_conduit( ('event_a', 'event_b') )
conduit.wait()

Arguments:

  • timeout (optional) - number of seconds (use a float to indicate fractions of seconds)

    If not even one of the relevant events has occurred after timeout seconds, this method will unblock and return None. The default timeout is infinite.

Returns:

None if the wait timed out, otherwise a dictionary that maps event_name -> event_occurrence_count.

In the code snippet above, if event_a occurred once and event_b did not occur at all, the return value from conduit.wait() would be the following dictionary: 

{
  'event_a': 1,
  'event_b': 0
}
close  (method; member of kinterbasdb.EventConduit)

Cancels the standing request for this conduit to be notified of events.

After this method has been called, this EventConduit object is useless, and should be discarded. (The boolean property closed is True after an EventConduit has been closed.)

This method has no arguments.

flush  (method; member of kinterbasdb.EventConduit)

This method allows the Python programmer to manually clear any event notifications that have accumulated in the conduit's internal queue.

From the moment the conduit is created by the Connection.event_conduit method, notifications of any events that occur will accumulate asynchronously within the conduit's internal queue until the conduit is closed either explicitly (via the close method) or implicitly (via garbage collection). There are two ways to dispose of the accumulated notifications: call wait to receive them one at a time (wait will block when the conduit's internal queue is empty), or call this method to get rid of all accumulated notifications.

This method has no arguments.

Returns:

The number of event notifications that were flushed from the queue. The "number of event notifications" is not necessarily the same as the "number of event occurrences", since a single notification can indicate multiple occurrences of a given event (see the return value of the wait method).



Example Program

The following code (a SQL table definition, a SQL trigger definition, and two Python programs) demonstrates KInterbasDB-based event notification.

The example is based on a database at 'localhost:/temp/test.db', which contains a simple table named test_table.  test_table has an after insert trigger that posts several events. Note that the trigger posts test_event_a twice, test_event_b once, and test_event_c once.

The Python event handler program connects to the database and establishes an EventConduit in the context of that connection. As specified by the list of RELEVANT_EVENTS passed to event_conduit, the event conduit will concern itself only with events named test_event_a and test_event_b. Next, the program calls the conduit's wait method without a timeout; it will wait infinitely until at least one of the relevant events is posted in a transaction that is subsequently committed.

The Python event producer program simply connects to the database, inserts a row into test_table, and commits the transaction. Notice that except for the printed comment, no code in the producer makes any mention of events--the events are posted as an implicit consequence of the row's insertion into test_table.

The insertion into test_table causes the trigger to conceptually post events, but those events are not physically sent to interested listeners until the transaction is committed. When the commit occurs, the handler program returns from the wait call and prints the notification that it received.

SQL table definition: 

create table test_table (a integer)

SQL trigger definition: 

create trigger trig_test_insert_event
  for test_table
    after insert
as
begin
  post_event 'test_event_a';
  post_event 'test_event_b';
  post_event 'test_event_c';

  post_event 'test_event_a';
end

Python event handler program: 

import kinterbasdb

RELEVANT_EVENTS = ['test_event_a', 'test_event_b']

con = kinterbasdb.connect(dsn='localhost:/temp/test.db', user='sysdba', password='pass')
conduit = con.event_conduit(RELEVANT_EVENTS)

print 'HANDLER: About to wait for the occurrence of one of %s...\n' % RELEVANT_EVENTS
result = conduit.wait()
print 'HANDLER: An event notification has arrived:'
print result
conduit.close()

Python event producer program: 

import kinterbasdb

con = kinterbasdb.connect(dsn='localhost:/temp/test.db', user='sysdba', password='pass')
cur = con.cursor()

cur.execute("insert into test_table values (1)")
print 'PRODUCER: Committing transaction that will cause event notification to be sent.'
con.commit()

Event producer output: 

PRODUCER: Committing transaction that will cause event notification to be sent.

Event handler output (assuming that the handler was already started and waiting when the event producer program was executed): 

HANDLER: About to wait for the occurrence of one of ['test_event_a', 'test_event_b']...

HANDLER: An event notification has arrived:
{'test_event_a': 2, 'test_event_b': 1}

Notice that there is no mention of test_event_c in the result dictionary received by the event handler program. Although test_event_c was posted by the after insert trigger, the event conduit in the handler program was created to listen only for test_event_a and test_event_b events.


Pitfalls and Limitations

Note: The restrictions on the number of active EventConduits in a process, and on the number of event names that a single EventConduit can listen for, have been removed in KInterbasDB 3.2.



Connection Timeouts

Connection timeouts allow the programmer to request that a connection be automatically closed after a specified period of inactivity. The simplest uses of connection timeouts are trivial, as demonstrated by the following snippet: 

import kinterbasdb

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={'period': 120.0} # time out after 120.0 seconds of inactivity
  )

...

The connection created in the example above is eligible to be automatically closed by KInterbasDB if it remains idle for at least 120.0 consecutive seconds. KInterbasDB does not guarantee that the connection will be closed immediately when the specified period has elapsed. On a busy system, there might be a considerable delay between the moment a connection becomes eligible for timeout and the moment KInterbasDB actually closes it. However, the thread that performs connection timeouts is programmed in such a way that on a lightly loaded system, it acts almost instantaneously to take advantage of a connection's eligibility for timeout.

After a connection has timed out, KInterbasDB reacts to attempts to reactivate the severed connection in a manner dependent on the state of the connection when it timed out. Consider the following example program: 

import time
import kinterbasdb

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={'period': 3.0}
  )
cur = con.cursor()

cur.execute("recreate table test (a int, b char(1))")
con.commit()

cur.executemany("insert into test (a, b) values (?, ?)",
    [(1, 'A'), (2, 'B'), (3, 'C')]
  )
con.commit()

cur.execute("select * from test")
print 'BEFORE:', cur.fetchall()

cur.execute("update test set b = 'X' where a = 2")

time.sleep(6.0)

cur.execute("select * from test")
print 'AFTER: ', cur.fetchall()

So, should the example program print 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]
AFTER:  [(1, 'A'), (2, 'X'), (3, 'C')]

or 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]
AFTER:  [(1, 'A'), (2, 'B'), (3, 'C')]

or should it raise an exception? The answer is more complex than one might think.

First of all, we cannot guarantee much about the example program's behavior because there is a race condition between the obvious thread that's executing the example code (which we'll call "UserThread" for the rest of this section) and the KInterbasDB-internal background thread that actually closes connections that have timed out ("TimeoutThread"). If the operating system were to suspend UserThread just after the kinterbasdb.connect call for more than the specified timeout period of 3.0 seconds, the TimeoutThread might close the connection before UserThread had performed any preparatory operations on the database. Although such a scenario is extremely unlikely when more "realistic" timeout periods such as 1800.0 seconds (30 minutes) are used, it is important to consider. We'll explore solutions to this race condition later.

The likely (but not guaranteed) behavior of the example program is that UserThread will complete all preparatory database operations including the cur.execute("update test set b = 'X' where a = 2") statement in the example program, then go to sleep for not less than 6.0 seconds. Not less than 3.0 seconds after UserThread executes the cur.execute("update test set b = 'X' where a = 2") statement, TimeoutThread is likely to close the connection because it has become eligible for timeout.

The crucial issue is how TimeoutThread should resolve the transaction that UserThread left open on con, and what should happen when UserThread reawakens and tries to execute the cur.execute("select * from test") statement, since the transaction that UserThread left open will no longer be active.

User-Supplied Connection Timeout Callbacks

In the context of a particular client program, it is not possible for KInterbasDB to know the best way for TimeoutThread to react when it encounters a connection that is eligible for timeout, but has an unresolved transaction. For this reason, KInterbasDB's connection timeout system offers callbacks that the client programmer can use to guide the TimeoutThread's actions, or to log information about connection timeout patterns.

The "Before Timeout" Callback

The client programmer can supply a "before timeout" callback that accepts a single dictionary parameter and returns an integer code to indicate how the TimeoutThread should proceed when it finds a connection eligible for timeout. Within the dictionary, KInterbasDB provides the following entries:

Based on those data, the user-supplied callback should return one of the following codes:

If the user does not supply a "before timeout" callback, KInterbasDB considers the timeout transparent only if the connection does not have an unresolved transaction.

If the user-supplied "before timeout" callback returns anything other than one of the codes listed above, or if it raises an exception, the TimeoutThread will act as though kinterbasdb.CT_NONTRANSPARENT had been returned.

You might have noticed that the input dictionary to the "before timeout" callback does not include a reference to the kinterbasdb.Connection object itself. This is a deliberate design decision intended to steer the client programmer away from writing callbacks that take a long time to complete, or that manipulate the kinterbasdb.Connection instance directly. See the caveats section for more information.

The "After Timeout" Callback

The client programmer can supply an "after timeout" callback that accepts a single dictionary parameter. Within that dictionary, KInterbasDB currently provides the following entries:

KInterbasDB only calls the "after timeout" callback after the connection has actually been closed by the TimeoutThread. If the "before timeout" callback returns kinterbasdb.CT_VETO to cancel the timeout attempt, the "after timeout" callback will not be called.

KInterbasDB discards the return value of the "after timeout" callback, and ignores any exceptions.

The same caveats that apply to the "before timeout" callback also apply to the "after timeout" callback.

User-Supplied Connection Timeout Callback Caveats
User-Supplied Connection Timeout Callback Example: CT_VETO

The following program registers a "before timeout" callback that unconditionally returns kinterbasdb.CT_VETO, which means that the TimeoutThread never times the connection out. Although an "after timeout" callback is also registered, it will never be called. 

import time
import kinterbasdb

def callback_before(info):
    print
    print 'callback_before called; input parameter contained:'
    for key, value in info.items():
        print '  %s: %s' % (repr(key).ljust(20), repr(value))
    print
    # Unconditionally veto any timeout attempts:
    return kinterbasdb.CT_VETO

def callback_after(info):
    assert False, 'This will never be called.'

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={
        'period': 3.0,
        'callback_before': callback_before,
        'callback_after':  callback_after,
      }
  )
cur = con.cursor()

cur.execute("recreate table test (a int, b char(1))")
con.commit()

cur.executemany("insert into test (a, b) values (?, ?)",
    [(1, 'A'), (2, 'B'), (3, 'C')]
  )
con.commit()

cur.execute("select * from test")
print 'BEFORE:', cur.fetchall()

cur.execute("update test set b = 'X' where a = 2")

time.sleep(6.0)

cur.execute("select * from test")
rows = cur.fetchall()
# The value of the second column of the second row of the table is still 'X',
# because the transaction that changed it from 'B' to 'X' remains active.
assert rows[1][1] == 'X'
print 'AFTER: ', rows

Sample output: 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]

callback_before called; input parameter contained:
  'dsn'               : 'localhost:D:\\temp\\test.db'
  'idle_secs'         : 3.0
  'has_transaction'   : True

AFTER:  [(1, 'A'), (2, 'X'), (3, 'C')]
User-Supplied Connection Timeout Callback Example: Supporting Module timeout_authorizer

The example programs for CT_NONTRANSPARENT, CT_ROLLBACK, and CT_COMMIT rely on the TimeoutAuthorizer class from the module below to guarantee that the TimeoutThread will not time the connection out before the preparatory code has executed. 

import threading
import kinterbasdb

class TimeoutAuthorizer(object):
    def __init__(self, opCodeWhenAuthorized):
        self.currentOpCode = kinterbasdb.CT_VETO
        self.opCodeWhenAuthorized = opCodeWhenAuthorized

        self.lock = threading.Lock()

    def authorize(self):
        self.lock.acquire()
        try:
            self.currentOpCode = self.opCodeWhenAuthorized
        finally:
            self.lock.release()

    def __call__(self, info):
        self.lock.acquire()
        try:
            return self.currentOpCode
        finally:
            self.lock.release()
User-Supplied Connection Timeout Callback Example: CT_NONTRANSPARENT 
import threading, time
import kinterbasdb
import timeout_authorizer

authorizer = timeout_authorizer.TimeoutAuthorizer(kinterbasdb.CT_NONTRANSPARENT)
connectionTimedOut = threading.Event()

def callback_after(info):
    print
    print 'The connection was closed nontransparently.'
    print
    connectionTimedOut.set()

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={
        'period': 3.0,
        'callback_before': authorizer,
        'callback_after':  callback_after,
      }
  )
cur = con.cursor()

cur.execute("recreate table test (a int, b char(1))")
con.commit()

cur.executemany("insert into test (a, b) values (?, ?)",
    [(1, 'A'), (2, 'B'), (3, 'C')]
  )
con.commit()

cur.execute("select * from test")
print 'BEFORE:', cur.fetchall()

cur.execute("update test set b = 'X' where a = 2")

authorizer.authorize()
connectionTimedOut.wait()

# This will raise a kinterbasdb.ConnectionTimedOut exception because the
# before callback returned kinterbasdb.CT_NONTRANSPARENT:
cur.execute("select * from test")

Sample output: 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]

The connection was closed nontransparently.

Traceback (most recent call last):
  File "connection_timeouts_ct_nontransparent.py", line 42, in ?
    cur.execute("select * from test")
kinterbasdb.ConnectionTimedOut: (0, 'A transaction was still unresolved when
this connection timed out, so it cannot be transparently reactivated.')
User-Supplied Connection Timeout Callback Example: CT_ROLLBACK 
import threading, time
import kinterbasdb
import timeout_authorizer

authorizer = timeout_authorizer.TimeoutAuthorizer(kinterbasdb.CT_ROLLBACK)
connectionTimedOut = threading.Event()

def callback_after(info):
    print
    print 'The unresolved transaction was rolled back; the connection has been'
    print ' closed transparently.'
    print
    connectionTimedOut.set()

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={
        'period': 3.0,
        'callback_before': authorizer,
        'callback_after':  callback_after,
      }
  )
cur = con.cursor()

cur.execute("recreate table test (a int, b char(1))")
con.commit()

cur.executemany("insert into test (a, b) values (?, ?)",
    [(1, 'A'), (2, 'B'), (3, 'C')]
  )
con.commit()

cur.execute("select * from test")
print 'BEFORE:', cur.fetchall()

cur.execute("update test set b = 'X' where a = 2")

authorizer.authorize()
connectionTimedOut.wait()

# The value of the second column of the second row of the table will have
# reverted to 'B' when the transaction that changed it to 'X' was rolled back.
# The cur.execute call on the next line will transparently reactivate the
# connection, which was timed out transparently.
cur.execute("select * from test")
rows = cur.fetchall()
assert rows[1][1] == 'B'
print 'AFTER: ', rows

Sample output: 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]

The unresolved transaction was rolled back; the connection has been
 closed transparently.

AFTER:  [(1, 'A'), (2, 'B'), (3, 'C')]
User-Supplied Connection Timeout Callback Example: CT_COMMIT 
import threading, time
import kinterbasdb
import timeout_authorizer

authorizer = timeout_authorizer.TimeoutAuthorizer(kinterbasdb.CT_COMMIT)
connectionTimedOut = threading.Event()

def callback_after(info):
    print
    print 'The unresolved transaction was committed; the connection has been'
    print ' closed transparently.'
    print
    connectionTimedOut.set()

con = kinterbasdb.connect(dsn=r'localhost:D:\temp\test.db',
    user='sysdba', password='masterkey',
    timeout={
        'period': 3.0,
        'callback_before': authorizer,
        'callback_after':  callback_after,
      }
  )
cur = con.cursor()

cur.execute("recreate table test (a int, b char(1))")
con.commit()

cur.executemany("insert into test (a, b) values (?, ?)",
    [(1, 'A'), (2, 'B'), (3, 'C')]
  )
con.commit()

cur.execute("select * from test")
print 'BEFORE:', cur.fetchall()

cur.execute("update test set b = 'X' where a = 2")

authorizer.authorize()
connectionTimedOut.wait()

# The modification of the value of the second column of the second row of the
# table from 'B' to 'X' will have persisted, because the TimeoutThread
# committed the transaction before it timed the connection out.
# The cur.execute call on the next line will transparently reactivate the
# connection, which was timed out transparently.
cur.execute("select * from test")
rows = cur.fetchall()
assert rows[1][1] == 'X'
print 'AFTER: ', rows

Sample output: 

BEFORE: [(1, 'A'), (2, 'B'), (3, 'C')]

The unresolved transaction was committed; the connection has been
 closed transparently.

AFTER:  [(1, 'A'), (2, 'X'), (3, 'C')]


Advanced Transaction Control

For the sake of simplicity, KInterbasDB lets the Python programmer ignore transaction management to the greatest extent allowed by the Python Database API Specification 2.0. The specification says, "if the database supports an auto-commit feature, this must be initially off". At a minimum, therefore, it is necessary to call the commit method of the connection in order to persist any changes made to the database. Transactions left unresolved by the programmer will be rollbacked when the connection is garbage collected.

Remember that because of ACID, every data manipulation operation in the Interbase®/Firebird database engine takes place in the context of a transaction, including operations that are conceptually "read-only", such as a typical SELECT. The client programmer of KInterbasDB establishes a transaction implicitly by using any SQL execution method, such as Connection.execute_immediate, Cursor.execute, or Cursor.callproc.

Although KInterbasDB allows the programmer to pay little attention to transactions, it also exposes the full complement of the database engine's advanced transaction control features: transaction parameters, retaining transactions, savepoints, and distributed transactions.


Transaction Parameters

The database engine offers the client programmer an optional facility called transaction parameter buffers (TPBs) for tweaking the operating characteristics of the transactions he initiates. These include characteristics such as "whether the transaction has read and write access to tables, or read-only access, and whether or not other simultaneously active transactions can share table access with the transaction" (IB 6 API Guide, page 62).

In addition to the implicit transaction initiation mentioned in the introduction of this section, KInterbasDB allows the programmer to start transactions explicitly via the Connection.begin method. Connections have a default_tpb attribute that can be changed to set the default TPB for all transactions subsequently started on the connection. Alternatively, if the programmer only wants to set the TPB for a single transaction, he can start a transaction explicitly via the Connection.begin method and pass a TPB for that single transaction.

For details about TPB construction, see Chapter 5 of the Interbase® 6 API Guide. In particular, page 63 of that document presents a table of possible TPB elements--single bytes that the C API defines as constants whose names begin with isc_tpb_. KInterbasDB makes all of those TPB constants available (under the same names) as module-level constants in the form of single-character strings. A transaction parameter buffer is handled in C as a character array; KInterbasDB requires that TPBs be constructed as Python strings. Since the constants in the kinterbasdb.isc_tpb_* family are single-character Python strings, they can simply be concatenated to create a TPB.

The following program uses explicit transaction initiation and TPB construction to establish an unobtrusive transaction for read-only access to the database: 

import kinterbasdb

con = kinterbasdb.connect(dsn='localhost:/temp/test.db', user='sysdba', password='pass')

# Construct a TPB by concatenating single-character strings (bytes)
# from the kinterbasdb.isc_tpb_* family.
customTPB = (
      kinterbasdb.isc_tpb_read
    + kinterbasdb.isc_tpb_read_committed
    + kinterbasdb.isc_tpb_rec_version
  )

# Explicitly start a transaction with the custom TPB:
con.begin(tpb=customTPB)

# Now read some data using cursors:
...

# Commit the transaction with the custom TPB.  Future transactions
# opened on con will not use a custom TPB unless it is explicitly
# passed to con.begin every time, as it was above, or
# con.default_tpb is changed to the custom TPB, as in:
#   con.default_tpb = customTPB
con.commit()

Retaining Operations

The commit and rollback methods of kinterbasdb.Connection accept an optional boolean parameter retaining (default False) to indicate whether to recycle the transactional context of the transaction being resolved by the method call.

If retaining is True, the infrastructural support for the transaction active at the time of the method call will be "retained" (efficiently and transparently recycled) after the database server has committed or rolled back the conceptual transaction.

In code that commits or rolls back frequently, "retaining" the transaction yields considerably better performance. However, retaining transactions must be used cautiously because they can interfere with the server's ability to garbage collect old record versions. For details about this issue, read the "Garbage" section of this document by Ann Harrison.

For more information about retaining transactions, see page 291 of the Interbase® 6 API Guide.


Savepoints

Firebird 1.5 introduced support for transaction savepoints. Savepoints are named, intermediate control points within an open transaction that can later be rolled back to, without affecting the preceding work. Multiple savepoints can exist within a single unresolved transaction, providing "multi-level undo" functionality.

Although Firebird savepoints are fully supported from SQL alone via the SAVEPOINT 'name' and ROLLBACK TO 'name' statements, KInterbasDB also exposes savepoints at the Python API level for the sake of convenience. The method Connection.savepoint(name) establishes a savepoint with the specified name. To roll back to a specific savepoint, call the Connection.rollback method and provide a value (the name of the savepoint) for the optional savepoint parameter. If the savepoint parameter of Connection.rollback is not specified, the active transaction is cancelled in its entirety, as required by the Python Database API Specification.

The following program demonstrates savepoint manipulation via the KInterbasDB API, rather than raw SQL. 

import kinterbasdb

con = kinterbasdb.connect(dsn='localhost:/temp/test.db', user='sysdba', password='pass')
cur = con.cursor()

cur.execute("recreate table test_savepoints (a integer)")
con.commit()

print 'Before the first savepoint, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

cur.execute("insert into test_savepoints values (?)", [1])
con.savepoint('A')
print 'After savepoint A, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

cur.execute("insert into test_savepoints values (?)", [2])
con.savepoint('B')
print 'After savepoint B, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

cur.execute("insert into test_savepoints values (?)", [3])
con.savepoint('C')
print 'After savepoint C, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

con.rollback(savepoint='A')
print 'After rolling back to savepoint A, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

con.rollback()
print 'After rolling back entirely, the contents of the table are:'
cur.execute("select * from test_savepoints")
print ' ', cur.fetchall()

The output of the example program is shown below. 

Before the first savepoint, the contents of the table are:
  []
After savepoint A, the contents of the table are:
  [(1,)]
After savepoint B, the contents of the table are:
  [(1,), (2,)]
After savepoint C, the contents of the table are:
  [(1,), (2,), (3,)]
After rolling back to savepoint A, the contents of the table are:
  [(1,)]
After rolling back entirely, the contents of the table are:
  []

Distributed Transactions

 XXX: KInterbasDB's support for distributed transactions has not yet been thoroughly documented. In the meantime, read the source code for the kinterbasdb.ConnectionGroup class and examine the brief example program below. 
import kinterbasdb

# Establish multiple connections the usual way:
con1 = kinterbasdb.connect(dsn='weasel:/temp/test.db', user='sysdba', password='pass')
con2 = kinterbasdb.connect(dsn='coyote:/temp/test.db', user='sysdba', password='pass')

# Create a ConnectionGroup to associate multiple connections in such a
# way that they can participate in a distributed transaction.
# !!!
# NO TWO MEMBERS OF A SINGLE CONNECTIONGROUP SHOULD BE ATTACHED TO THE SAME DATABASE!
# !!!
group = kinterbasdb.ConnectionGroup( connections=(con1,con2) )

# Start a distributed transaction involving all of the members of the group
# (con1 and con2 in this case) with one of the following approaches:
#   - Call  group.begin()
#   - Call  con1.begin(); the operation will "bubble upward" and apply to the group.
#   - Call  con2.begin(); the operation will "bubble upward" and apply to the group.
#   - Just start executing some SQL statements on either con1 or con2.
#     A transaction will be started implicitly; it will be a distributed
#     transaction because con1 and con2 are members of a ConnectionGroup.
group.begin()

# Perform some database changes the usual way (via cursors on con1 and con2):
...

# Commit or roll back the distributed transaction by calling the commit
# or rollback method of the ConnectionGroup itself, or the commit or
# rollback method of any member connection (con1 or con2 in this case).
group.commit()

# Unless you want to perform another distributed transaction, disband the
# group so that member connections can operate independently again.
group.clear()

Notes:

While a Connection belongs to a ConnectionGroup, any calls to the connection's transactional methods (begin, prepare, commit, rollback) will "bubble upward" to apply to the distributed transaction shared by the group as a whole.

Connections can be dynamically added and removed from a ConnectionGroup provided that neither the group nor the connection itself has an unresolved transaction at the time of the addition/removal.

Pitfalls and Limitations

Parameter Conversion

KInterbasDB converts bound parameters marked with a ? in SQL code in a standard way. However, the module also offers several extensions to standard parameter binding, intended to make client code more readable and more convenient to write.

Implicit Conversion of Input Parameters from Strings

The database engine treats most SQL data types in a weakly typed fashion: the engine may attempt to convert the raw value to a different type, as appropriate for the current context. For instance, the SQL expressions 123 (integer) and '123' (string) are treated equivalently when the value is to be inserted into an integer field; the same applies when '123' and 123 are to be inserted into a varchar field.

This weak typing model is quite unlike Python's dynamic yet strong typing. Although weak typing is regarded with suspicion by most experienced Python programmers, the database engine is in certain situations so aggressive about its typing model that KInterbasDB must compromise in order to remain an elegant means of programming the database engine.

An example is the handling of "magic values" for date and time fields. The database engine interprets certain string values such as 'yesterday' and 'now' as having special meaning in a date/time context. If KInterbasDB did not accept strings as the values of parameters destined for storage in date/time fields, the resulting code would be awkward. Consider the difference between the two Python snippets below, which insert a row containing an integer and a timestamp into a table defined with the following DDL statement: 

create table test_table (i int, t timestamp)

i = 1
t = 'now'
sqlWithMagicValues = "insert into test_table (i, t) values (?, '%s')" % t
cur.execute( sqlWithMagicValues, (i,) )

i = 1
t = 'now'
cur.execute( "insert into test_table (i, t) values (?, ?)", (i, t) )

If KInterbasDB did not support weak parameter typing, string parameters that the database engine is to interpret as "magic values" would have to be rolled into the SQL statement in a separate operation from the binding of the rest of the parameters, as in the first Python snippet above. Implicit conversion of parameter values from strings allows the consistency evident in the second snippet, which is both more readable and more general.

It should be noted that KInterbasDB does not perform the conversion from string itself. Instead, it passes that responsibility to the database engine by changing the parameter metadata structure dynamically at the last moment, then restoring the original state of the metadata structure after the database engine has performed the conversion.

A secondary benefit is that when one uses KInterbasDB to import large amounts of data from flat files into the database, the incoming values need not necessarily be converted to their proper Python types before being passed to the database engine. Eliminating this intermediate step may accelerate the import process considerably, although other factors such as the chosen connection protocol and the deactivation of indexes during the import are more consequential. For bulk import tasks, the database engine's external tables also deserve consideration. External tables can be used to suck semi-structured data from flat files directly into the relational database without the intervention of an ad hoc conversion program.

Dynamic Type Translation

Dynamic type translators are conversion functions registered by the Python programmer to transparently convert database field values to and from their internal representation.

The client programmer can choose to ignore translators altogether, in which case KInterbasDB will manage them behind the scenes. Otherwise, the client programmer can use any of several standard type translators included with KInterbasDB, register custom translators, or set the translators to None to deal directly with the KInterbasDB-internal representation of the data type. When translators have been registered for a specific SQL data type, Python objects on their way into a database field of that type will be passed through the input translator before they are presented to the database engine; values on their way out of the database into Python will be passed through the corresponding output translator. Output and input translation for a given type is usually implemented by two different functions.

Specifics of the Dynamic Type Translation API

Translators are registered with the [set|get]_type_trans_[in|out] methods of Connection and Cursor. The set_type_trans_[in|out] methods accept a single argument: a mapping of type name to translator. The get_type_trans[in|out] methods return a copy of the translation table. Cursors inherit their Connection's translation settings, but can override them without affecting the connection or other cursors (much as subclasses can override the methods of their base classes).

The following code snippet installs an input translator for fixed point types (NUMERIC/DECIMAL SQL types) into a connection: 

con.set_type_trans_in( {'FIXED': fixed_input_translator_function} )

The following method call retrieves the type translation table for con: 

con.get_type_trans_in()

The method call above would return a translation table (dictionary) such as this: 

{
  'DATE': <function date_conv_in at 0x00920648>,
  'TIMESTAMP': <function timestamp_conv_in at 0x0093E090>,
  'FIXED': <function <lambda> at 0x00962DB0>,
  'TIME': <function time_conv_in at 0x009201B0>
}

Notice that although the sample code registered only one type translator, there are four listed in the mapping returned by the get_type_trans_in method. By default, KInterbasDB uses dynamic type translation to implement the conversion of DATE, TIME, TIMESTAMP, NUMERIC, and DECIMAL values. For the source code locations of KInterbasDB's reference translators, see the table in the next section.

In the sample above, a translator is registered under the key 'FIXED', but Firebird has no SQL data type named FIXED. The following table lists the names of the database engine's SQL data types in the left column, and the corresponding KInterbasDB-specific key under which client programmers can register translators in the right column.

Mapping of SQL Data Type Names to Translator Keys
SQL Type(s) Translator Key
CHAR/VARCHAR 'TEXT' for fields with charsets NONE, OCTETS, or ASCII
'TEXT_UNICODE' for all other charsets
BLOB 'BLOB'
SMALLINT/INTEGER/BIGINT 'INTEGER'
FLOAT/DOUBLE PRECISION 'FLOATING'
NUMERIC/DECIMAL 'FIXED'
DATE 'DATE'
TIME 'TIME'
TIMESTAMP 'TIMESTAMP'

Consequences of the Availability of Dynamic Type Translation in KInterbasDB

Dynamic type translation has eliminated KInterbasDB's dependency on mx.DateTime. Although KInterbasDB will continue to use mx.DateTime as its default date/time representation for the sake of backward compatibility, dynamic type translation allows users to conveniently deal with database date/time values in terms of the new standard library module datetime, or any other representation they care to write translators for.

Dynamic type translation also allows NUMERIC/DECIMAL values to be transparently represented as decimal.Decimal objects rather than scaled integers, which is much more convenient. For backward compatibility, NUMERIC/DECIMAL values are still represented by default as Python floats, and the older API based on the Connection.precision_mode attribute is still present. However, all of these representations are now implemented "under the hood" via dynamic type translation.

Reference implementations of all of the translators discussed above are provided with KInterbasDB, in these modules:

Reference Translators Included with KInterbasDB
SQL Type(s) Python Type(s) Reference Implementation In Module
NUMERIC/DECIMAL float (imprecise) (default) kinterbasdb.typeconv_fixed_stdlib
scaled int (precise) kinterbasdb.typeconv_fixed_stdlib
fixedpoint.FixedPoint (precise) kinterbasdb.typeconv_fixe