Package qubx :: Module tree

Module tree

Classes for reading and writing all QuB binary files.

A QuB Tree is a hierarchy of Nodes. Each node may have a name and some text or numeric data. Specific QuB file types, e.g. QMF (qub model files), expect specific heirarchies and data types.

History:

In the late 1990s Dr. Lorin Milescu defined the binary layout and used it for QuB data files (QDF). He wrote a class in Delphi pascal called "QFS" to read and modify open files on disk.

By the early 2000s we were collaborating on a DLL interface between his GUI, written in Delphi, and older algorithms written in C. We found ourselves spending a lot of time defining struct- and pointer-based containers for all the varieties of information shared between GUI and DLL. At the same time, folks were asking us for new extensible formats for models and idealized data. Lorin naturally proposed QFS. I was feeling some pressure to use human-readable ASCII text, so I proposed a file that could be stored as binary or text, depending whether efficiency or transparency was foremost. Then I went ahead and implemented it in C/C++, with bindings to Delphi. I never really got the text part right, but no-one really cared.

So now we had two implementations: QFS, and "qubtree" or "native". We had some idea of changing everything over to the newer one, but it would have meant a ton of extra work re-implementing the core of QuB. When I made the Delphi bindings, I had our DLL quagmire in mind. qubtree.dll took care of memory management and reference counting so you could pass trees willy-nilly between the EXE and DLL without headaches. And now that everything boiled down to one or two ref-counted qubtree pointers, we weren't constantly breaking binary compatibility between DLL and EXE. Similarly, I added Python bindings and rudimentary callback support, so we could do a lot of stuff across the three languages without too much intricate boilerplate.

This unit can interface to the DLL/.so, and it introduces a third implementation -- in Python, using numpy and mmap. This 'numpy' flavor behaves exactly like the 'native' one, except:

it doesn't even try to ReadString or ReadText; like I said, always broken anyway
it ignores lock/unlock since python's single-threaded for ref-counting purposes
you can't edit string data one character at a time, due to a limitation in module mmap
you can get direct access to numeric data as numpy.array: node.storage.data
you can resize a node with empty data, by specifying a default type

Flavors:

By default this module still uses the 'native' flavor, so there are no surprises for existing programs. If you don't have the compiled _qubtree library, it falls back on the 'numpy' flavor. Of course, you must have numpy, available separately. If you have both, you can switch the default for new nodes:

>>> qubx.tree.CHOOSE_FLAVOR('numpy')

and even convert a tree between the two flavors:

>>> as_numpy = qubx.tree.AS_FLAVOR(native_node, 'numpy')

Limitations:

32 bits, except when signed/unsigned are treated inconsistently; then it's more like 31. (max safe file size 2GB - 1)

Supported by no-one else

Not very efficient when you have lots of nodes.

Older qubtree documentation

QUB Express is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

QUB Express is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License, named LICENSE.txt, in the QUB Express program directory. If not, see <http://www.gnu.org/licenses/>.

Classes

[hide private]

dtree
A light intermediary between a file and a reader of qubtree nodes, for debugging.

Truncation

NullData
Represents the data of a null node, which is empty and will stay empty.

NullNode_numpy
Represents the absence of a qubtree node.

Data

DataRow
A handler for the data in one row of a qubtree node.

DataCol
A handler for the data in one column of a qubtree node.

NodeStorage
Abstract base class for managing a concrete representation of one qubtree node.

NodeInMem
Manages the attributes and data of a Node that's not in a file.

NodeFile
Manages a memory-mapped qubtree file, either on disk or an anonymous buffer.

NodeInFile
Manages the attributes and data of a Node inside a NodeFile.

Dirtee

Node_numpy
Represents the (sub-)tree under a particular node in a qub tree, in an open file or in memory.

NodeFile_NoMap
Manages a non-memory-mapped qubtree file on disk.

NodeInFile_NoMap
Manages the attributes and data of a Node inside a NodeFile_NoMap.

Node_numpy_NoMap
Represents the (sub-)tree under a particular node in a qub tree.

Node
Represents the (sub-)tree under a particular node in a qub tree, in an open file or in memory.

NullNode
Represents the absence of a qubtree node.

Functions

[hide private]

CHOOSE_FLAVOR(flav)
Selects the default flavor for new Nodes, either 'native' (default) or 'numpy'.

source code

AS_FLAVOR(node, flav, copyAnyway=False)
Returns node in the requested flavor.

source code

children(tree, name=None)
Yields all children of tree, matching name if specified.

source code

tree_to_dict(tree, is_root=True)

source code

treedata_to_dict(data)

source code

dict_to_tree(d, is_root=True, name=None)

source code

node_data_or_def(node, default, index=None)
Returns the numeric or string data of a qubx.tree.Node_numpy, or default if the node has no data. source code

node_data_or_set_def(parent, child_name, default)
Returns parent[child_name].data[0] or default; sets default in tree if missing.

source code

ignore_dirtied(storage)

source code

can_data_in_node(type, count)
Returns True if the data will fit inside the 4 byte dataPos field.

source code

Open_numpy(path, readOnly=False)
Returns the root Node of the qubtree file at path, or the NullNode.

source code

ReadBytes_numpy(buf)
Returns the root node of the qubtree copied from the binary in buf.

source code

Open(path, readOnly=False)
Returns the root Node of the qubtree file at path, or the NullNode.

source code

ReadBytes(buf)
Returns the root node of the qubtree copied from the binary in buf.

source code

Variables

[hide private]

FLAVOR_DEFAULT = 'native'

FLAVORS_AVAIL = ['native', 'numpy']

have_lib = True

have_numpy = True

TYPES = (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)

TYPENAMES = {0: 'empty', 1: 'unknown', 2: 'pointer', 3: 'strin...

TYPE_DEFAULTS = (None, None, None, ' ', 0, 0, 0, 0, 0, 0, 0, 0...

SIZE_OF_TYPE = (0, 0, 4, 1, 1, 1, 2, 2, 4, 4, 8, 8, 4, 8, 10)

framedef = '<BBBBiiiiiiBBBB'

framelen = 32

frame_dataoff = 12

QTR_FLAG_PRELOAD = 2

QTR_FLAG_DATA_IN_NODE = 4

QTR_MAGIC = 'QUB_(;-)_QFS'

TYPES_NUMPY = [None, None, dtype('uint32'), dtype('uint8'), dt...

TYPE_OF_NUMPY = defaultdict(<function <lambda> at 0x7fd129ae38...

MAX_QUICK_READ_BYTES = 524288

MAX_QUICK_WRITE_BYTES = 10485760

QTR_INITIAL_CAP = 524288

ZERO_BATCH = 262144

NullNode_numpy_instance = <QUBTree Node : NULL>

FLAVOR = 'numpy'

QTR_TYPE_CHAR = 5

QTR_TYPE_DOUBLE = 13

QTR_TYPE_EMPTY = 0

QTR_TYPE_FLOAT = 12

QTR_TYPE_INT = 9

QTR_TYPE_LDOUBLE = 14

QTR_TYPE_LONG = 11

QTR_TYPE_POINTER = 2

QTR_TYPE_SHORT = 7

QTR_TYPE_STRING = 3

QTR_TYPE_UCHAR = 4

QTR_TYPE_UINT = 8

QTR_TYPE_ULONG = 10

QTR_TYPE_UNKNOWN = 1

QTR_TYPE_USHORT = 6

__package__ = 'qubx'

i = 14

Function Details

[hide private]

AS_FLAVOR(node, flav, copyAnyway=False)

source code

Returns node in the requested flavor. If it is already, we can return the original (default) or copyAnyway.

node_data_or_def(node, default, index=None)

source code

Returns the numeric or string data of a qubx.tree.Node_numpy, or default if the node has no data.

Suitable for reading preferences that might be missing.

Open_numpy(path, readOnly=False)

source code

Returns the root Node of the qubtree file at path, or the NullNode. The file remains open until close()d or unref'd. Small read-only trees may not be kept open.

This is the 'numpy' flavor of qubx.tree.Open

ReadBytes_numpy(buf)

source code

Returns the root node of the qubtree copied from the binary in buf.

This is the 'numpy' flavor of qubx.tree.ReadBytes

Open(path, readOnly=False)

source code

Returns the root Node of the qubtree file at path, or the NullNode. The file remains open until close()d or unref'd. Small read-only trees may not be kept open.

This is the 'numpy' flavor of qubx.tree.Open

ReadBytes(buf)

source code

Returns the root node of the qubtree copied from the binary in buf.

This is the 'numpy' flavor of qubx.tree.ReadBytes

Variables Details

[hide private]

TYPENAMES

Value:

{0: 'empty',
 1: 'unknown',
 2: 'pointer',
 3: 'string',
 4: 'unsigned 8-bit int',
 5: 'signed 8-bit int',
 6: 'unsigned 16-bit int',
 7: 'signed 16-bit int',
...

TYPE_DEFAULTS

Value:

(None, None, None, ' ', 0, 0, 0, 0, 0, 0, 0, 0, 0.0, 0.0, 0.0)

TYPES_NUMPY

Value:

[None,
 None,
 dtype('uint32'),
 dtype('uint8'),
 dtype('uint8'),
 dtype('int8'),
 dtype('uint16'),
 dtype('int16'),
...

TYPE_OF_NUMPY

Value:

defaultdict(<function <lambda> at 0x7fd129ae3848>, {'uint64': 10, 'int
32': 9, 'int16': 7, 'complex128': 14, 'float64': 13, 'uint8': 4, 'int8
': 5, 'uint16': 6, 'int64': 11, 'bool': 9, 'uint32': 8, 'float32': 12}
)