This is for Debian woody! Below are some benchmarking figures obtained while reading and writing to a file with three tables, each table containing 10000 records. For reference, the same tests have been repeated using the shelve module that comes with Python. The tests were conducted on a platform with a 2 GHz AMD Athlon chip, an IDE disk at 4600 rpm, and 256 MB of RAM. Version 0.2 | medium size records | small size records | (47 Bytes) | (16 Bytes) +---------------------------+------------------------------ | rows/s filesize | rows/s filesize | write read | write read ------------+---------------------------+------------------------------ no compress| | record | 24400 39000 1184 KB | 32600 52600 506 KB tupla | 17100 81100 1184 KB | 66666 107142 506 KB ------------+---------------------------+------------------------------ compress | | record | 22200 37500 494 KB | 31900 51700 94 KB tupla | 16100 75000 494 KB | 63900 107142 94 KB ------------+---------------------------+------------------------------ Shelve | 25800 14400 2500 KB | 68200 17000 921 KB New version (15-Jan-2003) PyTables pre-0.3 Rec length | rows/s | KB/s | rows | filesz | memory | | write read | write read | | (MB) | (MB) | ------------+-----------------+-----------------+-------+--------+--------+ 16 B | 31000 166600 | 480 2600 | 3.e4 | 0.49| 6.5 | ------------+-----------------+-----------------+-------+--------+--------+ 56 B | 17300 136000 | 942 7460 | 3.e4 | 1.7 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 56 B* | 1560 136000 | 85 7560 | 3.e4 | 1.7 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 64 B* | 1540 130000 | 96 8152 | 3.e4 | 1.9 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B* | 879 81100 | 472 43500 | 3.e4 | 19 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B** | 12000 103000 | 6440 55400 | 3.e5 | 168 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B** | 15500 81100 | 8350 43500 | 3.e4 | 19 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B**c| 909 1100 | 490 1081 | 3.e4 | 0.76| 8.5 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B***| 3600 81100 | 1950 43500 | 3.e4 | 19 | 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ * These are figures obtained with a numarray as part of the record ** The numarray record fields are not set in each iteration *** Some numarray elements of a record field are changed on each iteration **c Like ** but with compression (level 1) New version (10-March-2003) PyTables pre-0.4 Rec | rows/s | KB/s | rows | filesz | memory |%CP|%CP length | write read | write read | | (MB) | (MB) |(w)|(r) --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B |434000 469000 | 6800 7300 | 3.e4 | 0.49| 6.5 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 Bc |326000 435000 | 5100 6800 | 3.e4 | 0.12| 6.5 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B |663000 728000 | 10400 11400 | 3.e5 | 4.7 | 7.0 | 99|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B |679000 797000 | 10600 12500 | 3.e6 | 46.0 | 10.0 | 98| 98 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 Bc |452000 663000 | 7100 10400 | 3.e6 | 9.3 | 10.0 | 98| 98 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B |576000 590000 | 9000 9200 | 3.e7 | 458.0 | 11.0 | 78| 76 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 3050 380000 | 163 20700 | 3.e4 | 1.7 | 7.2 | 98|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B* |194000 340000 | 10600 18600 | 3.e4 | 1.7 | 7.2 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B*c |142000 306000 | 7800 16600 | 3.e4 | 0.3 | 7.2 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B* |273600 589000 | 14800 32214 | 3.e5 | 16.0 | 9.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B*c |184000 425000 | 10070 23362 | 3.e5 | 2.7 | 9.7 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B* |203600 649000 | 11100 35500 | 3.e6 | 161.0 | 12.0 | 72| 99 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B* |184000 229000 | 10000 12500 | 1.e7 | 534.0 | 17.0 | 56| 40 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B*np|184000 229000 | 10000 12500 | 1.e7 | 534.0 | 17.0 | 56| 40 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B | 2230 143000 | 1195 76600 | 3.e4 | 19 | 9.4 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B* | 76000 250000 | 40900 134000 | 3.e4 | 19 | 9.4 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B*c | 13900 30000 | 7400 16100 | 3.e4 | 0.7 | 10.0 | 99| 99 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B* | 25400 325000 | 13600 174000 | 3.e5 | 167 | 11.0 | 71| 96 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B* | 18700 28000 | 10000 15100 | 6.e5 | 322 | 13.0 | 76| 9 --------+-----------------+-----------------+-------+--------+--------+---+---- 550 B*c | 7300 21000 | 3900 11300 | 6.e5 | 11 | 17.0 | 98| 99 --------+-----------------+-----------------+-------+--------+--------+---+---- * These are figures obtained with a numarray as part of the record ** The numarray record fields are not set in each iteration c With compression (level 1) np No psyco optimizations Shelve Rec length | rows/s | KB/s | rows | filesz | memory | | write read | write read | | (MB) | (MB) | ------------+-----------------+-----------------+-------+--------+--------+ 16 B | 68200 17000 | 1070 266 | 3.e4 | 0.94| 7.2 | ------------+-----------------+-----------------+-------+--------+--------+ 56 B | 25000 14400 | 1367 784 | 3.e4 | 2.5 | 10.6 | ------------+-----------------+-----------------+-------+--------+--------+ 56 B* | 2980 2710 | 162 148 | 3.e4 | 7.3 | 33 | ------------+-----------------+-----------------+-------+--------+--------+ 64 B* | 2900 2700 | 182 168 | 3.e4 | 7.5 | 33 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B* | 1090 1310 | 590 710 | 3.e4 | 58 | 122 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B** | 16000 14900 | 2400 1200 | 3.e4 | 2.4 | 10.6 | ------------+-----------------+-----------------+-------+--------+--------+ 550 B***| 28000 11900 | 2400 1100 | 3.e4 | 2.5 | 10.6 | ------------+-----------------+-----------------+-------+--------+--------+ * These are figures obtained with a numarray as part of the record ** The nuamrray records are not set on each iteration *** Some numarray elements of a record field are changed on each iteration Python cPickle & bsddb3 RECNO with variable length Rec | Krows/s | MB/s | Krows | filesz | memory |%CP|%CP length | write read | write read | | (MB) | (MB) |(w)|(r) --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 23.0 4.3 | 0.65 0.12 | 30 | 2.3 | 6.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 22.0 4.3 | 0.60 0.12 | 300 | 24 | 25.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 12.3 2.0 | 0.68 0.11 | 30 | 5.8 | 6.2 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 8.8 2.0 | 0.44 0.11 | 300 | 61 | 6.2 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- Python struct & bsddb3 RECNO with fixed length Rec | Krows/s | MB/s | Krows | filesz | memory |%CP|%CP length | write read | write read | | (MB) | (MB) |(w)|(r) --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 61 71 | 1.6 1.9 | 30 | 1.0 | 5.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 56 65 | 1.5 1.8 | 300 | 10 | 5.8 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 51 61 | 1.4 1.6 | 3000 | 100 | 6.1 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 51 52 | 2.7 2.8 | 30 | 1.8 | 5.8 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 18 50 | 1.0 2.7 | 300 | 18 | 6.2 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 16 48 | 0.9 2.6 | 1000 | 61 | 6.5 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- PySqlLite Rec | rows/s | KB/s | rows | filesz | memory |%CP|%CP length | write read | write read | | (MB) | (MB) |(w)|(r) --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 4290 1400000 | 200 48000 | 3.e4 | 1.4 | 5.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 3660 1030000 | 182 51000 | 3.e5 | 15 | 5.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 16 B | 3580 230000 | 192 12380 | 6.e6 | 322 | 5.0 |100| 25 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 2990 882000 | 250 76000 | 3.e4 | 2.6 | 5.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 2900 857000 | 270 80000 | 3.e5 | 28 | 5.0 |100|100 --------+-----------------+-----------------+-------+--------+--------+---+---- 56 B | 2900 120000 | 302 13100 | 3.e6 | 314 | 5.0 |100| 11 --------+-----------------+-----------------+-------+--------+--------+---+----