DataPak

DataPak (*.dpk) is mainly intended as a compression method for application assets. Technically it can store folder info, but the fixed filename length is shorter to save some extra space. There's some support for expanding the path, but currently unimplemented.

General layout of a file: <ul> <li>DataPak signature</li> <li>Header</li> <li>Extension area. Ignored by default, handling must be implemented by whoever wants to use it. Can be compressed alongside with the main data to save some space (not recommended if it's needed for compression, e.g. dictionaries), but only if the index field is also compressed.</li> <li>Array of DataPak indexes. Each entry can have some extension. Can be compressed alongside with the main data to save some space.</li> <li>CRC32 checksum at the begining of compressed block or at the end of file information description table.</li> <li>Data</li> </ul>

class DataPak {

enum char[8] SIGNATURE;

enum CompressionMethod;

enum ChecksumType;

ubyte[14] CHECKSUM_LENGTH;

struct Header;

struct Index;

Header header;

File file;

Index[] indexes;

string[] paths;

uint nextIndex;

ubyte[] extField;

ubyte[][uint] indexExtFields;

bool readOnly;

bool createNew;

void* compStream;

ubyte[] readBuf;

ubyte[] compBuf;

ZSTD_inBuffer inBuff;

ZSTD_outBuffer outBuff;

size_t prevOutPos;

ulong compPos;

ulong compPos0;

void delegate(size_t pos) progress;

bool enableThrowOnChecksumError;

size_t readBufferSize;

this(string filename);

this(File f);

this(Header header, string targetName, ubyte[] extField = []);

Index addFile(string filename, string newName = null, ubyte[] indexExtField = []);

void initComp();

void initDecomp();

Index getIndex(uint i);

Index getNextIndex();

ubyte[] getNextAsArray();

bool checkFile(ubyte[] data, ubyte[] checksum);

void finalize();

void compress(string source, Index index);

ubyte[] decompressFromFile(const size_t amount);

}

Constructors

this this(string filename)
this(File f): Loads a DataPak file from disk for reading.
this this(Header header, string targetName, ubyte[] extField = []): Creates a DataPak file from scratch.

ChecksumType enum ChecksumType: Selects between checksums. Please note that the more bytes the checksum needs, the less will be left for the filename. Values between 32-63 can be used for custom implementations.
CompressionMethod enum CompressionMethod: Default compression methods for the file.

Functions

addFile Index addFile(string filename, string newName = null, ubyte[] indexExtField = []): Adds a file to be compressed later. Returns the created index for it.
checkFile bool checkFile(ubyte[] data, ubyte[] checksum): Checks the integrity of a file.
compress void compress(string source, Index index): Compresses a single file into the stream.
decompressFromFile ubyte[] decompressFromFile(const size_t amount): Decompresses a given amount from the file from the current position.
finalize void finalize(): Begins compression into file.
getIndex Index getIndex(uint i): Returns a given index.
getNextAsArray ubyte[] getNextAsArray(): Returns the next file as an ubyte[] array.
getNextIndex Index getNextIndex(): Returns the index of next file.
initComp void initComp(): Initializes compression.
initDecomp void initDecomp(): Initializes decompression.

Static variables

CHECKSUM_LENGTH ubyte[14] CHECKSUM_LENGTH;: Stores the length of each checksum result
SIGNATURE enum char[8] SIGNATURE;: Every DataPak file begins with this. The dot will be replaced with numbers if I ever decide to make any upgrades to the format
enableThrowOnChecksumError bool enableThrowOnChecksumError;: Doesn't throw on errors, returns data regardless of error
readBufferSize size_t readBufferSize;: Sets the read buffer size of all instances (default is 32kB)

Structs

Header struct Header: Stores important informations about the file.
Index struct Index: Index representing data for a file.

Variables

compPos0 ulong compPos0;: Current position of data; position of all currently decompressed data
paths string[] paths;: Only used during compression
prevOutPos size_t prevOutPos;: 0 if no data left from previous decompression
progress void delegate(size_t pos) progress;: Called to inform host on decompression process (will be replaced with something more useful in the future)

DataPak

Constructors

Destructor

Members

Enums

Functions

Static variables

Structs

Variables

Meta

Source