This is Part 1 of a multi-part series explaining how TexTools does what it does. I've been meaning to give an overview of all the processes and so here we are.
In this part, we're going to go over how the raw data that TexTools reads is formatted.
The raw data is stored in .dat files located in the Game_Directory/game/sqpack folder which is then separated into folders based on expansion. /ffxiv/ is the base game, /ex1/ is the first expansion, /ex2/ is the second expansion, and so on.
Inside these folders we have .dat files and .index files.
- .index files are used to determine what part of the .dat file the data for a particular item is stored in.
- .dat files contain the raw data for all of the items in the game.
Note: I use the term 'item' to refer to any data entry.
The Index File
First we'll go over the relevant parts of the .index files that TexTools uses.
More details here: Index File Research
We begin at offset 0x400
00000400: 00 04 00 00 01 00 00 00 00 08 00 00 30 91 1F 00
The bytes highlighted above are the byte size of the file list contained within this index file, in this case 2,068,784 bytes.
Each entry in the file list is 16 bytes in length so dividing the total byte size by the individual file size gives us 129,299 files.
Next we go to offset 0x450
00000450: 04 00 00 00 50 C1 1A 00 00 01 00 00 5E 9D 28 D0
00000460: 48 5D A8 38 F6 2D 71 3C 3D B6 96 1A 6E 13 D8 3B
The highlighted bytes above determine the number of .dat files the game will attempt to read.
In the above case, it is from the 040000 index file, so the game will attempt to read the following
- 040000.win32.dat0
- 040000.win32.dat1
- 040000.win32.dat2
- 040000.win32.dat3
TexTools changes this value to 5, because the fact that it stores all of the modified data in a separate dat file, so that the game now attempts to read
- 040000.win32.dat4
The next relevant part begins at offset 0x800, where the file list begins.
00000800: 7D F3 D5 6F 8F 21 00 00 94 49 5F 02 00 00 00 00
Each entry is separated into 4 parts, each part 4 bytes each.
- Hash value of file name Example: v01_c0101e0001_dwn_d.tex
- Hash value of folder path Example: chara/equipment/e0001/texture
- Offset data is located in .dat file
- Empty
In order to determine which dat file to look in, we take the second digit of the first byte in the offset (in bold above) and divide it by 2.
In the example above:
- 4 / 2 = 2
So we locate the data in 040000.win32.dat2
And to get to the correct offset in the .dat file we fist need to multiply the offset by 8.
In the example above:
- 0x025F4994 * 0x8 = 0x12FA4CA0
So all together we are to go to the offset 12FA4CA0 in 040000.win32.dat2 to locate the data for a file whos hash value is 6FD5F37D and folder hash value is 0000218F.
The Dat File
Now that we determined what dat file and offset to got to we need to know what to expect in the dat file.
More details here: Dat File Research
The first few bytes of the dat file are the most important to determine how to read the data.
4017a980: 80 01 00 00 04 00 00 00 C0 55 05 00 18 03 00 00
4017a990: 18 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1: The header length.
2: The type of data of the file.
These are the known file type:
- Type 2: Binary data that can be any number of different things
- Type 3: 3D model data
- Type 4: Texture data
3: The size of the file after is is decompressed
4: Is believed to be the largest size the data buffer can be
5: Is believed to be the actual data buffer size
After these 20 bytes, everything is parsed differently depending on the file type, which I'll go over in their appropriate part of the series.
Part 2 will go over how TexTools fills its TreeView with all the item names and categories directly from the game files.
Part 2 will go over how TexTools fills its TreeView with all the item names and categories directly from the game files.
