TexTools Explained Part 1: The Raw Data

Hello all,

This is Part 1 of a multi-part series explaining how TexTools does what it does. I've been meaning to give an overview of all the processes and so here we are.

In this part, we're going to go over how the raw data that TexTools reads is formatted.

---

The raw data is stored in .dat files located in the Game_Directory/game/sqpack folder which is then separated into folders based on expansion. /ffxiv/ is the base game, /ex1/ is the first expansion, /ex2/ is the second expansion, and so on.

Inside these folders we have .dat files and .index files.

• .index files are used to determine what part of the .dat file the data for a particular item is stored in.
• .dat files contain the raw data for all of the items in the game.

Note: I use the term 'item' to refer to any data entry.

The Index File

First we'll go over the relevant parts of the .index files that TexTools uses.

More details here: Index File Research

We begin at offset 0x400

00000400: 00 04 00 00 01 00 00 00 00 08 00 00 30 91 1F 00

The bytes highlighted above are the byte size of the file list contained within this index file, in this case 2,068,784 bytes.

Each entry in the file list is 16 bytes in length so dividing the total byte size by the individual file size gives us 129,299 files.

Next we go to offset 0x450

00000450: 04 00 00 00 50 C1 1A 00 00 01 00 00 5E 9D 28 D0

00000460: 48 5D A8 38 F6 2D 71 3C 3D B6 96 1A 6E 13 D8 3B

The highlighted bytes above determine the number of .dat files the game will attempt to read.

In the above case, it is from the 040000 index file, so the game will attempt to read the following

• 040000.win32.dat0
• 040000.win32.dat1
• 040000.win32.dat2
• 040000.win32.dat3

TexTools changes this value to 5, because the fact that it stores all of the modified data in a separate dat file, so that the game now attempts to read

• 040000.win32.dat4

The next relevant part begins at offset 0x800, where the file list begins.

00000800: 7D F3 D5 6F 8F 21 00 00 94 49 5F 02 00 00 00 00

Each entry is separated into 4 parts, each part 4 bytes each.

1. Hash value of file name     Example: v01_c0101e0001_dwn_d.tex
2. Hash value of folder path  Example: chara/equipment/e0001/texture
3. Offset data is located in .dat file
4. Empty

In order to determine which dat file to look in, we take the second digit of the first byte in the offset (in bold above) and divide it by 2.

In the example above:

• 4 / 2 = 2

So we locate the data in 040000.win32.dat2

And to get to the correct offset in the .dat file we fist need to multiply the offset by 8.

In the example above:

• 0x025F4994 * 0x8 = 0x12FA4CA0

So all together we are to go to the offset 12FA4CA0 in 040000.win32.dat2 to locate the data for a file whos hash value is 6FD5F37D and folder hash value is 0000218F.

The Dat File

Now that we determined what dat file and offset to got to we need to know what to expect in the dat file.

More details here: Dat File Research

The first few bytes of the dat file are the most important to determine how to read the data.

4017a980: 80 01 00 00 04 00 00 00 C0 55 05 00 18 03 00 00

4017a990: 18 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00

1: The header length.

2: The type of data of the file.

These are the known file type:

• Type 2: Binary data that can be any number of different things
• Type 3: 3D model data
• Type 4: Texture data

3: The size of the file after is is decompressed

4: Is believed to be the largest size the data buffer can be

5: Is believed to be the actual data buffer size

After these 20 bytes, everything is parsed differently depending on the file type, which I'll go over in their appropriate part of the series.


Part 2 will go over how TexTools fills its TreeView with all the item names and categories directly from the game files.

Responses to some bug reports and feature requests

So I haven't made much progress these last few weeks because a bunch of stuff keeps coming up, family graduations, weddings, reunions, so I've been in and out of town. I swear it's whenever you want to get the most work done that you can least do it. Thankfully all that's over and I won't have to be away for a while.

Anyway, I've been meaning to address some bug reports and other details that have been submitted since I can't reply directly if no contact info was provided so here we go...

I am using a patch which modifies the models and which requires the index2 that was removed by the program when i debugged it for textures not showing up. the file I mean is 040000.win32.index2.
Is there any way to get your texture mod program to work with this index2 file?

I understand that NaughtyWeaver's patch for some odd reason requires the index2 file, I think this is due to the way VRS does things, I can confirm that the patch does not modify anything but 040000.win32.dat0. It's not really TexTools that requires it to be removed, it's that if we leave it alone, some textures you modify will not show up in-game.

So what I'll do is instead of having the .index2 be erased, I'll have the application rename it by adding the extension .bak for backup, so that in case you ever need it again, you can just remove the .bak extension.

So there's support for dye palettes for equipment, but any chance of getting some kind of option for the face paint colors? Is this possible? They're somewhat limited and difficult to do exactly what I want to with.

The colors for anything you set up during character creation seem to be somewhere else, my guess is  chara/xls/charamake/human.cmp, I believe it is possible but I have not messed around with it just yet, but I will try and modify it as it not only would affect face paint, but also skin and hair color.

I was editing a face texture, but when I loaded it up, it crashed in-game. However, when I went to revert it, sometimes I have an issue where a midlander body texture is reverted over the texture in question (a face texture, in this case.)

This means something is wrong with the modlist, when a texture gets imported for the first time, It saves the original offset so that when you revert, it places that original offset back so that it points to the original texture.

When something like this happens the best thing to do is to start over, the thing is, the original offset is lost since the correct one was not recorded in the modlist, this is why I recommend creating a backup of your 040000.win32.index, as that is the only original SE file that gets modified.

I'll have a copy of the most recent 040000.win32.index up for download in case anyone needs it in the future. What you would need to do is replace that 040000.win32.index in the ffxiv folder with the backup, then delete 040000.modlist and 040000.win32.dat3, then start modding again.

It's not working in china server

The offsets are most likely completely different, I was provided with a download link to the Chinese client, I haven't looked into it just yet, but I'll see if I can get it to work.

Issue: Its very difficult to find exactly what assets are used in a specific item, even though tex tools already knowsit internally

What I would like:
1. Open new item with tex tools
2. See the full path in the info box
3. Copy/paste path into data explorer and calculate it from hash (if needed)
4. Navigate to it in data explorer
5. Profit

That should be pretty simple, I'll make sure to get that added for the next version

---

There are other bugs that have been submitted which are much more specific about certain pieces of gear, importing, and crashing, I'll be investigating and fixing these.

There are also some bugs that are very vague and need a lot more specifics, so just as a reminder, be as specific as possible when submitting bug reports, if the application crashed, try and copy paste the detailed error message. If you were working with a specific texture/item which one was it, what steps can I take to reproduce the error if possible.

Also keep in mind it is really important to save the textures in the correct format, which you can see in the tutorial.

3D model progress

Ok, so here's where I'm at with 3D files...

I've successfully extracted the data and put it into meshes as wavefront .obj files, which is a pretty standard format for 3D objects that can be opened and extracted from several different 3D modeling programs.

I was able to then get that same .obj file and import it into the game, putting it in the .dat3 that is used for modded textures with TexTools, meaning that the official model is not modified in any way.

So while that is all well and good, the issue I'm running into is when importing meshes after they've been exported as a .obj from Blender. I think it may have to do with me not having enough experience with Blender or 3D modeling in general, its been a learning experience.


As far as I can tell the problem lies with the vertex normals, the reason I say this is that the original file I extract from the game has one normal per vertex, but when I export from blender without even modifying the file there seem to be less vertex normals.

For example,

The body portion of the Bridesmaids Tights has 100 verts and therefore 100 normals, after exporting from blender, I still have 100 verts but only 92 normals, and the game crashes after I try importing it.

After doing a bit of research, I was able to get it back to 100 normals by switching off the Auto Smooth checkbox

Now that it had 1 normal per vertex I was able to get it to work in-game but then it ends up looking like this

So that's not going to work out at all :(



I decided to then test it with modifying the mesh, I extended the left side by taking a piece from the body and joining it with the gear

I also had to turn off smoothing to get it to work, but then when I went in-game...

The body mesh is the only piece that looks semi-decent the rest is messed up


So I'm trying to figure out how to get this to work correctly, which I think has to do with the vertex normals, because when comparing the file exported from Blender to the original, the only section that is different is the vertex normal section.

Here's the fist 5 as an example:

Blender Exported (with Auto Smooth off)
vn -0.1565 -0.0623 0.9857
vn 0.0000 -0.0027 1.0000
vn -0.1585 0.0072 0.9873
vn 0.0000 -0.1064 0.9943
vn 0.1565 -0.0623 0.9857

Original
vn -0.155273 -0.045380 0.986816
vn -0.000031 -0.001118 1.000000
vn -0.158569 0.007191 0.987305
vn 0.000000 -0.127319 0.991699
vn 0.155273 -0.045380 0.986816

I have a few ideas as to what's going on that I'll be testing, there's also the matter of additional data that is in the file that may need to be calculated like blend weights and tangents, there's also some data that remains unknown.

I'll keep posting updates as I do more tests and figure things out.

Feel free to comment if you have any suggestions, questions, feedback, etc.

Bug Reports

So I've received several bug reports from the form on the website, but it seems I forgot to place an option to put a contact e-mail, and some people asked some questions, I'll do my best to answer some of those questions here, and I'll be placing an optional contact info box on the bug report/feedback form soon.

Thanks to everyone who has been submitting bug reports and sending in feedback, I'll be working on squashing those bugs and updating the application, I'll post some updates on the status of some of the bugs as I get around to them.

Dev Blog

I've decided it's time to set up a developer blog for FFXIV TexTools, because updates are likely to become less frequent as the features remaining to be added increase in complexity.

I'll be sharing some of the progress, successes, and challenges while building up to the next version, you can probably expect some pictures of the changes that go into the application, as well as some in-game pictures of some of the tests I'll be running, and general progress reports.

I may also rant about how things never seem to work out and all seems hopeless, only to post an hour later how everything is magically working and I may have overreacted.

Feel free to post some comments, I always welcome any suggestions and/or feedback.