Hacking the TBL files

Inferis · 20-03-2002, 14:05

I was wondering... Is anyone trying to figure out those TBL files?

I've been trying to, but it's not simple.

I already figured some stuff out, but the lightmaps (for one) keep me puzzled.

Greetz, Inferis

Tayran · 20-03-2002, 16:43

What have you figuered out? Ive tried myself and I have find no hints.

Petrus · 20-03-2002, 17:55

Let us know if you do.

Inferis · 20-03-2002, 23:25

Well, I got boundbox.tbl and idgrid.tbl figured out completely.

Boundbox.tbl contains the bounding box for each province. I guess these are a quick means of figuring out which province is under a mouse click. Since several bounding boxes overlap (the box effectively bounds the "extremes" of the province shape) there needs to be another way of checking the real boundaries. But it's a start.

idgrid.tbl contains 40 regions of maximum 256 provinces. I don't know where these regions cogme from, they don't map to the "region" field in the province.csv file.

When you combine both, you can see the regions nicely on a map. See the screenshots below:

and

(You can clearly see the provinces and TI in north africa here)

I already created a .NET classlibrary to encapsulate the data in both files. The shots above are of a program using that class lib.

I could post the details, but I'd rather wait until I got everything figured out.

As I said, the other files still keep me puzzled, especially the lightmaps.

I do know id.tbl contains 7692 blocks of data that are usually more or less the same. But what those data are, I still have no idea.

incgrid.tbl, incognita.tbl and adjacent.tbl I haven't really looked at, so not much to share there.

Greetz, Inferis

*Txini* · 21-03-2002, 00:10

but the question is, Do you think you can redraw the map??

Inferis · 21-03-2002, 00:36

I honestly don't know. I'm pretty sure the actual map is in the lightmap files, but I'd like to keep those for last, because I think they're closely linked to the other files.

I'm trying to focus on getting the real province boundaries now.

But, I repeat: it's not easy.

*Txini* · 21-03-2002, 15:10

Quote:

Originally posted by Inferis
I honestly don't know. I'm pretty sure the actual map is in the lightmap files, but I'd like to keep those for last, because I think they're closely linked to the other files.

I'm trying to focus on getting the real province boundaries now.

But, I repeat: it's not easy.

I supposse it is not easy but show us every adavnce you make!

Adler · 22-03-2002, 01:37

Yes tell us any progress of this.

Inferis · 22-03-2002, 02:42

Heh, I found out more, but it's useless.

The incognita.tbl contains the shapes of the provinces allright... But of the EU1 map!

Stupid me, I should have noticed that it was the only file with a creation date somewhere in 1998 instead of 2001 like the other ones.

Also, adjacent.tbl contains just that: for every province a list of adjacent provinces, with a flag for each adjacent province if you need to cross a river (which are also a provinces) or not to get there.

So, now incgrid.tbl and the lightmaps remain. id.tbl is partially mined, but needs more work.

I'll keep you posted, but I can't promise anymore progress soon since I'll be very busy doing other stuff the next two weeks!

Inferis.

Adler · 23-03-2002, 00:19

Don't worry we are here waiting for you.

Inferis · 25-03-2002, 11:33

More info, guys.

The id.tbl contains the information needed to know what province is under a mouse (ie when you click). We already had the boundbox info, but that was rather rudimentary.
It's pretty simple: for each line on the map, there is a list of province id's, accompanied by a start and end horizontal location.

So, if you have a coordinate, you need to lookup the correct line (based on the y value), and then loop the list until you found the correct interval (based on the X value). And then you have a province id. Neat.

BTW, i also found out the map size: 18944 pixels wide, 7296 pixels high.

So now the lightmaps remain... I'll keep you posted.

Tayran · 25-03-2002, 19:09

which program did you use? Can you post a link or something?

Inferis · 25-03-2002, 19:32

My tools in finding this out were:

* textpad (binary view), but any hexeditor will do
* windows calculator
* visual studio.net

Basically, i stared at the files in hex mode, looking for clue. When I thought I had something, i started coding a program to view the data as I thought it would be formatted. From there on, I further tested and coded more, until I got the format completely covered. Of course, this took a lot of guessing, especially in the early stages.

That's it.

Inferis · 26-03-2002, 17:54

The lightmaps are revealing their secrets. But very slowly.

There are 5 lightmap files, but in EU2, only 3 are used. Essentially, this means the each zoom level has a separate file. The lightmap4.tbl and lightmap5.tbl seem to contain valid data (I haven't focused on those since they're not being used), but I'm not sure if they're just a leftover from EU1 or not.

Then, the lightmap format itself.
The map (18944x7296) is divided in 32x32 chunks. This means that lightmap1.tbl contains 134976 chunks (normal zoom), lightmap2.tbl contains 33744 chunks (zoomed out 1 level, ie every is twice as small) and lightmap3.tbl contains 8346 chunks (zoomed out 2 levels).

In the beginning of the file is (as most other tbl files do) a table with offset into the rest of the file, for every chunk.

So far so good. But then the information for each chunk is something I have yet to find a *good* clue for. I've been doing some guesswork, and I found out some stuff, but it's still rather inconclusive. The data certainly is not a simple bitmap image.

Anyway, here is a shot: you can see the seas west of the north-american westcoast (I forget their names). The boundaries are clearly visible, but rather crude.

For comparison, this is the same region taken from an in-game screenshot.

I doubt if i'll have more time now, but if/when I find out more, I'll let you know!

Inferis

PS: sorry for the large screenshots!

Lambert Simnel · 26-03-2002, 19:28

Quote:

The lightmap4.tbl and lightmap5.tbl seem to contain valid data (I haven't focused on those since they're not being used),

Allegedly there was atleast one more zoom level in the beta version that never made it to publication because it looked too ugly.

Adler · 27-03-2002, 02:53

Yes, we were told that, thanks for your good work so far Inferis.

*Txini* · 27-03-2002, 11:46

you think you would be able to make changes in the map??
don't forget to make a copy of the file before ;D

Inferis · 27-03-2002, 13:42

Changes... I hope so.

Of course, I don't know how tightly linked the map files are to the EU2 executable. If the links are virtually non-existant then a new world of EU2 modding opens once i get the full file specs figured out. When the links are there, we need to find them out and work with/around them.

Oh well, we'll see. It would be cool to make a completely custom map (think a Europa Mediterranis scenario

), for example.

Inferis.

Adler · 28-03-2002, 22:06

Yes it would be cool.

Licou · 29-03-2002, 00:09

Or deleting the PTI for the AON project

20-03-2002, 14:05	#1
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	Hacking the TBL files I was wondering... Is anyone trying to figure out those TBL files? I've been trying to, but it's not simple. I already figured some stuff out, but the lightmaps (for one) keep me puzzled. Greetz, Inferis __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

20-03-2002, 16:43	#2
Tayran Major Join Date: Jan 2001 Location: Praha Posts: 714	What have you figuered out? Ive tried myself and I have find no hints. __________________ Yes indeed I am powerful The rise of Sweden - a fascist AAR The rise of Sweden 2 - a fascist AAR for HOI2

20-03-2002, 17:55	#3
Petrus Field Marshal Join Date: Apr 2001 Location: Hail Slovakia! Posts: 2,621	Let us know if you do. __________________ "That's like hypnotizing chickens.." --Iggy Pop

20-03-2002, 23:25	#4
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	Well, I got boundbox.tbl and idgrid.tbl figured out completely. Boundbox.tbl contains the bounding box for each province. I guess these are a quick means of figuring out which province is under a mouse click. Since several bounding boxes overlap (the box effectively bounds the "extremes" of the province shape) there needs to be another way of checking the real boundaries. But it's a start. idgrid.tbl contains 40 regions of maximum 256 provinces. I don't know where these regions cogme from, they don't map to the "region" field in the province.csv file. When you combine both, you can see the regions nicely on a map. See the screenshots below: and (You can clearly see the provinces and TI in north africa here) I already created a .NET classlibrary to encapsulate the data in both files. The shots above are of a program using that class lib. I could post the details, but I'd rather wait until I got everything figured out. As I said, the other files still keep me puzzled, especially the lightmaps. I do know id.tbl contains 7692 blocks of data that are usually more or less the same. But what those data are, I still have no idea. incgrid.tbl, incognita.tbl and adjacent.tbl I haven't really looked at, so not much to share there. Greetz, Inferis __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

21-03-2002, 00:10	#5
Txini ¡Por qué no te callas! Moderator Join Date: Apr 2001 Location: Gijón, Kingdom of Asturias, Spanish Empire Posts: 7,795	but the question is, Do you think you can redraw the map?? __________________ Hay quien ha venido al mundo para enamorarse de una sola y determinada mujer, y consecuentemente, no es probable que tropiece con ella.-José Ortega y Gasset Ven a Ierusalem y sigue las aventuras de los Judíos contra el Yugo Romano. - Votaciones Aquí - terminado ¡Ahora, ahora, ahora Quini ahora!

21-03-2002, 00:36	#6
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	I honestly don't know. I'm pretty sure the actual map is in the lightmap files, but I'd like to keep those for last, because I think they're closely linked to the other files. I'm trying to focus on getting the real province boundaries now. But, I repeat: it's not easy. __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

22-03-2002, 01:37	#8
Adler Kung Join Date: Aug 2001 Location: Sweden , Kalmar Län Oskarshamn Posts: 2,678	Yes tell us any progress of this. __________________ * Email No nation was ever ruined by trade. There never was a good war or a bad peace. B Franklin

22-03-2002, 02:42	#9
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	Heh, I found out more, but it's useless. The incognita.tbl contains the shapes of the provinces allright... But of the EU1 map! Stupid me, I should have noticed that it was the only file with a creation date somewhere in 1998 instead of 2001 like the other ones. Also, adjacent.tbl contains just that: for every province a list of adjacent provinces, with a flag for each adjacent province if you need to cross a river (which are also a provinces) or not to get there. So, now incgrid.tbl and the lightmaps remain. id.tbl is partially mined, but needs more work. I'll keep you posted, but I can't promise anymore progress soon since I'll be very busy doing other stuff the next two weeks! Inferis. __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

23-03-2002, 00:19	#10
Adler Kung Join Date: Aug 2001 Location: Sweden , Kalmar Län Oskarshamn Posts: 2,678	Don't worry we are here waiting for you. __________________ * Email No nation was ever ruined by trade. There never was a good war or a bad peace. B Franklin

25-03-2002, 11:33	#11
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	More info, guys. The id.tbl contains the information needed to know what province is under a mouse (ie when you click). We already had the boundbox info, but that was rather rudimentary. It's pretty simple: for each line on the map, there is a list of province id's, accompanied by a start and end horizontal location. So, if you have a coordinate, you need to lookup the correct line (based on the y value), and then loop the list until you found the correct interval (based on the X value). And then you have a province id. Neat. BTW, i also found out the map size: 18944 pixels wide, 7296 pixels high. So now the lightmaps remain... I'll keep you posted. __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

25-03-2002, 19:09	#12
Tayran Major Join Date: Jan 2001 Location: Praha Posts: 714	which program did you use? Can you post a link or something? __________________ Yes indeed I am powerful The rise of Sweden - a fascist AAR The rise of Sweden 2 - a fascist AAR for HOI2

25-03-2002, 19:32	#13
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	My tools in finding this out were: * textpad (binary view), but any hexeditor will do * windows calculator * visual studio.net Basically, i stared at the files in hex mode, looking for clue. When I thought I had something, i started coding a program to view the data as I thought it would be formatted. From there on, I further tested and coded more, until I got the format completely covered. Of course, this took a lot of guessing, especially in the early stages. That's it. __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

26-03-2002, 17:54	#14
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	The lightmaps are revealing their secrets. But very slowly. There are 5 lightmap files, but in EU2, only 3 are used. Essentially, this means the each zoom level has a separate file. The lightmap4.tbl and lightmap5.tbl seem to contain valid data (I haven't focused on those since they're not being used), but I'm not sure if they're just a leftover from EU1 or not. Then, the lightmap format itself. The map (18944x7296) is divided in 32x32 chunks. This means that lightmap1.tbl contains 134976 chunks (normal zoom), lightmap2.tbl contains 33744 chunks (zoomed out 1 level, ie every is twice as small) and lightmap3.tbl contains 8346 chunks (zoomed out 2 levels). In the beginning of the file is (as most other tbl files do) a table with offset into the rest of the file, for every chunk. So far so good. But then the information for each chunk is something I have yet to find a good clue for. I've been doing some guesswork, and I found out some stuff, but it's still rather inconclusive. The data certainly is not a simple bitmap image. Anyway, here is a shot: you can see the seas west of the north-american westcoast (I forget their names). The boundaries are clearly visible, but rather crude. For comparison, this is the same region taken from an in-game screenshot. I doubt if i'll have more time now, but if/when I find out more, I'll let you know! Inferis PS: sorry for the large screenshots! __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

27-03-2002, 02:53	#16
Adler Kung Join Date: Aug 2001 Location: Sweden , Kalmar Län Oskarshamn Posts: 2,678	Yes, we were told that, thanks for your good work so far Inferis. __________________ * Email No nation was ever ruined by trade. There never was a good war or a bad peace. B Franklin

27-03-2002, 11:46	#17
Txini ¡Por qué no te callas! Moderator Join Date: Apr 2001 Location: Gijón, Kingdom of Asturias, Spanish Empire Posts: 7,795	you think you would be able to make changes in the map?? don't forget to make a copy of the file before ;D __________________ Hay quien ha venido al mundo para enamorarse de una sola y determinada mujer, y consecuentemente, no es probable que tropiece con ella.-José Ortega y Gasset Ven a Ierusalem y sigue las aventuras de los Judíos contra el Yugo Romano. - Votaciones Aquí - terminado ¡Ahora, ahora, ahora Quini ahora!

27-03-2002, 13:42	#18
Inferis TBL File Hacker Join Date: Jul 2001 Location: EU3 beta cage Posts: 1,574	Changes... I hope so. Of course, I don't know how tightly linked the map files are to the EU2 executable. If the links are virtually non-existant then a new world of EU2 modding opens once i get the full file specs figured out. When the links are there, we need to find them out and work with/around them. Oh well, we'll see. It would be cool to make a completely custom map (think a Europa Mediterranis scenario ), for example. Inferis. __________________ Master TBL File Hacker » Site · Thread Magellan Creator » Download · Info · Thread EU3 Beta

28-03-2002, 22:06	#19
Adler Kung Join Date: Aug 2001 Location: Sweden , Kalmar Län Oskarshamn Posts: 2,678	Yes it would be cool. __________________ * Email No nation was ever ruined by trade. There never was a good war or a bad peace. B Franklin

29-03-2002, 00:09	#20
Licou Phobophile Join Date: Jun 2001 Location: Saint-Bruno-de-Montarville (NOT LONGOEIL) - Québec, Canada Posts: 1,337	Or deleting the PTI for the AON project __________________ Don't miss my smily site for the Europa Universalis forum!