Tropico 3/4/5 HPK Archiver

[R-T-B]

Going to do that soon, just taking some time to play the base game and get a feel for it first... then maybe I can make some of my own mods too.

 

[FordGT90Concept]

I took at that English.hpk file jianeddie uploaded and it appears the problem is that it isn't inflating. CSharpZipLib is returning 0 (error).

Edit: My best guess is that ICSharpZipLib doesn't match whatever they used in Tropico 5. It appears to error at chunk 10.

Edit: zlib1 only gets to 8.

Edit: Tried DotZLib wrapper for zlib1 and it causes vshost32-clr2.exe to stop working. I've never seen that happen before.

Edit: Ionic.Zlib.ZlibStream is getting to 10 too. I've tried multiple compression modes in this one too and all yield the same result.

By the way:
8 = 7 * 2^15 = 229,376 bytes
10 = 9 * 2^15 = 294,912 bytes

Here's the first five lines of that:

ID,Revision,Text,Location,Context,Actor,Voice Actor,Old Revision,Old Text,Old Translation,New Translation,Status,Warnings,Xbox Terminology/Whitespace/Status Warnings
1000001,174226,Common,CommonLua/Classes/ComponentObject.lua(00015),,,,174226,Common,Common,Common,OK,,
1000002,174226,Default,CommonLua/Core/EngineOptions.lua(00003),options refresh rate,,,174226,Default,Default,Default,OK,,
1000003,174226,<arg1> x <arg2>,CommonLua/Core/EngineOptions.lua(00233),screen resolution,,,174226,<arg1> x <arg2>,<arg1> x <arg2>,<arg1> x <arg2>,OK,,
1000004,174226,Windowed,CommonLua/Core/EngineOptions.lua(00243),Options dialog resolution setting,,,174226,Windowed,Windowed,Windowed,OK,,

The error is: Bad state (unknown compression method (0x0D))
Which isn't surprising...

Edit: I'm beginning to think it isn't the compression, it is the ZLIB header. It is suggesting there is way more data in there than there really is.

Edit: Hmm, I think I found a bug that affected all versions. It would cause the compressed file to be smaller than it really is if extracting failed. Still nothing in terms of progress on decompressing.

 

[yoysef]

I hope you can strip the text of hpk file ...
I have followed since the tropic 1 ....
liked all ..
Some for here in Brazil is impossible to play without translation hope I can really fix the HPK archiver!

 

[FordGT90Concept]

I found a major problem. What it is doing in the for loop on line 125 of HpkFile is wrong because the file isn't continuous. It is broken up. The compression likely errors when hitting one of those breaks. It's going to be difficult to fix but it may fix retroactively too.

Edit: Note the gap between [4] and [5], [5] and [6], and [7] is completely out of order (belongs between [4] and [5]):
[0] {92, 8751}
[1] {8843, 4910}
[2] {13753, 3750}
[3] {17503, 4474}
[4] {21977, 12716}
[5] {45968, 33596}
[6] {85603, 481839}
[7] {34693, 2512}

I think the easiest solution will be to create a temporary file that is one continuous file then sick the inflater on it and hopefully it works out.

Edit: I think this hopefully means archive code is fine. It removes all the garbage and straightens the file out.

 

[jianeddie]

Maybe more files can help?
this crash : http://1drv.ms/1nEVlxR
this ok : http://1drv.ms/1nEVG3A

Sorry for nothing I can help.

 

[FordGT90Concept]

Success! All three files (English, Fonts, Data) appear to have extracted/inflated. Downloads are in the original post.

I coded the fix for speed, not compatibility. It takes a separate code path if the file is discovered to be fragmented. This can cause erratic bugs. If there is demand, I can change it over to maximum compatibility but it will substantially increase the hard drive load. It would also need thorough testing going all the way back to Tropico 3.

I did not touch the archival code. I see no reason why the old code wouldn't work.

 

[jianeddie]

Well done! Thanks for update!

 

[FordGT90Concept]

The bandage fix is driving me nuts. I have a good idea how to fix it without any performance penalty but it will require recoding all of the extraction code as well as a custom class for handling fragmented files. No guarantees it is going to happen and I probably won't say anything about it until it is done should I decide to do it.

 

[nmihaiv]

I have been looking for a way to mod the houses in the game, from what I see you are the only one working on a way to get any modding done.

Would help if i had any idea how i could help, but i wish you the best in your attempt to make the editor work

 

[FordGT90Concept]

The bandage fix is driving me nuts. I have a good idea how to fix it without any performance penalty but it will require recoding all of the extraction code as well as a custom class for handling fragmented files. No guarantees it is going to happen and I probably won't say anything about it until it is done should I decide to do it.

Abandoning it. It gets way too complicated way too fast. Complicated means a lot of opportunity to error so it wouldn't actually accomplish anything.

I have been looking for a way to mod the houses in the game, from what I see you are the only one working on a way to get any modding done.

HPK Archiver should be able to get you access to the models but I can't help beyond that.

 

[nmihaiv]

Abandoning it. It gets way too complicated way too fast. Complicated means a lot of opportunity to error so it wouldn't actually accomplish anything.


HPK Archiver should be able to get you access to the models but I can't help beyond that.

Will download it, and play around with it a little ... hopefully won't break my game

Thanks a lot, and cheer up, don't give up, take a few days break before starting again

 

[FordGT90Concept]

So I got to thinking my approach was overcomplicated so I rethought it. Mission accomplished. It no longer defragments files before unzipping them. It defragments as it unzips/extracts. This appears to have fixed the HGO\SRP\srp-init.bat from Modern Times too. It did not fix the 1300+ errors here but I'm still 99% convinced they're simply broke.

Anyway, see the original post for the latest version.


Edit: My theory didn't pan out. There is a chance that directory information could be fragmented as well so I checked Expansion.hpk for directory fragmentation (actually, fragmentation of any kind). There was none. Still 1334 errors. Looking at the binary, I'm still convinced they are defective. The file is supposed to inflate to about 16 KiB but the data in the header suggests it is much, much larger. Then there's only three chunks when each chunk can't expand to more than 2^15. Long story short, the file length is wrong and the offsets in the ZLIB header are wrong. They're broken files.

 

[R-T-B]

I did not touch the archival code. I see no reason why the old code wouldn't work.

It works in that it does pack files, and unpack them. Tropico 5 however fails to generate a playlist from a archive packed full of Tropico 1's music. It plays the first song and stops forever... very weird.

Interestingly, simply deleting Tropico 5's Music.hpk and creating an external directory in the program folder called "Music" allows you to create your own playlists and it even randomizes them. It will play opus files natively, but also accepts .wav's and .oggs. Might play mp3's too.

EDIT: Nope, it doesn't. Probably didn't license it.

If anyone wants the old Tropico 1 music in wave format, PM me. It's the best way of doing it as converting MP3s to another supported lossy format like Opus is a really bad idea from a quality perspective...

 

[FordGT90Concept]

So is the archival code not working for Tropico 5 or does Tropico 5 just not like the modified files in the archive? Ehm, is this something I need to look into in the archival code?

 

[nmihaiv]

Archiving worked well for me.

 

[R-T-B]

So is the archival code not working for Tropico 5 or does Tropico 5 just not like the modified files in the archive? Ehm, is this something I need to look into in the archival code?

I don't know tbh. It's odd because the exact same files in an external directory work fine, but in the archive in the same format they don't. I don't know why that would be but it's probably not a big worry for most folks.

 

[Zur13]

So is the archival code not working for Tropico 5 or does Tropico 5 just not like the modified files in the archive? Ehm, is this something I need to look into in the archival code?

Thanks for your work. If this could help

I've tried to unpack Lua.hpk from Tropico 5 and repack it back without changing anything.
After unpack error log with this line appeared:
G:\Tropico 5\Packs\Unpacked\Lua.hpk\Lua\Config\_LuaRevision.lua Could not decompress the file.

And _LuaRevision.lua was empty.

First i've tried to ignore this and just repack all back to hpk and launch the game with debug. The game has failed to start and in the the debug console was smth about lua revision 1815!= 181599... i don't remember the exact message.

I've made small changes in your code so if it fails to unzip file it writes it as is (i.e. zipped).
Zipped _LuaRevision.lua contains this line after zip header "LuaRevision = 181599".

And I've changed ArchiveZippedFile(ref BinaryWriter bw, int index) method so it checks if input starts with "ZLIB" it calls ArchiveFile(ref bw, index).

But the trick failed after repack new hpk file has less size and the game fails to start without error messages in debug log.

Lua.hpk http://rghost.ru/56047154

Modified HpkFile.cs http://rghost.ru/56047232 (my edits marked with //ZUR: )

 

[FordGT90Concept]

On my end, _LuaRevision.lua is not empty. It contains:
ZLIB
20 byte long
32768 chunk size
16 offset of data
Data:
LuaRevision = 181599

It's one of those files that lies about being compressed. I'll see if I can find a logical solution but if I can't, I might just have to write an exception for that file.


Edit: 1.0.7 up. The change I made is that, if decompression fails, it checks if there is only one chunk and the resulting file size is supposed to be less than 32 bytes in length. If both conditions are true, it dumps the contents of the file without the ZLIB header. If they are not, it dumps the whole file (ZLIB header and all).

I checked the archival code and it already was not compressing files less than 256 in length so the contents of this decompressed file remain decompressed in the output HPK. That is to say, it fixes the file. Haemimont Games' tools apparently compress all LUA files regardless if they should or shouldn't be compressed. If ZLIB fails, they keep the ZLIB header and simply write the uncompressed contents to the rest of the file instead of eliminating the ZLIB header.

 

[FordGT90Concept]

Uploaded 1.0.8. It started out as trying to find out if Tropico 5 added anything new to the header (doesn't appear so) but I ended up cleaning a lot of older code so that it is much easier to follow and/or more appropriate.

In addition to the above, I updated my internal "ReadHeaders" program. Could someone that has Tropico 5 run it? The first argument in the commandline needs to be the path to the Tropico 5 folder. The second argument needs to be where to save the results (e.g. a text file on the desktop). If someone could attach the output file to a reply, that would be awesome. I'm thinking about putting the list in the original post for Tropico 3, 4, and 5. Mostly, I want to just take a look at it to make sure there are no surprises. Make sure ReadHeaders and HPK Archiver are in the same directory when you run it. ReadHeaders pulls a lot of stuff from HPK Archiver.

[removed attachment]

 

[spectatorx]

Awesome tool! Time to mod El Presidente and his country

I was always dreaming about such tool for any assassin's creed game...
FordGT90Concept, could you try to create one?

 

[jianeddie]

Uploaded 1.0.8. It started out as trying to find out if Tropico 5 added anything new to the header (doesn't appear so) but I ended up cleaning a lot of older code so that it is much easier to follow and/or more appropriate.

In addition to the above, I updated my internal "ReadHeaders" program. Could someone that has Tropico 5 run it? The first argument in the commandline needs to be the path to the Tropico 5 folder. The second argument needs to be where to save the results (e.g. a text file on the desktop). If someone could attach the output file to a reply, that would be awesome. I'm thinking about putting the list in the original post for Tropico 3, 4, and 5. Mostly, I want to just take a look at it to make sure there are no surprises. Make sure ReadHeaders and HPK Archiver are in the same directory when you run it. ReadHeaders pulls a lot of stuff from HPK Archiver.

here is the result file: http://1drv.ms/1pIB3ou

 

[Zur13]

Uploaded 1.0.8. It started out as trying to find out if Tropico 5 added anything new to the header (doesn't appear so) but I ended up cleaning a lot of older code so that it is much easier to follow and/or more appropriate.

In addition to the above, I updated my internal "ReadHeaders" program. Could someone that has Tropico 5 run it? The first argument in the commandline needs to be the path to the Tropico 5 folder. The second argument needs to be where to save the results (e.g. a text file on the desktop). If someone could attach the output file to a reply, that would be awesome. I'm thinking about putting the list in the original post for Tropico 3, 4, and 5. Mostly, I want to just take a look at it to make sure there are no surprises. Make sure ReadHeaders and HPK Archiver are in the same directory when you run it. ReadHeaders pulls a lot of stuff from HPK Archiver.

Got results too

 

[FordGT90Concept]

I was always dreaming about such tool for any assassin's creed game...
FordGT90Concept, could you try to create one?

Have you tried this one? Disclaimer: I have never used it so don't consider this a recommendation:
http://www.moddb.com/groups/assassins-creed-fans/downloads/forge-extractorreplacer-by-turfster

here is the result file: http://1drv.ms/1pIB3ou

Many thanks. It doesn't look like there are any surprises in there other than a general lack of cleanliness. In Tropico 4, for example, only about 3 files were fragmented and "fragments per file" reflected that. Now they have fragments specified even if they don't use them. Additionally, the amount of residual data is staggering. They should have repacked everything before shipping to eliminate all the wasted space. Oh well, it doesn't look like there's anything more I need to do at this time.

 

[Zur13]

Uploaded 1.0.8. It started out as trying to find out if Tropico 5 added anything new to the header (doesn't appear so) but I ended up cleaning a lot of older code so that it is much easier to follow and/or more appropriate.

In addition to the above, I updated my internal "ReadHeaders" program. Could someone that has Tropico 5 run it? The first argument in the commandline needs to be the path to the Tropico 5 folder. The second argument needs to be where to save the results (e.g. a text file on the desktop). If someone could attach the output file to a reply, that would be awesome. I'm thinking about putting the list in the original post for Tropico 3, 4, and 5. Mostly, I want to just take a look at it to make sure there are no surprises. Make sure ReadHeaders and HPK Archiver are in the same directory when you run it. ReadHeaders pulls a lot of stuff from HPK Archiver.

I've tried to unpack the Lua.hpk and repack it with v.1.0.8. Tropico 5 not starting with repacked Lua.hpk (repacked size 3 667 848 bytes, original Lua.hpk size 3 726 025 bytes).

 

[FordGT90Concept]

I've tried to unpack the Lua.hpk and repack it with v.1.0.8. Tropico 5 not starting with repacked Lua.hpk (repacked size 3 667 848 bytes, original Lua.hpk size 3 726 025 bytes).

Is it the same error or something else?

They aren't going to repack the same size because...
1) I eliminate fragmentation (1/8 the size of the file system).
2) I eliminate residual data (in Lua.hpk, that is 1029 bytes).
3) I don't necessarily compress the same file types.
4) I have a minimum file size for compressing.
5) I added _filedates which preserves the "last modified" time.

Anyway, I did more tinkering (none of it should have impacted the file size) but I get 3,718,144 bytes, not "3,667,848 bytes." I'll upload it (1.0.9) shortly.