Tropico 3/4/5 HPK Archiver

[FordGT90Concept]

HPK Archiver is no longer receiving updates. Try nicklec's "hpk" instead.

Original content follows:
---------------------------------------
This application can extract and archive data into and out of the HPK formatted file.

You need .NET Framework 4.6.1 or Mono to run it. Extract before running or else it won't be able to find LZ4.dll.

Works with:

• Grand Ages: Rome
• Tropico 3
• Tropico 4
• Omerta: City of Gangsters
• Tropico 5
• Victor Vran
• Surviving Mars

Spoiler: Start Up Arguments/Console

Example, argument command:
-extract "D:\Steam\SteamApps\common\Tropico 4\dlc\Expansion.hpk" "C:\Users\Admin\Desktop\Expansion" -wait

Produced this result:

Commands currently supported (you should be able to do many at once)...
-extract "path to HPK" "path to folder to create/fill"

-archive "path to folder to pack (will be consumed)" "save HPK here"
Optional: Adding -zip to -archive will pack it. Do not do this on Tropico 5 because it won't be able to load it. Note: -zip only has to appear once to apply to ALL -archive commands after it. For example -archive INPUT 1.hpk -zip -archive INPUT 2.hpk would cause 1.hpk to be unzipped and 2.hpk to be zipped.

-unzip "path of file to unzip" "place to save unzipped file"

-wait
Special: This command only needs to appear once and if it does, it will stop the console from closing and show "Press any key to close program."

If no console arguments, the console will hide and the GUI will pop up.

Spoiler: Version History

1.1.0: Significant rewrite of code to add support for LZ4.
1.0.14: Archiving would have problems when the deflated size equals chunk size (usually 32768 bytes).
1.0.13: Fixed issue with small files giving inaccurate message/doing stupid things, no longer crashes when encountering a file with no data, and displays message when processing file dates.
1.0.12: Fixed 1333 out of 1334 errors in Tropico 4: Modern Times' Expansion.hpk
1.0.11: Writes data to output that doesn't inflate (fix for DLC).
1.0.10: Separated Tropico 5 game option which disables "Compress It" when checked because the game doesn't support it.
1.0.9: Lots of minor tweaks in the code.
1.0.8: Cleaned up file system code (part of FragmentedFile now) and renamed a lot of variables to more accurately reflect what it represents.
1.0.7: Catches some files that lie about being compressed and dumps the headerless contents.
1.0.6: No longer creates a temporary file for fragmented files and handles all files as fragmented files.
1.0.5: Removed a temp file and cleaned up code relating to all of the temp files (three of them now).
1.0.4: Now handles fragmented files properly.
1.0.3: Tons of bug fixes including the ability to decompress/extract challenges and ShaderCache.hpk.
1.0.2: No longer crashes if the output directory doesn't exist. It will simply create it.
1.0.1: Added bug fixes for abnormal files in Game.hpk.
1.0.0: Fixed a bug with the time stamps when archiving and also added support for Tropico 4 (experimental).
Beta 7: Fixed a bug in the archival code.
Beta 6: Archive feature is now functional, it automatically saves/loads settings, and many other tweaks.
Beta 5: Overhauled the GUI. It now shows statistics about the file and how far along it is while extracting.
Beta 4: It applies the "_filedates" to the files (modified date) and deletes the _filedates file because it is now irrelevant. Made the Zip decompression algorithm more efficient (one output stream per file).
Beta 3: Added Unix support for use on Mono.
Beta 2: Extracts all files successfully via C#ZipLib.
Beta 1: Erred on some files during extracting using zlib.

Implements:

• lz4net for handling LZ4 compression.
• SharpZipLib for handling ZLIB compression.

 

[FordGT90Concept]

As far as I can tell, Extracting is complete in beta 4. Next is archiving, followed by multithreading and other GUI improvements.

 

[ShiBDiB]

i dont get why we'd need this

 

[FordGT90Concept]

If you want to mod the game, you need it.

Beta 5 uploaded. I'm going to start work on the archiving component soon.

 

[FordGT90Concept]

Beta 6 upload. The application is now fully functional and fully featured.

 

[sharpegg]

Thanks

Thanks for this great tool. I was able to extract the music from the game using this.

 

[FordGT90Concept]

Heh, finally someone uses it. XD

You are welcome.

 

[madames]

HEY FORDGT90!

This is going to make you double-take, but it's Michael Adames. I used to moderate Hardware Analysis. Ya.

Big Tropico 3 player here, really enjoying the game! Found myself quite bothered with El Presidente's annoying sayings as well as some of the ambient files such as the seagull and rooster.

This tool worked perfectly for me and did just as you said. This has made the game infinitely more enjoyable, thanks again.

When I downloaded this and saw your name I was like "OH SHI---" clicked on your name and recognized a few people from Hardware Analysis and knew it was you. Hope all is well with you. Thanks again for the tool.

 

[FordGT90Concept]

Heh, yeah I remember you. Stick around.

 

[Ignasius]

Hello, thank you for this program, but I need two components to the archive file .hpk someone advise me how to do it?

Please reply. Thank


I need to archive two folders.

How do I do?

it must be in format .hpk

 

[FordGT90Concept]

selected folder
  -> folder 1
    -> file 1
    -> file 2
  -> folder 2
    -> file 1
    -> file 2
  -> file 1
  -> file 2

I'm pretty sure "selected folder" is consumed (it is implied in the format but doesn't actually exist). Just make sure to have multiple subfolders in there and it should work fine.

 

[Ignasius]

Please, try to decompress and re-archiving.

Please try this file. HPK
unpack and wrap

http://www.megaupload.com/?d=S2D8ROSO

after unpacking, you will receive two components

 

[FordGT90Concept]

It says it is unavailable.

 

[Ignasius]

download here: http://no-mercy.byl.cz/Data.hpk

 

[FordGT90Concept]

Ah, I see the problem. Archive should not include the given directory, only everything inside. For instance, if you extracted to C:\Stuff and you archived the contents of C:\Stuff, "Stuff" shouldn't appear in the output HPK file where currently it does.

I am working on it and will get a fix up as soon as possible but it could take a week or two.


Edit: It was easier to fix than I thought but I'm not 100% certain Tropico will like it. I attached the modified version to this post. Try it out and let me know if it works as expected or not (specifically, will Tropico read the outputs from it).

 

[Ignasius]

Yes, this correction is already functional

Thank you so much, now you can, pack more components into a file .hpk

game is fully functional, even after adjusting file
I'm working on another language version in this game

thanks very much
Sincerely, Ignasius


Czech Republic

 

[kenji_03]

Music Pack

To make sure you know your work is appreciated I wanted to let you know that your tool is the only way to add music to Tropico 3 and that I am very grateful for your efforts.

I've been playing Tropico 3 for just a few hours and already the (offensively small) music track is getting repetitive, so you just made T3 infinitely more fun by allowing me to add more music to the game.

I am at work right now, but as soon as I get home I'll be installing this program and adding more songs to the game. Thank you!

If you happen to have your own, custom, Tropico3 music pack I'd LOVE for you to upload it btw.

Edit: I almost forgot, people on Steam Forums appreciate your work too btw.

 

[FordGT90Concept]

I don't have a custom music pack but I know there are instructions on the official forum for how to add Tropico (original game) music to Tropico 3 if you want to do that.

 

[FordGT90Concept]

Updated to support Tropico 4, theoretically.

Edit: FYI, the only difference is in the _filedates file. All the filetimes in Tropico 3 are standard Windows FileTimes. In Tropico 4, it appears they divided them by 2000 so that's what I did. The reason why it crashed was because without multiplying by 2000, it would produce an invalid filetime so when it tries to work with it when archiving, the numbers would be all wrong. It was a pretty easy fix but a very strange change they made there.

 

[zbug]

It's a nice tool you've made, but it seems it cant extract big files (1mb+), or is it only bugging on my computer? Been trying to extract some hpk extension files in AppData/Roaming/Tropico 4/Challenges.

 

[FordGT90Concept]

I don't even have an Challenges folder. I'll need specifics in order to debug it (how you obtained one that fails and the name of it).

 

[Frejoh466]

Are you going to update it so it works with modern times? Trying to extract Game.hpk but the program just keep crashing all the times.

 

[FordGT90Concept]

I don't have Modern Times yet (in transit). I could take a look at it when it arrives.

 

[FordGT90Concept]

I think I see the problem. There is a file (Game\CommonEntities.txt) that is...messed up. The file is supposedly compressed but compressed files require at least 16 bytes just for the header and more for the data. This has 12 bytes total (header + data) making my algorithm run into the next file entry causing it to eventually crash when. The data literally is: 0000000000800000

That 8 in there is suspicious (value of 128) but I don't know if that belongs in the file or if it is part of the broken header. I think what I'm going to do is simply create an empty file and hopefully all will be good.

Will edit this post soon...


Edit: The 8 is the chunksize which is more or less always set to 32768. In other words, it is, in fact, a blank file.

There's a problem with another file...a batch file (HGO\SRP\srp-init.bat) which has the ZLIB header for compression but is, in fact, not compressed. I'm trying to find a solution that problem now. Other than those two hiccups, it appears to be working fine on Game.hpk.


Edit: To address the batch problem, it looks for that file specifically, checks if it is 70 bytes in length, compares it with what it stores internally, if a match, it cuts off the ZLIB header and outputs the remaining 54 bytes. If not a match, it outputs it normally.


In both cases, I would think that Tropico would crash if it tried to access them. Fortunately, the batch file looks like a dev tool and the text file is useless so it doesn't access them. I'll update the op with the updated executable shortly.

 

[Frejoh466]

Thanks for the update, and for the great tool.