1. Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Need help compressing and archiving

Discussion in 'General Software' started by tidyboyd, Feb 10, 2013.

  1. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    I have a 1TB collection of stuff (mainly .avi's and some pdf's) I would like to compress down into as small amount as possible for blu ray backup, heres my problem

    I've already compressed the 1tb down (removing 100gb) with 7Zip using LZMA2 with the maximum amount of other selections the program gives you for compression apart from the file being non-solid. After reading about compressing I thought I could open the .002 and .003 etc 7-zip parts individually but from trying it seems I have to open the first file .001 otherwise none of the others will open.

    My main issue is that I really need the files to be openable individually without the need for the first file just in case something goes wrong with the first bluray disc, I would be stuck with no way to open my archives if any damage or loss was to come to that specific disc.

    So my question is really is there any way to make each archive openable individually or if not with 7zip is there another program that can?

    Also is there any way of compressing the files even more ie making .avi's into MKV's? Would this save space even with .avi clips? and if so is there a batch application to convert them that you know of?

    Any help on this would be massively appreciated as this is my most important and cherished data
     
  2. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    Is there nobody that has any idea on this? Ive been looking but cant find anything :(
     
  3. Fourstaff

    Fourstaff Moderator Staff Member

    Joined:
    Nov 29, 2009
    Messages:
    9,197 (5.14/day)
    Thanks Received:
    1,982
    Location:
    Home
    You will have to zip each file individually, I'm afraid. Otherwise 7zip will not be able to get all the parts it want. I am not sure if there are any advantage in changing avi into MKV, and I am not sure how to compress further without detail loss. I used to use Handbrake, but there are probably better alternatives.
     
    tidyboyd says thanks.
  4. McSteel

    McSteel

    Joined:
    Nov 19, 2012
    Messages:
    594 (0.84/day)
    Thanks Received:
    284
    You could use UHARC for this, it's able to compress multimedia to a certain degree, but it really depends on the nature of your files. Of course, it does everything losslessly. The other alternatives would be FreeArc or CCMX, but that's about it. 7zip doesn't do multimedia compression at all.
     
    tidyboyd says thanks.
  5. digibucc

    digibucc

    Joined:
    May 21, 2009
    Messages:
    4,925 (2.48/day)
    Thanks Received:
    1,493
    you can use uharc, but imo it won't be worth it. you may be able to compress the pdfs more but the videos are what they are. they are already compressed as tightly as current technology allows, otherwise they would be tens of gigs in size instead of what they are.

    what compression really does is remove information with an algorithm that tells it how to restore said information. documents like txt, doc, pdf etc have a lot of whitespace so this is easily done. media like pictures, videos & sound do not have that whitespace. they are already compressed (stripped of as much information) as they can be without sacrificing quality.
     
    Chevalr1c and tidyboyd say thanks.
  6. newtekie1

    newtekie1 Semi-Retired Folder

    Joined:
    Nov 22, 2005
    Messages:
    20,043 (6.15/day)
    Thanks Received:
    6,104
    I'd be better to just buy a 1TB external and use that, probably more cost effective in the long run too.
     
    Chevalr1c, Aquinus and digibucc say thanks.
    Crunching for Team TPU 50 Million points folded for TPU
  7. Kreij

    Kreij Senior Monkey Moderator Staff Member

    Joined:
    Feb 6, 2007
    Messages:
    13,881 (4.93/day)
    Thanks Received:
    5,616
    Location:
    Cheeseland (Wisconsin, USA)
    You can get 1TB portable, external USB hard drives that will fit in your shirt pocket for under $100.
    Buy two. Make two copies. Keep one at home, put the other somewhere offsite.

    You will be able to put the data on BD-R cheaper (you can find them for about $1 each if you look), but you are creating an unnecessary archival management nightmare for yourself.

    If you are still not comfortable with that level of redundancy (it is after all your most treasured data), get a web site with unlimited storage and use it for nothing but cloud based backup. The hosting company will back it up for you as part of the deal. ;)
     
  8. lemonadesoda

    lemonadesoda

    Joined:
    Aug 30, 2006
    Messages:
    6,260 (2.10/day)
    Thanks Received:
    967
    Here's my tip:

    1./ Don't backup the .avi files... since you own the original DVD/Blurays already ;)
    2./ If your PDFs are professional publications, you won't be able to shrink them better than what 7zip can do. But if they are scans, you can use something like PDF Compressor (IRIS) to get then down to about 10% of their size

    Yes, you could re-encode your .avi files, but you won't save much unless you reduce resolution / increase compression. I don't think you want to do that. I wouldn't think a 7zip archive of an .avi is going to save much... so why not just archive the raw .avi files individually.

    This whole operation is going to take you SO LONG. If you value your time at more than $0.10 per hour, then do as newtekie said... get yourself an external HDD for your archive. (NAS box or HDD rack). See the news sections on this website for ideas.
     
  9. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    Thanks for all your replys there most kind, so just to answer a few of those suggestions

    I do currently have 2 x 3tb drives with another 3tb external that I'm borrowing from the Mrs as a third backup at the moment. In all reality I just want this data to be as secure as humanly possible and as I've found out recently no hard drive lasts forever (2 x seagate 3tb went down the pan within a day of each other) so I would like to have a hard copy in my hands hence the dual layer blu rays I have and the 256gb SD cards I've got an idea to buy.

    My aim is that the data needs to be as small as possible, readily transportable (very rugged) and each archive must not dependent of each other.

    I also really need a software program that will break all my files and folders down to manageable sizes even if there not zipped up as now I've got it all organised, quite a few folders stretch past the 44gb dual layer blu ray limit, is there any such program?

    So there is literally no way of opening a 7zip part file without the first corresponding header archive? Absolutely no program that can perform that operation that anyone knows of?

    Just tried to use UHARC and it has the same limitation as KGB Archiver but more stringent on file size as UHARC only allows 2gb and KGB only allows 4gb. This is a real problem as some of the files in there are more than twice as large leading to errors from the get go with both programs

    I'm not really after cost effective on this one, I'm literally after the best method of storing files that is transportable and is sure to arrive unbroken at the destination. On a side note I was even considering using these new 1.5tb and 3tb tapes I've seen floating around for archiving important data, any input on those? Better than SD cards? Are they really that rugged? Any weaknesses?

    Again any ideas or comments would be hugely appreciated :)
     
  10. lemonadesoda

    lemonadesoda

    Joined:
    Aug 30, 2006
    Messages:
    6,260 (2.10/day)
    Thanks Received:
    967
    Have a look at what WinRAR calls "Recovery volumes" which will help you with ONE of the problems you mentioned: " in case something goes wrong with the first bluray disc, I would be stuck with no way to open my archives if any damage or loss was to come to that specific disc. "
     
    tidyboyd says thanks.
  11. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    Thats appreciated, I'm checking it out right now, thanked
     
  12. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    Ok had a brief check through on recovery volumes and that's a great feature I will most definitely add to my archive, thank you for that. I suppose this doesn't necessarily mean each archive will open individually but more in case some damage befalls one of archives?

    Can you tell me then, whats the difference between recovery volumes and recovers records? I'm searching around for info and its either a little thin on the ground or its a bit over my head, they seem to relate to the same thing and do the same job yet there are 2 different options in Winrar, sorry for what must be n00b questions, in all honesty this is the first time I've ever tried to archive anything digital :)

    I will continue reading for the answer, I'm not that lazy :)
     
  13. lemonadesoda

    lemonadesoda

    Joined:
    Aug 30, 2006
    Messages:
    6,260 (2.10/day)
    Thanks Received:
    967
    http://www.rarreg.com/users_manual.php

    Recovery records are used WITHIN a single archive file, to provide data recovery through data parity methods. You can define the amount of parity data the archive stores. Think of this as a CD backup where the CD is scratched or HDD is unreadable at some sector. The recovery records will reconstruct the missing/corrupt data.

    Recovery volumes are used ACROSS multi-file archives, to provide archive recovery if one or more of the split volumes gets lost or corrupted. You can define how many recovery volumes are created. Depending on the number of recovery volumes you create, you can "lose" that many original archive volumes... or thereabouts. Think of this as multi-CD backup and one of the CDs gets lost or totally screwed. You can rebuild the missing CD/archive from the recovery volume.

    You can also think of recovery volumes as "RAID over CD" if you catch my drift.
     
    tidyboyd says thanks.
  14. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    This is fantastic, I'm very happy to have asked the original question now, this is exactly what I was after. I'll get zipping up now and see what comes out the other end. I'll add both to the mix as a multi layered backup procedure. Just another quick one if I may, will the recovery volume take up as much space as the original 44gb archive I intend to break them up into?

    Heres some rockouts as this is how im feeling :rockout::rockout::rockout::rockout::rockout:
     
  15. lemonadesoda

    lemonadesoda

    Joined:
    Aug 30, 2006
    Messages:
    6,260 (2.10/day)
    Thanks Received:
    967
    Let's say you break that nearly 1TB of data into 20x 44gb bluray discs. If you do a SINGLE recovery volume, you will need an additional 1x 44gb recovery archive. That will allow you to recover from one damaged or lost archive. 1 recovery disc to protect 20 discs. ONLY ONE of those 20 can get lost or damaged. If you want to increase you backup chances, then you could create 2 or 3 recovery volumes... allowing up to 2 or 3 of those original archives to get lost or damaged.

    You could just stick with recovery volumes. It's the simplest. But if you were to have a faulty DVD/Bluray drive or faulty batch of discs that were consistently creating some bad sectors, then ALL the discs might be problematic... in which case the use of recovery records on each disc in addition to the recovery volumes is a good idea.

    Does that sound odd? Well no. When CD writers first came out... I had an external SCSI CD writer... built like a tank. It could read and write anything IT WROTE ITSELF. But it didn't like all brands of CD blanks. There were particular brands it preferred. Especially with the read/write discs. And if you took a disc to a different PC with a different CD drive it would sometimes not read all sectors properly. Things have improved a lot... but it can still happen.

    Now, getting back to what some people said earlier... creating those archives and backup volumes, and then burning 21x discs or more... it is going to take you a long time. If you want to do that regularly then getting a 1TB external will not only be cheaper in the long run... it will save you a LOT OF TIME.

    ####

    I backup regularly. I have a dedicated HDD for backups. A 1TB drive is cheap today. Even if I cost my time at just $1 per hour... the cost of the discs and the time creating and burning archives means that the better option is a 1TB HDD.
     
    tidyboyd says thanks.
  16. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    This is duly noted, its a one time backup that I would like to keep for 10-20 years so the blu ray is the way I'm going even if it does take me a while but I appreciate your thoughts.

    The archive is zipping up now with 10% recovery record and 3 recovery volumes, its got a day and a half to go so lets hope for the best, ill let you know how it turns out :)
     
  17. cheesy999

    Joined:
    Jul 2, 2010
    Messages:
    3,919 (2.49/day)
    Thanks Received:
    581
    Location:
    UK
    You may want to re-burn the disks every 5-10 years, burned disks do not last forever.
     
  18. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    Hello again, all done with only a few mistakes, I think because the directory word count had been reached.

    Each archive opens individually separate of the rest which I was very chuffed about, theres only 60gb extra data with 10% recovery record and 3 recovery volumes.

    So just to make this crystal clear before putting this on bluray, the recovery volumes can replace any one missing archive and provide me with the original data that was lost in the missing archive?

    And thanks cheesy999. I've read that taiyo yuden discs generally last around 25 years but I'd replenish the media every 10 years or so, does that sound reasonable? Am I at much risk of losing data with that time frame do you reckon?
     
    Last edited: Feb 18, 2013
  19. RejZoR

    RejZoR

    Joined:
    Oct 2, 2004
    Messages:
    4,742 (1.29/day)
    Thanks Received:
    982
    Location:
    Europe/Slovenia
    You don't understand the concept of compression too well. Let me explain a bit.

    If you have NTFS filesystem you don't need to split the archive in several parts. Just make sure you really use Non-Solid archive so you can extract individual files from the "big" archive. Other than that 7z with LZMA/LZMA2 doesn't have any limitations you will ever hit. Not in archive size and not in file count and separate file size. You can basically pack millions of files resulting in several terabytes of data.

    As for the split parts .001, .002, .003 etc, the files are a continuous stream of data split into several parts. If one file has part of itself in .001 and part in .002 you can't extract data only from individual part. Thats why 7zip and also RAR always take all the parts into consideration and you always have to open only the first one and it will gather the data from the rest. WinRAR used to support opening any part but in reality that only meant program itself auto checked which is the first part and it opened that one even though you clicked part file number 009 or 050.

    Also if you have a lot of incompressible data like JPG images or already compressed videos and music, exclude it from the start as you won't gain any compression on those.

    And as far as videos go, like i said above, they are incompressible. Changing it from AVI to MKV won't do anything at all as both AVI and MKV are containers. Basically just like a box into which you pack video file, audio file and subtitles. So changing the box from AVI to MKV won't make any difference since the content inside the box will still be exactly the same.
    Only way to gain space is to re-encode the videos using more efficient methods (H.264 instead of DivX). But remember, re-encoding files will always slightly degrade quality even if you do it with max possible bitrate and quality settings.
     
  20. tidyboyd

    tidyboyd New Member

    Joined:
    Feb 10, 2013
    Messages:
    17 (0.03/day)
    Thanks Received:
    0
    I require the archive to be broken up for storage reasons as stated and I've now got it so each part file opens up totally separately from each other. As for the MKV and AVI I really dont know what I'm talking about there at all, I just notice that MKV's in general seem to be smaller than AVI's but that could just be luck of the draw. Re encoding is going to be too time consuming with all the other stuff Ive got to do but thanks for the tip.
     
  21. micropage7

    micropage7

    Joined:
    Mar 26, 2010
    Messages:
    5,949 (3.55/day)
    Thanks Received:
    1,380
    Location:
    Jakarta, Indonesia
    avi files or multimedia files are hard to compress, you may zip or whatever but you just could reduce the size a little.
    the bigger size that you compress the bigger chance you gonna have corrupt data too
    the other option is convert it but you may lose some quality if you do that.
     
  22. RejZoR

    RejZoR

    Joined:
    Oct 2, 2004
    Messages:
    4,742 (1.29/day)
    Thanks Received:
    982
    Location:
    Europe/Slovenia
    It doesn't matter, you CANNOT extract individual files that are otherwise split across several archive parts without having alll the part files at that location.

    If you have 3 boxes with toy horse (imagine them as archive parts), box 1 contains horse head, box 2 contains horse torso, box 3 contains horse legs. You can't open just box 2 and assemble a horse from that, you need to open other 2 boxes as well. The same is with archives.

    As for AVI and MKV, i've explained it to you already. Both are just containers, a shell or a box. But anything can be stored inside. AVI or MKV doesn't define a type of compression, it just stores the video stream, audio stream and subtitles inside. The reason why you often see MKV's smaller is because AVI files are often compressed with DivX or XviD and MKV files are most often compressed with H.264 which is newer and more efficient video compressor.
    But you can just as well have MKV files that contain video compressed by DivX/XviD or AVI with H.264 compressed video. Very uncommon but still possible.

    I suggest you learn a bit more about compression basics because you're confusing the most basic things around compression and file formats and video encoding and video codecs.
     
  23. lemonadesoda

    lemonadesoda

    Joined:
    Aug 30, 2006
    Messages:
    6,260 (2.10/day)
    Thanks Received:
    967
    ^CONFUSING

    The situation is exactly like this:

    SCENARIO
    You have hundreds, nay, thousands of files. You create a NON-SOLID archive with (say) 10 parts. You create recovery records (say) 5 percent. You create 1 recovery volume. For those looking for definitions: a recovery record is extra data stored WITHIN each archive to help with data correction/parity errors. For example, the filestructure or data becomes slightly corrupted somehow. e.g. slightly damaged ZIP disk, CD disc, Floppy etc. A recovery volume is an extra archive part that contains a bit-parity file to reconstruct a missing volume of data. For example, you create an archive with 10 parts... then a recovery volume would allow you to lose a whole archive part, and you could still perform a recovery of the data.

    There are 11 files created.
    [​IMG]archive.part1.rar
    [​IMG]archive.part2.rar
    ...
    [​IMG]archive.part10.rar
    [​IMG]archive.part1.rev

    EXTRACTION OF FILES
    With a non-solid archive you can extract individual files from individual archive parts... SO LONG AS... THAT particular file doesnt cross an archive.part boundary. That is, if the file is spread over two or more parts, then all those relevant parts are needed. You DO NOT need ALL archive parts for the recovery of one file.

    Obviously if your archive contains only one giant megafile then yes you need all parts. But our scenario is an archive containing hundreds, nay, thousands of files.

    DAMAGED ARCHIVE
    If a single archive is somehow damaged, for example parity errors due to scratched disc, faulty HDD sector, then the recovery records you created will help fix that. No guarantees, but if only a "bit" of the data is damaged then you will get 100% recovery.

    If one of the archive parts is somehow lost or damaged, for example, lost or broken media, then a recovery volume you created will fix that. If you created one recovery volume, you can loose a whole archive part. If you created more recovery volumes, you can "lose" more of the original data. I'm not sure if it is a one-one mapping, ie, two recovery volumes are guaranteed to fix two lost archive parts.

    LEARN BY TESTING
    Make a temp directory. Put a bunch of file copies in there. Create a WINRAR archive. In "GENERAL options" uncheck create solid. Check "split to volumes" and put an appropriate size, e.g. one tenth of what the total data will be. In "ADVANCED options" dial up the creation of 1 recovery volume.

    You will get an output that looks like the SCENARIO icons above.

    How, delete one of those archive parts, say, archive.part3.rar.

    Now try to extract the files from what is left. You can extract all the files that are stored in the archive parts 1,2, 4, 5, 6, 7, 8, 9, 10. You cannot extract any files from part3 because we deleted it... and that includes any files that start in part2 and continue into part3, or start in part3 and continue into part4. etc.

    NOW click on the file with the first aid icon archive.part1.rev

    ** HEY BINGO ** your missing archive.part3 will be reconstructed. Now delete another archive parts, say archive.part4.rar. Then click on the file with the first aid icon archive.part1.rev again. ** HEY BINGO ** your missing archive.part4 will be reconstructed.

    Welcome to recovery volumes in WINRAR.

    http://2.bp.blogspot.com/_-OTldnNHgeE/TMSOEKSLnEI/AAAAAAAAAgs/j1EqmsK9N8k/s1600/medal.jpg
     
    tidyboyd says thanks.
  24. RejZoR

    RejZoR

    Joined:
    Oct 2, 2004
    Messages:
    4,742 (1.29/day)
    Thanks Received:
    982
    Location:
    Europe/Slovenia
    Remember that these recovery records cannot do wonders. I know it first hand from the days when we were still handling files around on floppy disks.
     

Currently Active Users Viewing This Thread: 1 (0 members and 1 guest)

Share This Page