Archiving¶
Genozip has two separate capabilities of aggregating files, each with its pros and cons:
1. tar with –tar
Using the genozip --tar
option, genozip compresses files directly into a standard tar file.
Each file is compressed independently and written directly into a standard tar file as it is being formed. This is faster and consumes less disk space than first genozipping files and then packaging them into a tar file, since no separate .genozip files are created - just the tar file.
Example 1:
> # Compressing
> genozip --tar mydata.tar sample1.bam sample2.bam variants.vcf
> # Listing the contents of the tar file
> tar tvf mydata.tar
-rw-rw-rw- USER/USER 3424847 2021-06-01 11:34 sample1.bam.genozip
-rw-rw-rw- USER/USER 6765323 2021-03-04 22:04 sample2.bam.genozip
-rw-rw-rw- USER/USER 765323 2021-03-04 22:08 variants.vcf.genozip
> # Unarchiving and decompressing all files
> tar xvf mydata.tar |& genounzip --files-from - --replace
Example 2: compress all files in a directory and its sub-directories, using --subdirs
:
> genozip --tar mydata.tar --subdirs my-data-dir
Example 3: compress and archive all BAM files in the current directory and its sub-directories, preserving the directory struture:
> find . -name "*.bam" | genozip --tar mydata.tar -T-
Implementation note: Genozip implements the IEEE 1003.1-1988 (“ustar”) standard of tar files, with the size field in binary format for files 8GB or larger. The GNU-tar LongLink extension is used for file names longer than 99 characters. This is compatible with most modern tar implementations, including GNU tar.
2. Binding with –output
Unlike --tar
, binding does not record the directory structure.
Unlike tar that merely aggregates files, binding leverages similarities and data redundancies between files to improve compression - typically, the size of a bound file is smaller than the combined sizes of files compressed separately.
Binding also allows analyzing the combined file as a concatenated data set.
Example:
> # binding
> genozip chr1.vcf chr2.vcf --output variants.vcf.genozip
> # unbinding back to the original files
> genounzip variants.vcf.genozip
> # viewing the concatenated data
> genocat variants.vcf.genozip
> # extracting the first component (chr1.vcf in this case) from the bound file
> genocat --component 1 variants.vcf.genozip
> # listing the components of a bound file
> genols variants.vcf.genozip
In the following example, we bind all the BAM files in the current directory:
> # binding
> genozip *.bam --output mydata.bam.genozip
> # unbinding back to the original files
> genounzip mydata.bam.genozip
> # unbinding into a different directory
> genounzip mydata.bam.genozip --prefix=mydir/