Sparse File

File System Support
"Unixy" filesystems : ext2, ext3, ufs, xfs, reiserfs, jfs,... all support Sparse files

FAT16 and FAT32 do not

NTFS does support Sparse files

References:
 * Sparse files - MLDonkey - http://mldonkey.sourceforge.net/Sparse_files#FAT16.2C_FAT32
 * File System Functionality Comparison (Windows) - https://msdn.microsoft.com/en-us/library/windows/desktop/ee681827%28v=vs.85%29.aspx

Sparse File
Wiki: Sparse Files

In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently. When space has been allocated to a file but not actually filled with data it is not written to the file system. Instead, meta-information about these "empty" regions is stored until they are filled with data.

The obvious advantage of sparse files is that storage is only allocated when actually needed. Large files can be created even if there isn't enough free space yet. A disadvantage is that sparse files can become very fragmented. Also, filling up partitions to the maximum can have unpleasant effects.

Will create a file of one megabyte in size, but with only one byte actually stored on disk: dd if=/dev/zero of=sparse-file bs=1 count=1 seek=1M

Sparse Tools
View sparse size (in blocks) ls -ls [file] stat [file]
 * 1) both will report the full file size, but the block size will show sparse size

du -s -B1 --apparent-size sparse-file du -s -B1 sparse-file

tar -S, --sparse	# handle sparse files efficiently

cp - # control creation of sparse files cp --sparse=WHEN [...]
 * 1) cp tries to detect sparse files automatically.  This control can be forced
 * 1) --sparse=auto
 * 2) --sparse=always
 * 3) --sparse=never

man cp: By default,  sparse SOURCE files are detected by a crude heuristic and the corresponding DEST file is made sparse as well. That is the behav- ior selected  by  --sparse=auto. Specify --sparse=always to create a  sparse DEST file whenever  the  SOURCE  file  contains  a  long  enough sequence of  zero  bytes. Use --sparse=never to inhibit creation of   sparse files.

This is especially useful if a sparse-file has somehow become non-sparse (i.e. the empty blocks have been written out to disk in full). Disk space can be recovered by doing: cp --sparse=always formerly-sparse-file recovered-sparse-file

rsync -S, --sparse	# handle sparse files efficiently rsync -aP --sparse local-file remote-host:remote-file

cpio --sparse

Not supported tools: pax, scp, sftp, and ftp
 * 1) Using such a utility would make a sparse file balloon in size.

keywords
sparse file