[LC++]standard way to seek big files

[Jack Lloyd]

On Tue, 4 Dec 2001, Kai Vehmanen wrote:

> I've just added support for handling big (>2GB) files on 32bit platforms
> to a Linux C++ app. Looking at the standards, if using <cstdio>, there
> seems to be no standard way of either setting or getting numerical
> position of these big, "64bit" files.
>
> std::fseek() and std::ftell() are part of the standard, but take a 'long
> int' as their offset argument. std::fsetpos() and std::fgetpos() can be
> used with a 64bit fpos_t, but as 'fpos_t' is an opaque type, it's not a
> full replacement for fseek/ftell.
>
> Single Unix specification has fseeko/ftello and these are what I'm using
> now. But of course, they are not part of the C/C++ standard, so not an
> optimal solution.

This is pretty poor, but the best thing I can think of is have a seperate
file that implements your own seek functions.

bool can_do_large_seek();
// seek functions with 64-bit offsets

Then if you have to do a seek on a large file, call can_do_large_seek; if
it's true, your seek functions will handle it, otherwise, print some error
that this system can't seek a large file (or just have the seek functions
return an error if the seek is too large for the system to handle).

The implementation of these functions just calls the "best" of
seek/lseek/whatever.

A large seek could probably be implemented as a seqence of fseek's, though
ftell still wouldn't work.

This would require either some macro magic, or a program that checks for
these functions availability and generates the file automatically at build
time. But it does seem to be the solution for maximum portability and
usablity across all platforms.

> Any experiences with this? How's the situation when using streams, any
> problems handling large files?

That's going to be really system dependant. Hopefully _most_ C++ libraries
will work, but it's pretty iffy, and if this is an important part of your
application, I wouldn't rely on it being part of the C++ iostreams. OTOH,
if it's just a nice little feature, then you can state: this program
supports 64-bit file offsets IFF the C++ libraries do. Which is perfectly
reasonable, I would think, at least for most applications.

-Jack

PS - I know bzip2 supports large files, you might want to see how they do
it.