RCF::FileDownload with large files (> 2 Gb)

[flipped5]

Hello,

We're using RCF::FileDownload to do file transfer, and it appears to work well for our purpose. However, we encounter problems when downloading large files (size > 2 Gb) on Windows (our code is in 64-bit).

The symptom we're seeing is that file transfer stops when the number of bytes transferred is about 2 G.

When the debug build of the program does the transfer, there's an assert in RCF from inside FileIoRequest::doTransfer() as the file becomes larger than 2 Gb.

Looking at the variable values in the debugger, it appears that type casts such as

Code: Select all

boost::uint64_t pos1 = mFoutPtr->tellp()

doesn't result in the correct size of the file on Windows when the file is over 2 Gb. More precisely, pos0 is just under 2 Gb while pos1 is a very large number.

We've modified RCF locally to deal with the issue, but we're hoping that there will be an official fix if this is indeed a problem in RCF.

Thanks!

[jarl]

Thanks for reporting this. This is most likely a bug in RCF, and I'll make sure to fix it before the next release.

Are you able to send me a patch of the changes you made?

[jarl]

And one more question - which compiler are you using? Is it 32 bit or 64 bit?

[flipped5]

Thanks for looking into this.

We're primarily using the 64 bit compiler.

Here are the diffs generated from our local changes (we're still using RCF 2.0.0.2673):

FileIoThreadPool.cpp:

Code: Select all

252d251
<             boost::uint64_t pos0 = mFoutPtr->tellp();
254,258c253,254
<             boost::uint64_t pos1 = mFoutPtr->tellp();
< 
<             RCF_ASSERT_GTEQ(pos1 , pos0);
<             mBytesTransferred = pos1 - pos0;
<             RCF_ASSERT_EQ(mBytesTransferred , szBufferLen);
---
>             RCF_ASSERT(mFoutPtr->good());
>             mBytesTransferred = szBufferLen;

FileStream.cpp

Code: Select all

1114c1114
<                 RCF_ASSERT_EQ(currentPos , (boost::uint64_t) fout->tellp());
---
>                 //RCF_ASSERT_EQ(currentPos , (boost::uint64_t) fout->tellp());

[jarl]

I was able to reproduce the assert with Visual Studio 2008, but not with Visual Studio 2013. The problem is that in VS2008, the file offsets returned by tellp() are 32 bit. From what I've read, this is also the case in VS 2010. It's basically a bug in the standard library shipped with those compilers, and was fixed in VS 2012. Are you able to upgrade to VS2012 or later?

Does your patch allow you to transfer large files correctly? AFAICT it is only disabling the asserts?

[flipped5]

I did some more googling, and it does appear that the actual bug is in the standard library shipped with Visual Studio. With the patch I was able to download a video (> 2GB, our internal format) and played it back from start to end.

With the patch, the line

Code: Select all

mBytesTransferred = pos1 - pos0;

is changed to

Code: Select all

mBytesTransferred = szBufferLen;

which avoids using the values returned by the buggy tellp() function, and I believe the value stored in mBytesTransferred is used later.

It might still be a few months away before we make the switch to VS2012 (or later), but as you noted, most of the changes were disabling asserts, so it's not unmanageable for us if we have to apply the patch on our own to new versions of RCF.