Are there known file size limits with hash programs on Windows?

Gpg4win help-en
1

I am working on setting up GPG and have run into issues with the sha256sum.exe program not returning the same hash as certutil on Windows 10 x64 (certutil is included with Windows and has been my “go to” for many years after saying goodbye to md5summer program).

My file size in question is over 400GB and after much research I landed on using the Windows program fsutil to see if the size was the issue. (Initially I thought a transfer was defective so I had to obtain another storage device to eliminate that as the cause since I just didn’t have room to save everything.) I started with small sizes and worked my way up…here are the two last tests…the 202,400,000,000 bytes (188GB) test is successful (hashes match) but 302,400,000,000 bytes (281GB) was not successful (hashes do not match).

D:>fsutil file createnew test2 202400000000
File D:\test2 is created

D:>dir test2
Volume in drive D is Data

Directory of D:\

03/26/2025 10:26 AM 202,400,000,000 test2

D:>certutil -hashfile d:\test2 SHA256
SHA256 hash of d:\test2:
2141e811e916323a0d26e9c2303cbf9a789bb8ee771c8c3b27bb9355b11cae08
CertUtil: -hashfile command completed successfully.

D:>“c:\Program Files (x86)\Gpg4win”\bin\sha256sum.exe test2
2141e811e916323a0d26e9c2303cbf9a789bb8ee771c8c3b27bb9355b11cae08 test2

D:>fsutil file createnew test3 302400000000
File D:\test3 is created

D:>dir test3
Volume in drive D is Data

Directory of D:\

03/26/2025 10:29 AM 302,400,000,000 test3

D:>certutil -hashfile d:\test3 SHA256
SHA256 hash of d:\test3:
9a5fa3f85bdd4f7e7ce824cf9ed26e7778e95008d4e74f35114205e5a448a6a5
CertUtil: -hashfile command completed successfully.

D:>“c:\Program Files (x86)\Gpg4win”\bin\sha256sum.exe test3
5e030458bbb124d92eccc2c39ccf321e467f9c1cf1365848ff471e94ce904949 test3

2

Hi,
so it could be a defect in either of the two applications or in the file system (providing different bytes when reading very large files).

Is your result consistent, in the sense that any file that you create after a certain size will get different results from the tools?

Have you tried a third tool, that uses different code to see what the sha256sum is?

You could try python which would use openssl, e.g. see hashlib — Secure hashes and message digests — Python 3.13.2 documentation

The used function to read the file from the file system could have some limits.

3

I have tried multiple storage devices (all NTFS) and the results are consistently different. 7-ZIP matches the results of certutil. Certutil is also quite a bit faster, which I assume is due to GPG being 32-bits only on Windows right now. I will do some more testing with some file sizes between the two values to see if I can narrow down at what size sha256sum.exe returns different values but it does take some time.

4

So after a lot of testing I have found that the hashes match if the file is less than 256GB but when you make it 256GB things fall apart. So I am going to GUESS that there is a design limit of 256GB on a 32-bit program running on a 64-bit OS on the Windows 10 platform.

254GB per Windows Explorer
H:>fsutil file createnew file10 273650000000
File H:\file10 is created

H:>certutil -hashfile file10 SHA256
SHA256 hash of file10:
3804fda9b711bc2f3e9a60e1de221158e5e97f63be4c4d83dbaa7db84176bf80
CertUtil: -hashfile command completed successfully.

H:>“c:\Program Files (x86)\Gpg4win”\bin\sha256sum.exe file10
3804fda9b711bc2f3e9a60e1de221158e5e97f63be4c4d83dbaa7db84176bf80 file10

256 GB per Windows Explorer
H:>fsutil file createnew file9 274900000000
File H:\file9 is created

H:>certutil -hashfile file9 SHA256
SHA256 hash of file9:
24b88b164c430b7a1cdb06ad9c3ec495d018e2a21e727a2ec343d1ea33b5c605
CertUtil: -hashfile command completed successfully.

H:>“c:\Program Files (x86)\Gpg4win”\bin\sha256sum.exe file9
00d9d416b93f0b93da63620300393dbf8c6272d404036de351cfde6aabd0272d file9

5

Thanks @AustinFastER for the intensive testing! This is very helpful for catching problems like this - which potentially are in the runtime system.

I’ve created ⚓ T7592 sha256sum on Windows 32bit calculates wrong values for files > 256 GiByte to document the problem for the developers.

Three remarks:

  • You mean Gibibyte aka GiByte so 2^30 bytes.
  • Can you give the precise version and build that you are using for the latest tests? E.g. the version of Gpg4win.

Best Regards,
Bernhard

6

Werner (the lead developer) has put a fix in. So I expect this to be fixed with the next GnuPG/Gpg4win release.

There is also another way to get the value in ⚓ T7592 sha256sum on Windows 32bit calculates wrong values for files > 256 GiByte :

gpg --print-md sha256 <myfile

Thanks a lot @AustinFastER for the report and the good testing! :+1:

7

The actual byte counts were in my prior posts and the GB sizes were from Windows Explorer.

So the command below creates a file that is 274,900,000,000 bytes. Explorer says the file is 256GB in the properties but will show 268,457,032KB in the details pane. I used Google to learn a new term…it does appear to be Gibibyte.

fsutil file createnew file9 274900000000

I installed Gpg4win 4.4.0 and I do not see a build number. The Help, About says GnuPG 2.4.7 and Libgrypt 1.11.0. I tried serveral command lines for sha256sum.exe and --version gave me 1.2 with a copyright date of 2009.

8

This is a unique release number, thanks for the additional information!