Hundreds of petabytes are produced annually at weather and climate forecast centers worldwide. Compression is essential to reduce storage and to facilitate data sharing. Current techniques do not distinguish the real from the false information in data, leaving the level of meaningful precision unassessed. Here we define the bitwise real information content from information theory for the Copernicus Atmospheric Monitoring Service (CAMS). Most variables contain fewer than 7 bits of real information per value and are highly compressible due to spatio-temporal correlation. Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself. All CAMS data are 17× compressed relative to 64-bit floats, while preserving 99% of real information. Combined with four-dimensional compression, factors beyond 60× are achieved. A data compression Turing test is proposed to optimize compressibility while minimizing information loss for the end use of weather and climate forecast data.
References
Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Jun;69(6 Pt 2):066138
[PMID: 15244698]
Nature. 2014 Nov 20;515(7527):338-9
[PMID: 25409812]
Phys Rev Lett. 2000 Jul 10;85(2):461-4
[PMID: 10991308]
IEEE Trans Vis Comput Graph. 2006 Sep-Oct;12(5):1245-50
[PMID: 17080858]
Proc Math Phys Eng Sci. 2017 Sep;473(2205):20170144
[PMID: 28989303]
Science. 2020 Oct 2;370(6512):16-17
[PMID: 33004491]
Nature. 2015 Oct 1;526(7571):32-3
[PMID: 26432226]
IEEE Trans Vis Comput Graph. 2020 Sep;26(9):2891-2903
[PMID: 30869621]
