Recommendations vs. Obligations

Dear all


I just want to add a post about the recently ongoing discussion regarding "to compress or not to compress".

I personally believe that this question cannot be totally answered in a document like the presented recommendation, because the requirements are individual and change from one application to another. However, in my opinion is the selection of a file format and a well defined - known and understood - set of features very important. Why? Migration is, sooner or later, the only solution to preserve digital information. Migrating a disk or a storage system means a “simple” replication of the bit-stream. It is a copy process that can be controlled regarding quality and data consistency with e.g. a hash of the source and target objects. Such a process can to some extend be performed automatically and checked for correctness.

If the file format has to be migrated, the situation is slightly different. The migration of a file format means a transcoding, a process that is much more complex than the bit-identical copy process of a binary stream. Therefore our document hast one important aim: To make users in GLAM institutions aware of that issue and support them in finding a proper media standard that is sustainable and “ready” for archival purposes.

Data compression is certainly something that has to be offered as a possible feature, as long as the compression is still widely supported by common software. A global “do or do not“ is in my experience not possible because on one hand the people responsible for the digital assets – the content – want to stay with as raw data as possible and they mostly don’t like the idea of compression. On the other hand there is the problem of the costs for hosting data. In nearly all cases I know, the IT department of such a GLAM institution wants to see money for the archival storage services and that is quite a lot. It is clear that if you can reduce the data volume of e.g. a factor of two, this is attractive and in most cases inevitable due to the financial pressure. Therefore compression must be an option that must have its place in such a recommendation.