How is a JPEG image’s color type determined?

In order to decode and display a JPEG image, you have to know what “color type” (or “color model”) it uses. There are several possibilities that your software probably ought to handle:

  • Grayscale
  • YCbCr (transformed from RGB)
  • RGB
  • YCCK (transformed from CMYK)
  • CMYK

Unfortunately, if you look at all the bits and bytes that make up a JPEG file, you will not find a field for the color type. No such field exists. You will have to guess.

What your software really should do is “whatever most other software does”. Most other software uses the Independent JPEG Group’s software, which I will call IJG libjpeg. So now the question is, what does IJG libjpeg do?

It bases its decision on several pieces of information:

  • The number of components in the image. (A “component” is a color channel; e.g. an RGB image would probably have “red”, “green”, and “blue” components.)
  • The presence of a JFIF segment.
  • The presence of an APP14 “Adobe” segment, and the value of its “color transform” field.
  • The component ID numbers. A JPEG encoder assigns each component an ID number, which can potentially be used to convey information about the contents of that component.

“What about Exif”, you may ask. Exif’s “PhotometricInterpretation” field seems like it would help, but actually, it is not used. In fact, the Exif specification explicitly forbids it from being used for this. And anyway, IJG libjpeg doesn’t support Exif.

Note that I am only going to describe the default behavior of IJG libjpeg. It is possible for an application to use IJG libjpeg, but with its own custom logic for determining the color type.

The table below shows the logic used by older versions of IJG libjpeg, from at least version 6b through version 9. Use the first row in the table for which all the criteria match. An asterisk (*) means that that criterion is an automatic match.

# comps. JFIF? Adobe xform Comp. IDs Color type
1 * * * Grayscale
3 Yes * * YCbCr
3 No None 1,2,3 YCbCr
3 No None 82,71,66 (“RGB”) RGB
3 No None * YCbCr, with trace message “Unrecognized component IDs”
3 No 0 * RGB
3 No 1 * YCbCr
3 No * * YCbCr, with warning “Unknown Adobe color transform code”
4 * None * CMYK
4 * 0 * CMYK
4 * 2 * YCCK
4 * * * YCCK, with warning “Unknown Adobe color transform code”
* * * * Unknown

Starting with version 9a (Jan. 2014), through at least 9c, the logic for 3-component images is different:

# comps. JFIF? Adobe xform Comp. IDs Color type
3 * * 1,2,3 YCbCr
3 * * 1,34,35 JCS_BG_YCC
3 * * 82,71,66 (“RGB”) RGB
3 * * 114,103,98 (“rgb”) JCS_BG_RGB
3 Yes * * YCbCr
3 No None * YCbCr, with trace message “Unrecognized component IDs”
3 No 0 * RGB
3 No 1 * YCbCr
3 No * * YCbCr, with warning “Unknown Adobe color transform code”

The new JCS_BG color types are not important, and are beyond the scope of this post. The important thing is that the component ID numbers now have priority over the JFIF segment.

I am not going to argue about what the “correct” algorithm is. There is really no official authority in charge of this. I note that the new IJG libjpeg logic technically violates the JFIF specification, which says that 3-component images must be YCbCr. But even that doesn’t mean it’s bad.

Given this information, it seems we should be able to construct a JPEG image that is interpreted differently by different versions of IJG libjpeg. My idea is to start by creating an RGB JPEG image (with “RGB” component IDs), and artificially add a JFIF segment to it. We can also delete any APP14 segment, though that should make no difference. I’ve included such an image below, though I can’t be sure that the software used by this blog will leave it intact.


The older IJG libjpeg versions should interpret this image as YCbCr (making it look wrong, and green), while the newer versions should interpret it as RGB (making it look correct). Using the djpeg program included with the software, I verified that this does happen as predicted. For other JPEG software unrelated to IJG, who knows?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s