View Issue Details

IDProjectCategoryView StatusLast Update
0000937luatexluatex bugpublic2015-11-02 17:40
Reporterpdfkungfoo Assigned ToHans Hagen  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionwon't fix 
PlatformTex Live 2014OSOS X MavericksOS Version10.9.5
Product Version0.80.1 
Target Versionmaybe never (> v 1) 
Summary0000937: lualatex trips over some JPEGS and aborts PDF generating process (or leads to unexpected results)
DescriptionThe `lualatex` command trips over some JPEG files and aborts the PDF generating process.

Here is an MWE, `mini.tex`:

##############################
\documentclass[a4paper]{article}
\usepackage{graphicx}
\begin{document}
\thispagestyle{empty}
\begin{figure}[htbp]
\centering
\includegraphics{./testwiz.jpg}
\end{figure}
\end{document}
##############################

The `testwiz.jpg` was generated with ImageMagick from its built-in `wizard:` image:

     convert wizard: -frame 1 testwiz.jpg
    
As it happens, this JPEG now has the following JFIF headers set, as seen in a hex editor:

+-------------+-------------------------+------------+------------+
| Byte-Offset | 0x0d | 0x0e | 0x10 |
+-------------+-------------------------+------------+------------+
| Field-Name | "Units" | "XDensity" | "YDensity" |
+-------------+-------------------------+------------+------------+
| Value | 1 | 1 | 1 |
+-------------+-------------------------+------------+------------+
| Meaning | "X/YDensities are DPI" | 1 dpi | 1 dpi |
+-------------+-------------------------+------------+------------+

Exiftool reports this:

    exiftool testwiz.jpg
    
     Resolution Unit : inches
     X Resolution : 1
     Y Resolution : 1

Running `lualatex mini.tex` yields this:

    LuaTeX warning: arithmetic: number too big
    ! Dimension too large.
    <recently read> \dimen@
            
    l.7 \includegraphics{./testwiz.jpg}
                                     
    ?
    
Hitting [Enter] repeatedly completes the PDF generation process. The resulting document has 66 kBytes. It shows only a white page in the viewer.

Unpacking the PDF:

    qpdf --qdf --object-streams=disable mini.pdf unpacked-mini.pdf
    
Inspecting `unpacked-mini.pdf` in a text editor shows these points:

1. The image data is embedded and complete. It is in PDF object no. 8, containing 65456 Bytes of JPEG data in the object's stream.

1. The `/Contents` object is no. 5 which contains this line of PDF source code:

        -32645.5776 0 0 -32645.5776 124.802 706.129 cm /Im1 Do

This means the image is displaced and scaled by a huge amount and is outside the normal page `/MediaBox`. After changing above line to

            481 0 0 640 124.802 76.093 cm /Im1 Do
            
(be careful to ***overwrite*** the old bytes, and not add or subtract from the total number of bytes!) and saving the modification the image will become visible on the PDF page.

I think it is stupid for `lualatex` to rely on the values of the JFIF header for scaling the image in (absence of explicit user provided TeX commands giving orders about the wanted scaling) when the headers says the resolution is 1 DPI:

* Almost always a resolution of DPI = 1 does not make sense.
* Almost always the resulting image on the page comes out in un-expected ways.

Imagine a user with two identically looking files, where one file "works" on a PDF page and looks as expected while the other one exhibits the described behavior just because of one Bit being flipped in the JFIF header!

You can manipulate the JFIF headers of a JPEG without changing any image pixels with the help of `exiftool`:

    exiftool \
      -jfif:Xresolution=300 \
      -jfif:Yresolution=300 \
      -jfif:ResolutionUnit=inches \
       the.jpg

Setting `-jfif:ResolutionUnit=cm` sets the "Units" field to "centimeters per inch".
Setting `-jfif:ResolutionUnit=None` sets the "Units" field to "Xdensity and Ydensity are pixel ratios".


The following settings in the JFIF header all currently cause different PDF output with the `mini.tex` input file:

+-------------+-----------------------------+------------+------------+
| Byte-Offset | 0x0d | 0x0e | 0x10 |
+-------------+-----------------------------+------------+------------+
| Field-Name | "Units" | "XDensity" | "YDensity" |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 1 | 1 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 7 | 7 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 100 | 100 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 200 | 200 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 300 | 300 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 100 | 300 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 300 | 100 |
+-------------+-----------------------------+------------+------------+
| Value | 1 | 300 | 1 |
+-------------+-----------------------------+------------+------------+
| Meaning | "X/YDensities are DPI" | <above> px | <above> px |
+-------------+-----------------------------+------------+------------+

At least the behavior here is somehow consistent: the scaling, buckling and stretching happens according to the DPI settings.

But if the settings are changed to these:

+-------------+-----------------------------+------------+------------+
| Byte-Offset | 0x0d | 0x0e | 0x10 |
+-------------+-----------------------------+------------+------------+
| Field-Name | "Units" | "XDensity" | "YDensity" |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 7 | 7 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 100 | 100 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 200 | 200 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 300 | 300 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 100 | 300 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 300 | 100 |
+-------------+-----------------------------+------------+------------+
| Value | 0 | 300 | 1 |
+-------------+-----------------------------+------------+------------+
| Meaning | "X/YDensities pixel ratios" | <above> px | <above> px |
+-------------+-----------------------------+------------+------------+

then the output is identical for each case, not changing the pixel aspect ratios as it should. So...

1. `lualatex` does not respect the "X/Ydensity" JFIF fields if "Units" is set to "0", meaning "X/YDensities are pixel ratios".
2. `lualatex` does ONLY respect the "X/Ydensity" JFIF fields if "Units" is set to "1", meaning "X/YDensities are DPI".

This is inconsistent. If it creates different output in case 2 it should also do so in case 1.

I even think the `lualatex` PDF producing utility should NOT use the JFIF header metadata AT ALL for embedding JPEG images into pages. Because:

* Most users are unaware of these JFIF headers.
* Most users don't know how to change these headers.
* These headers control the scaling of JPEGs on PDF pages only indirectly.
* Most users expect to control the scaling of images in PDF output by different (LaTeX-provided) means directly.
* The treatment of the various JFIF headers is inconsistent:
    - if set as "X/Densities are DPI" they are NOT respected;
    - if set as "X/YDensities are pixel ratios" they are (somehow) respected.

# Executive summary

1. `lualatex` behaves unreasonable when it generates PDFs which include JPEG images:

    - On the one hand, "XDensity" and "YDensity" fields are completely IGNORED if they describe "DPI" (or "DPcm").

    - On the other hand, "XDensity" and "YDensity" fields ARE evaluated if they describ "pixel aspect ratios", but the outcome is totally unexpected for most users.

2. `lualatex` should ignore the JFIF fields completely and only obey the statements given by the document's LaTeX code. In the absence of such code, the PDF generators should assume 72 DPI.

----

Additional InformationThis is the first time I report on Mantis. I don't know if my writeup and my tables will come out readable or not. If not, my apologies already now.

Please also refer to these resources:

1. http://tex.stackexchange.com/q/244222/8747
2. http://tex.stackexchange.com/q/243753/8747
3. http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=27589
TagsNo tags attached.

Activities

pdfkungfoo

2015-05-11 18:32

reporter  

testwiz.jpg (65,456 bytes)   
testwiz.jpg (65,456 bytes)   

pdfkungfoo

2015-05-11 18:43

reporter   ~0001365

What a crappy user interface to use is this bug tracker here!

For a willing and well-meaning user to have to deal with this is a total pain in the a*#.

1. Can't you at least use a mono-spaced font?!
2. And tell it not to eat up multiple spaces!
3. And give me a preview of what I've written before I finally submit?
4. And let me edit my posting after I discover a typo or an error?!
5. [... no, I'm not wasting my time any more...]

As it stands, this will be used by me again... ...only in extreme emergencies!

luigi scarso

2015-05-11 18:51

developer   ~0001366

Thank you for the report. We will investigate asap. Can you check against the latest 0.80.0 ?

pdfkungfoo

2015-05-11 19:04

reporter   ~0001367

Luigi, I don't know how to check against 0.80.0.

I totally rely on the packages the "Tex Live Utility" gives me (which I update almost daily).

But if there isn't a *totally easy* and *easily revertible* way to update to 0.80.0 I'm not willing to jump through loops here. (I'm already handling other self-compiled software packages on my notebook. But these I *know* how to deal with in case of problems -- everything LaTeX/TeX related I have no clue. I only came to LaTeX/LuaLaTeX via `pandoc`...)

oneiros

2015-05-11 19:18

developer   ~0001368

Confirmed with 0.80.1 (TeX Live 2015) (rev 5241)

oneiros

2015-05-11 19:18

developer  

mini.tex (119 bytes)

oneiros

2015-05-11 19:18

developer  

mini.log (6,615 bytes)   
This is LuaTeX, Version beta-0.80.1 (TeX Live 2015) (rev 5241)  (format=lualatex 2015.5.8)  11 MAY 2015 19:17
 restricted \write18 enabled.
**./mini.tex
(./mini.tex
LaTeX2e <2014/05/01>
Babel <3.9l> and hyphenation patterns for 53 languages loaded.
No auxiliary output files.

(/opt/tex/texlive/2014/texmf-dist/tex/latex/base/minimal.cls
Document Class: minimal 2001/05/25 Standard LaTeX minimal class
) (/opt/tex/texlive/2014/texmf-dist/tex/latex/graphics/graphicx.sty
Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)

(/opt/tex/texlive/2014/texmf-dist/tex/latex/graphics/keyval.sty
Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
\KV@toks@=\toks14
)
(/opt/tex/texlive/2014/texmf-dist/tex/latex/graphics/graphics.sty
Package: graphics 2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR)

(/opt/tex/texlive/2014/texmf-dist/tex/latex/graphics/trig.sty
Package: trig 1999/03/16 v1.09 sin cos tan (DPC)
)
(/opt/tex/texlive/2014/texmf-dist/tex/latex/latexconfig/graphics.cfg
File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
)
Package graphics Info: Driver file: pdftex.def on input line 91.

(/opt/tex/texlive/2014/texmf-dist/tex/latex/pdftex-def/pdftex.def
File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX

(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/infwarerr.sty
Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO)
)
(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/ltxcmds.sty
Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
)
\Gread@gobject=\count79
))
\Gin@req@height=\dimen102
\Gin@req@width=\dimen103
)
No file mini.aux.
LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 4.
LaTeX Font Info:    ... okay on input line 4.
LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 4.
LaTeX Font Info:    ... okay on input line 4.
LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 4.
LaTeX Font Info:    ... okay on input line 4.
LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 4.
LaTeX Font Info:    ... okay on input line 4.
LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 4.
LaTeX Font Info:    ... okay on input line 4.
LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 4.
LaTeX Font Info:    ... okay on input line 4.
(/opt/tex/texlive/2014/texmf-dist/tex/context/base/supp-pdf.mkii
[Loading MPS to PDF converter (version 2006.09.02).]
\scratchcounter=\count80
\scratchdimen=\dimen104
\scratchbox=\box26
\nofMPsegments=\count81
\nofMParguments=\count82
\everyMPshowfont=\toks15
\MPscratchCnt=\count83
\MPscratchDim=\dimen105
\MPnumerator=\count84
\makeMPintoPDFobject=\count85
\everyMPtoPDFconversion=\toks16
) (/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty
Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO)


(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/ifluatex.sty
Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO)
Package ifluatex Info: LuaTeX detected.
)
(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/ifpdf.sty
Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO)
Package ifpdf Info: pdfTeX in PDF mode is detected.
)
Package pdftexcmds Info: \pdf@primitive is available.
Package pdftexcmds Info: \pdf@ifprimitive is available.
Package pdftexcmds Info: \pdfdraftmode found.

(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/luatex-loader.sty
Package: luatex-loader 2010/03/09 v0.4 Lua module loader (HO)

(/opt/tex/texlive/2014/texmf-dist/scripts/oberdiek/oberdiek.luatex.lua))
\pdftexcmds@toks=\toks17
)
(/opt/tex/texlive/2014/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty
Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf

(/opt/tex/texlive/2014/texmf-dist/tex/latex/oberdiek/grfext.sty
Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO)

(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty
Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO)
))
(/opt/tex/texlive/2014/texmf-dist/tex/latex/oberdiek/kvoptions.sty
Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO)

(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty
Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO)

(/opt/tex/texlive/2014/texmf-dist/tex/generic/oberdiek/etexcmds.sty
Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
)))
Package grfext Info: Graphics extension search list:
(grfext)             [.png,.pdf,.jpg,.mps,.tif,.jpeg,.PNG,.PDF,.JPG,.JPEG,.eps]
(grfext)             \AppendGraphicsExtensions on input line 452.

(/opt/tex/texlive/2014/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live

))
LuaTeX warning (file ./testwiz.jpg): The image specifies an unusual resolution o
f 1dpi by 1dpi. 

LuaTeX warning: arithmetic: number too big

LuaTeX warning: arithmetic: number too big
! Dimension too large.
<recently read> \dimen@ 
        
l.5 \includegraphics{./testwiz.jpg}
                                 
? s
OK, entering \scrollmode...
<./testwiz.jpg, id=1, --32768.0pt x 0.0pt>
! Dimension too large.
<to be read again> 
\advance 
l.5 \includegraphics{./testwiz.jpg}
                                 
I can't work with sizes bigger than about 19 feet.
Continue and I'll use the largest value I can.

File: ./testwiz.jpg Graphic file (type jpg)
<use ./testwiz.jpg>
Package pdftex.def Info: ./testwiz.jpg used on input line 5.
(pdftex.def)             Requested size: 16383.99998pt x 0.0pt.

LuaTeX warning: arithmetic: number too big

LuaTeX warning: arithmetic: number too big

Overfull \hbox (15934.245pt too wide) in paragraph at lines 5--6
[][][] 
 []

[1<./testwiz.jpg
LuaTeX warning: The image specifies an unusual resolution of 1dpi by 1dpi. 
>])

Here is how much of LuaTeX's memory you used:
 1404 strings out of 494721
 100000,89155 words of node,token memory allocated
 240 words of node memory still in use:
   2 hlist, 1 vlist, 1 rule, 2 glue, 34 glue_spec, 1 write nodes
   avail lists: 2:12,3:1,4:15,6:10,7:1,9:11
 5049 multiletter control sequences out of 65536+600000
 15 fonts using 601983 bytes
 37i,4n,23p,215b,102s stack positions out of 5000i,500n,10000p,200000b,100000s
Output written on mini.pdf (1 page, 66532 bytes).

PDF statistics: 9 PDF objects out of 1000 (max. 8388607)
 4 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 131072)
 1 words of extra memory for PDF output out of 10000 (max. 10000000)

mini.log (6,615 bytes)   

luigi scarso

2015-05-11 20:34

developer   ~0001369

exiftools 9.46 with your image says
:
Resolution Unit : inches
X Resolution : 1
Y Resolution : 1
:
Image Size : 482x642

The image size is hence 482in x 642in, and both are bigger than 2^16pt (1pt =1/72.27 in)
which is the max dimension accepted by the TeX engines pdftex, xetex, luatex.
In this case the image is not loaded (in luatex) even if the user specifies a transformation matrix that gives an acceptable final dimensions. Note that 1dpi is not wrong by itself
--- 20x20 at 1dpi is perfectly fine --- the limits are on the metric dimensions.
Anyway, from luatex 0.80.0 there is a warning for 1dpi:
LuaTeX warning: The image specifies an unusual resolution of 1dpi by 1dpi.

LuaTeX warning: arithmetic: number too big

LuaTeX warning: arithmetic: number too big

pdfkungfoo

2015-05-12 00:59

reporter   ~0001370

@LuigiScarso: I know that the image size is 482x642 pixels, and a 1 DPI JFIF setting, if respected, will create an image of 482in x 642in.

That is exactly my argument: scaling the image based on a (most likely) very completely uncontrolled JFIF header setting is not a good idea.

The changed v0.80.0 warning does not help much, sorry. If the lualatex behavior does not change, it will still not create a visible image on the PDF page.

luigi scarso

2015-05-12 02:21

developer   ~0001371

The image is valid, and as I have said there are 1dpi images that are perfectly fine with luatex. To put the box with the image on the main vertical list, Luatex needs to know its metric dimensions and the only way, when the dpi is known, is to multiply the dpi for the width and the height.
In this case the dimensions are over 2^16pt, and the box cannot be built.
It is supposed that a user knows these features, checking if an image fits into this limit, and acting as consequence if this is not the case.
We agree that it's desirable that a transformation matrix could act before the calculation of the dimensions, but this doesn't solve the problem, as also
impose a scaling to 2^16pt .
This is not a bug, but a feature that could be improved in a next release.

luigi scarso

2015-05-12 08:40

developer   ~0001372

Ah, sorry, I meant 32768pt = 2^15pt not 2^16pt.

luigi scarso

2015-05-12 09:07

developer   ~0001373

Also the message
LuaTeX warning: arithmetic: number too big
comes from
scaled ext_xn_over_d(scaled x, scaled n, scaled d)
{
    double r = (((double) x) * ((double) n)) / ((double) d);
    if (r > DBL_EPSILON)
        r += 0.5;
    else
        r -= 0.5;
    if (r >= (double) max_integer || r <= -(double) max_integer)
        luatex_warn("arithmetic: number too big");
    return (scaled) r;
}

where
#define max_integer 0x7FFFFFFF
so it's related to a value that cannot fit into a scaled
(which is defined as int).
Even if the image fits into a scaled, it can
be still problematic because the \maxdimen is 16383.99998pt ~= 2^14pt .

Hans Hagen

2015-10-23 09:07

manager   ~0001473

it's dangerous to change behaviour of the inclusion and interpretation of fields without much testing, we keep an eye on it but zero or one pixel images are ot common in tex related workflows

Hans Hagen

2015-11-02 17:40

manager   ~0001495

we will not touch the inclusion code; this issue relates to the maximum dimension that tex can handle and 1 dpi is resulting in an overflow ... so be it

(LS/HH)

Issue History

Date Modified Username Field Change
2015-05-11 18:32 pdfkungfoo New Issue
2015-05-11 18:32 pdfkungfoo File Added: testwiz.jpg
2015-05-11 18:43 pdfkungfoo Note Added: 0001365
2015-05-11 18:51 luigi scarso Note Added: 0001366
2015-05-11 19:04 pdfkungfoo Note Added: 0001367
2015-05-11 19:18 oneiros Note Added: 0001368
2015-05-11 19:18 oneiros File Added: mini.tex
2015-05-11 19:18 oneiros File Added: mini.log
2015-05-11 19:20 oneiros Product Version 0.79.1 => 0.80.1
2015-05-11 20:34 luigi scarso Note Added: 0001369
2015-05-12 00:59 pdfkungfoo Note Added: 0001370
2015-05-12 02:21 luigi scarso Note Added: 0001371
2015-05-12 08:40 luigi scarso Note Added: 0001372
2015-05-12 09:07 luigi scarso Note Added: 0001373
2015-10-23 09:07 Hans Hagen Note Added: 0001473
2015-10-23 09:10 Hans Hagen Target Version => maybe never (> v 1)
2015-10-24 21:01 Hans Hagen Status new => acknowledged
2015-11-02 17:40 Hans Hagen Note Added: 0001495
2015-11-02 17:40 Hans Hagen Status acknowledged => closed
2015-11-02 17:40 Hans Hagen Assigned To => Hans Hagen
2015-11-02 17:40 Hans Hagen Resolution open => won't fix