WIP: lightMapping: blend a bit the normal map with the deluxe map

[illwieckz]

HACK: blend a bit the normal map with the deluxe map to make the normal map align with the deluxe map to prevent NdotL collapsing to zero, while retaining enough normal details.

See: #1905

Disclosure: the blend trick was suggested by an LLM (ChatGPT) in a “discussion” with it where I described the visual look of the bug.

The trick works but that can be pure AI overconfident bullshit. Here are the « explanations » provided :

• High-frequency normal maps locally rotate the surface normal away from the smooth underlying surface.
• The baked direction [what we call deluxe maps] corresponds to the smooth underlying geometry, not the bumped normal.
• Therefore, in some pixels the dot product between the bumped normal and the baked direction can be very small or even negative: dot(normalMapNormal, bakedNormal)≈0 or negative
• This is not a bug in your code — it’s a physical consequence of combining a high-frequency normal with a low-frequency baked direction.

[illwieckz]

• Xonotic map xoylent, setviewpos 1100 30 880 170 40

Deluxe:

Normal:

Normal (as input for computeDeluxeLight()):

LightDir (as input for computeDeluxeLight()):

Blended Normal + LightDir (what this branch computes):

---

Deluxe mapping disabled:

Deluxe mapping enabled, before:

Deluxe mapping enabled, after:

[illwieckz]

This interacts with :

• Deluxe maps are wrong on and near transparent surfaces close to a skybox. Unvanquished/Unvanquished#2908

This cannot fix the bad baked data, but that can prevent translucent surface to become invisible.

• Unvanquished map station15, setviewpos 1420 -1130 101 -40 0

Deluxe (broken):

Normal (correct):

Before:

(The fact the translucent surface is very opaque is a side effect of the linear pipeline and is unrelated to the deluxe mapping bug demonstrated, the shader has to be tweaked for the linear pipeline).

After:

[illwieckz]

This can also attenuate glitches caused by broken normals, like this model used in the Vega map that has broken normals.

• Unvanquished map vega, setviewpos 1990 460 115 -120 10

Deluxe (correct):

Normal (broken):

Before:

After:

[slipher]

It has a general effect of making everything brighter. Some of the worse casualties from my test suite (before/after):

• The bolts look flattened out:
  
  

• The lower parts of the floor grate that aren't really supposed to be visible are lit up (note: heightmapping is off):