[Opendnssec-user] Sudden failure related to NSEC3

[Havard Eidnes]

Hi,

I just had the misfortune of experiencing OpenDNSSEC emit zones
which were malformed related to NSEC3.

I have a script which on our public distribution master (which is
just downstream of OpenDNSSEC) which checks zones for DNSSEC
consistency, and this evening it suddenly flagged this same
problem for a largish number of zones.

Running "ldns-verify-zone -V 2" on the copy of the zone file
resulted in:

Error: Bogus DNSSEC signature for uninett.no. NSEC3PARAM
There were errors in the zone

and running BIND's dnssec-verify on the zone results in a large
ream of messages, some of which are

No correct RSASHA256 signature for uninett.no NSEC3PARAM
Missing NSEC3 record for uninett.no (0A3GIVR501P8JRGSHTNNEJPE7B4TAL6S.uninett.no)
Missing NSEC3 record for 6etg-landskap-sw.uninett.no (ANBEOTL2U7EG7OC0B9T1AO66L1K99P55.uninett.no)
Missing NSEC3 record for _original-serial.uninett.no (883NRS6S8UQ6HD0JH4GORS1557P10PSH.uninett.no)
...
Missing NSEC3 record for zino.uninett.no (S1NO4RE0NMDCL99IG2VOD6PHSK0TH5SV.uninett.no)
Expected and found NSEC3 chains not equal
Break in NSEC3 chain at: 0MUSUIT1FJV4V39O42NVI078P60KQ2RV
Expected: 0NC9UMP6FIAJVDB6GSSJO45MCH0TJLE3
Found: 0S6OOOD7J6JB1ID779G5OBUDM330UI2H
Break in NSEC3 chain at: 0S6OOOD7J6JB1ID779G5OBUDM330UI2H
Expected: 0SD02H0M8OH2GMG91BQODJQH7RHB76K1
Found: 0UMI9EQVLJ4S92QSQBGQK2VJI6CE1VQ3
Break in NSEC3 chain at: 0UMI9EQVLJ4S92QSQBGQK2VJI6CE1VQ3
Expected: 0UMT16216AHUQF0G4ASM4CKFOEOFQQBL
Found: 1AS9JVDUK936DH23D9IJO44C9AD3IJST
...

It seems that out of the 378 zones we have in our setup, some 252
of those zones suddenly had developed this disease.

I took a copy of the tmp/ directory in OpenDNSSEC (and then
removed the files there), and have a copy of the "bad" zones
which came out at the other end if someone wants to take a look
at it to possibly find out how this could happen.

Regards,

- Håvard

[Yuri Schaeffer]

Hi Havard,

Post by Havard Eidnes
I took a copy of the tmp/ directory in OpenDNSSEC (and then
removed the files there), and have a copy of the "bad" zones
which came out at the other end if someone wants to take a look
at it to possibly find out how this could happen.

This sound bad, please send me as much you can from before and after the
problem, including policies and database if possible.

You are running 1.4.10 I presume?

//Yuri

[Havard Eidnes]

Post by Yuri Schaeffer

Post by Havard Eidnes
I took a copy of the tmp/ directory in OpenDNSSEC (and then
removed the files there), and have a copy of the "bad" zones
which came out at the other end if someone wants to take a look
at it to possibly find out how this could happen.

This sound bad, please send me as much you can from before and after the
problem, including policies and database if possible.

Will do in a separate private message.

Post by Yuri Schaeffer
You are running 1.4.10 I presume?

I'm actually running 1.4.9 with some fixes applied, some of them
remnants from debugging earlier problems. In particular, the
earlier fix which added zone_del_nsec3params() which came from
your end is part of my running code.

Regards,

- Håvard

[Berry A.W. van Halderen]

Post by Havard Eidnes

Post by Yuri Schaeffer

Post by Havard Eidnes
I took a copy of the tmp/ directory in OpenDNSSEC (and then
removed the files there), and have a copy of the "bad" zones
which came out at the other end if someone wants to take a look
at it to possibly find out how this could happen.

This sound bad, please send me as much you can from before and after the
problem, including policies and database if possible.

Will do in a separate private message.

Post by Yuri Schaeffer
You are running 1.4.10 I presume?

I'm actually running 1.4.9 with some fixes applied, some of them
remnants from debugging earlier problems. In particular, the
earlier fix which added zone_del_nsec3params() which came from
your end is part of my running code.

I think my colleague might already have mentioned it, but we think
you are in a situation that is half way 1.4.9 and 1.4.10. And half way
is not there yet ;-). After some initial fixes, it was found that
there was more to the issue. So we believe the released version
properly fix the issue you are facing.
In general, we can of course only support the released version. An
easy upgrade should get you onto track.

With kind regards,
Berry van Halderen

[Havard Eidnes]

Post by Berry A.W. van Halderen

Post by Havard Eidnes

Post by Yuri Schaeffer
You are running 1.4.10 I presume?

I'm actually running 1.4.9 with some fixes applied, some of them
remnants from debugging earlier problems. In particular, the
earlier fix which added zone_del_nsec3params() which came from
your end is part of my running code.

I think my colleague might already have mentioned it, but we think
you are in a situation that is half way 1.4.9 and 1.4.10. And half way
is not there yet ;-).

This seems like a quite probable explanation. The .axfr (IIRC)
file on the OpenDNSSEC host had only one NSEC3PARAM, but the
downstream slave still held on to two, so IXFR had become
confused by the removal, and the change from my half-way 1.4.10
to 1.4.10 proper appears to fix this.

I've now upgraded my installation to be based on 1.4.10.

Regards,

- Håvard