Discussion:
[Opendnssec-user] ods 2.0.1 ZSK roll-over problem
Fred Zwarts, KVI, Groningen
2016-09-16 20:38:19 UTC
Permalink
We have ods 2.0.1 running for some time, but now a ZSK roll-over is giving
a problem.
Currently the situation is as follows:

# ods-enforcer key list --verbose
Keys:
Zone: Keytype: State: Date of next transition:
Size: Algorithm: CKA_ID: Repository: KeyTag:
KVI.nl KSK retire 2016-09-17 11:00:06
2048 8 d70448361bf9ded4888de4bb681a9963 SoftHSM 23384
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 664dd2e6d61153c53f99ac2dcafddbda SoftHSM 31771
KVI.nl KSK active 2016-09-17 11:00:06
2048 8 333e0824ef6fc70c2729b02a88be92c7 SoftHSM 61849
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 6d31f5b7f2db0bc65fcb35f60ecceb1e SoftHSM 15381
KVI.nl ZSK ready 2016-09-17 11:00:06
1024 8 3c246656cd56b7cfd5294f5cb8e02229 SoftHSM 43923
key list completed in 0 seconds.

Parts from the signed zone as written by ods:

kvi.nl. 86400 IN SOA dns1.kvi.nl. hostmaster.kvi.nl. 2016091613
43200 3600 345600 10800
kvi.nl. 86400 IN RRSIG SOA 8 2 86400 20160930151204 20160916173547
43923 KVI.nl.
a1quYQgmEnAmt2BUdt3PAcEQ4mFCoLIULLEKKoICataE7OuXAbdhfjE9hT0nJeJPiLm6jmJmyj6fM2PwEb9DHS+PMulUc1L
snwayUoylsXm0HUFiAvG7+/tt2UYgybGCrXYWrrTJuu/VxMPSb4Qy5uEdwfEQRKs5w5Aeqci7aUQ=
kvi.nl. 3600 IN DNSKEY 257 3 8
AwEAAb9i0ycPgnT71XuBrWg7XuvEcUcmhLsWtXsO/vmg3xpWiYR1wW15rEMvloZ7Bl7O4/42to8GlQHx0yY1r1Kx4mkFtH6Mol31QXE8vwk4JaG7dW3UJKCWAjLD2mrBhp0umzDQK5dlkE+9o
m0sjcz2aUASNAQqwh38qOl8+3jNGbfjaw9MGK1WMYRv805NGGgPnmQ1BoB/4d99nhzqAfAWLWRLCoxD2FWjbUm+cQCft+YMtzEk46Ua1H/g/0B38E/2A71fUMWfGM5tE0XuArpFc7ri81MAzEHl5gsYGgn4QnGlsg8ip0wFZns/1NndgXpnjMlSel
vp4EEC8RCBKJ7E5IM= ;{id = 61849 (ksk), size = 2048b}
kvi.nl. 3600 IN DNSKEY 256 3 8
AwEAAbuEIkm1DbfRGZVFEfJ2BfD2h1us5RD85wTAZpXI9UfHpEjj86ApLn4uctHza1/ekkNAwy4aOgsz+TxLrvAhfKLfQL17q44ty6PDw8jQcinA8LIqB9xo9umvVagCHQeTTkoTRdHjh3DLQ
Fw9ice4N+7emoi+NTtTEa5pg9r1L41X ;{id = 43923 (zsk), size = 1024b}
kvi.nl. 3600 IN RRSIG DNSKEY 8 2 3600 20160930105859
20160916140006 61849 KVI.nl.
LB2yvkZT3+8gKzLYlnlrhxbCmugYAe0R4mICsodskbBJaRDZUncObYJZv8a4ogZo6IIwswHj8EfwzofW6ZXfcrXAymNYQ
adD38Iht7Xc2S3axpAwZ2jKA/CnlBI9trB4WIwb8zLBbH1sCKrFIofa+2r8h1J2Gv6AU7hjbLHK5dCMP7MlkqO54t9ENDqC6AgvKMn6/miw7xrI+9hK6VLvxjv/zQddWa8S+EX8waYVUC9sZI2f2SYWVgS3xAkOyn0PXyr7/mZ6llssSLJ7UZ9AGB
sitpJpimw+1FqjiX5jls4tr8VsSONhsb+a7v/d8n5EoPgCwuhUT8viJxoSFcm5Iw==

kvi.nl. 86400 IN NS dns1.kvi.nl.
kvi.nl. 86400 IN NS dns2.kvi.nl.
kvi.nl. 86400 IN NS dwalin.nikhef.nl.
kvi.nl. 86400 IN RRSIG NS 8 2 86400 20160929021424 20160914141121
31771 KVI.nl.
xmrTUJo4xM9vzhah0tQ1sPoEub2KEajKEjUjgrKCXNFsdmrVge/3iP8rpcjukSxOXQ4zHTGprFKxzyBFgWtkzZRQHX9dD/DI
iLIWoJ2Wh1xKTfWSTydmrP5C3E7HR6y6fEZqJ16p6Wu/eAjbf3yPcRKHLXePWjbNFVXVrbuycw4=

The retiring keys are not present in the zone.


The retiring KSK is the old backup KSK from ods 1.4.10.
One of the retiring ZSK is the old backup ZSK from ods 1.4.10.
The other retiring ZSK is the old active ZSK.
The ready ZSK is the new ZSK. However, there is no active ZSK.

The ready ZSK is used to sign the SOA record, but the retiring ZSK 31771 is
still used to sign other records, but it is not present in the zone.
So, now many of the records do not have a validated signature.
Any idea how this can be solved? Will the ready key become active at the
next transition?
Yuri Schaeffer
2016-09-19 08:15:41 UTC
Permalink
Hi Fred,

Can you send me the output of:
ods-enforcer key list -d

If possible, can you send me off list your kasp.db (assuming sqlite),
your kasp.xml. and the produced signconf for that zone? Then I can see
if it is perhaps I migration related issue.

Regards,
Yuri
Post by Fred Zwarts, KVI, Groningen
We have ods 2.0.1 running for some time, but now a ZSK roll-over is
giving a problem.
# ods-enforcer key list --verbose
Zone: Keytype: State: Date of next
KVI.nl KSK retire 2016-09-17 11:00:06
2048 8 d70448361bf9ded4888de4bb681a9963 SoftHSM 23384
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 664dd2e6d61153c53f99ac2dcafddbda SoftHSM 31771
KVI.nl KSK active 2016-09-17 11:00:06
2048 8 333e0824ef6fc70c2729b02a88be92c7 SoftHSM 61849
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 6d31f5b7f2db0bc65fcb35f60ecceb1e SoftHSM 15381
KVI.nl ZSK ready 2016-09-17 11:00:06
1024 8 3c246656cd56b7cfd5294f5cb8e02229 SoftHSM 43923
key list completed in 0 seconds.
kvi.nl. 86400 IN SOA dns1.kvi.nl. hostmaster.kvi.nl.
2016091613 43200 3600 345600 10800
kvi.nl. 86400 IN RRSIG SOA 8 2 86400 20160930151204
20160916173547 43923 KVI.nl.
a1quYQgmEnAmt2BUdt3PAcEQ4mFCoLIULLEKKoICataE7OuXAbdhfjE9hT0nJeJPiLm6jmJmyj6fM2PwEb9DHS+PMulUc1L
snwayUoylsXm0HUFiAvG7+/tt2UYgybGCrXYWrrTJuu/VxMPSb4Qy5uEdwfEQRKs5w5Aeqci7aUQ=
kvi.nl. 3600 IN DNSKEY 257 3 8
AwEAAb9i0ycPgnT71XuBrWg7XuvEcUcmhLsWtXsO/vmg3xpWiYR1wW15rEMvloZ7Bl7O4/42to8GlQHx0yY1r1Kx4mkFtH6Mol31QXE8vwk4JaG7dW3UJKCWAjLD2mrBhp0umzDQK5dlkE+9o
m0sjcz2aUASNAQqwh38qOl8+3jNGbfjaw9MGK1WMYRv805NGGgPnmQ1BoB/4d99nhzqAfAWLWRLCoxD2FWjbUm+cQCft+YMtzEk46Ua1H/g/0B38E/2A71fUMWfGM5tE0XuArpFc7ri81MAzEHl5gsYGgn4QnGlsg8ip0wFZns/1NndgXpnjMlSel
vp4EEC8RCBKJ7E5IM= ;{id = 61849 (ksk), size = 2048b}
kvi.nl. 3600 IN DNSKEY 256 3 8
AwEAAbuEIkm1DbfRGZVFEfJ2BfD2h1us5RD85wTAZpXI9UfHpEjj86ApLn4uctHza1/ekkNAwy4aOgsz+TxLrvAhfKLfQL17q44ty6PDw8jQcinA8LIqB9xo9umvVagCHQeTTkoTRdHjh3DLQ
Fw9ice4N+7emoi+NTtTEa5pg9r1L41X ;{id = 43923 (zsk), size = 1024b}
kvi.nl. 3600 IN RRSIG DNSKEY 8 2 3600 20160930105859
20160916140006 61849 KVI.nl.
LB2yvkZT3+8gKzLYlnlrhxbCmugYAe0R4mICsodskbBJaRDZUncObYJZv8a4ogZo6IIwswHj8EfwzofW6ZXfcrXAymNYQ
adD38Iht7Xc2S3axpAwZ2jKA/CnlBI9trB4WIwb8zLBbH1sCKrFIofa+2r8h1J2Gv6AU7hjbLHK5dCMP7MlkqO54t9ENDqC6AgvKMn6/miw7xrI+9hK6VLvxjv/zQddWa8S+EX8waYVUC9sZI2f2SYWVgS3xAkOyn0PXyr7/mZ6llssSLJ7UZ9AGB
sitpJpimw+1FqjiX5jls4tr8VsSONhsb+a7v/d8n5EoPgCwuhUT8viJxoSFcm5Iw==
kvi.nl. 86400 IN NS dns1.kvi.nl.
kvi.nl. 86400 IN NS dns2.kvi.nl.
kvi.nl. 86400 IN NS dwalin.nikhef.nl.
kvi.nl. 86400 IN RRSIG NS 8 2 86400 20160929021424
20160914141121 31771 KVI.nl.
xmrTUJo4xM9vzhah0tQ1sPoEub2KEajKEjUjgrKCXNFsdmrVge/3iP8rpcjukSxOXQ4zHTGprFKxzyBFgWtkzZRQHX9dD/DI
iLIWoJ2Wh1xKTfWSTydmrP5C3E7HR6y6fEZqJ16p6Wu/eAjbf3yPcRKHLXePWjbNFVXVrbuycw4=
The retiring keys are not present in the zone.
The retiring KSK is the old backup KSK from ods 1.4.10.
One of the retiring ZSK is the old backup ZSK from ods 1.4.10.
The other retiring ZSK is the old active ZSK.
The ready ZSK is the new ZSK. However, there is no active ZSK.
The ready ZSK is used to sign the SOA record, but the retiring ZSK 31771
is still used to sign other records, but it is not present in the zone.
So, now many of the records do not have a validated signature.
Any idea how this can be solved? Will the ready key become active at the
next transition?
_______________________________________________
Opendnssec-user mailing list
https://lists.opendnssec.org/mailman/listinfo/opendnssec-user
Fred.Zwarts
2016-09-22 08:00:31 UTC
Permalink
I forced another ZSK roll-over on our test system and the same problem
popped up.
There are now two retiring ZSKs and one ready ZSK, but no active ZSK.
In the zone file, many records are still signed with the retiring ZSK.
However, this ZSK itself is no longer in the signed zone file.
Could it be that the option <Standby>1</Standby> causes these problems?
I know it is experimental, but it worked well in 1.4.10.

"Yuri Schaeffer" schreef in bericht news:84b78896-1d8a-aadb-2628-***@nlnetlabs.nl...

Hi Fred,

Can you send me the output of:
ods-enforcer key list -d

If possible, can you send me off list your kasp.db (assuming sqlite),
your kasp.xml. and the produced signconf for that zone? Then I can see
if it is perhaps I migration related issue.

Regards,
Yuri
Post by Fred Zwarts, KVI, Groningen
We have ods 2.0.1 running for some time, but now a ZSK roll-over is
giving a problem.
# ods-enforcer key list --verbose
Zone: Keytype: State: Date of next
KVI.nl KSK retire 2016-09-17 11:00:06
2048 8 d70448361bf9ded4888de4bb681a9963 SoftHSM 23384
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 664dd2e6d61153c53f99ac2dcafddbda SoftHSM 31771
KVI.nl KSK active 2016-09-17 11:00:06
2048 8 333e0824ef6fc70c2729b02a88be92c7 SoftHSM 61849
KVI.nl ZSK retire 2016-09-17 11:00:06
1024 8 6d31f5b7f2db0bc65fcb35f60ecceb1e SoftHSM 15381
KVI.nl ZSK ready 2016-09-17 11:00:06
1024 8 3c246656cd56b7cfd5294f5cb8e02229 SoftHSM 43923
key list completed in 0 seconds.
kvi.nl. 86400 IN SOA dns1.kvi.nl. hostmaster.kvi.nl.
2016091613 43200 3600 345600 10800
kvi.nl. 86400 IN RRSIG SOA 8 2 86400 20160930151204
20160916173547 43923 KVI.nl.
a1quYQgmEnAmt2BUdt3PAcEQ4mFCoLIULLEKKoICataE7OuXAbdhfjE9hT0nJeJPiLm6jmJmyj6fM2PwEb9DHS+PMulUc1L
snwayUoylsXm0HUFiAvG7+/tt2UYgybGCrXYWrrTJuu/VxMPSb4Qy5uEdwfEQRKs5w5Aeqci7aUQ=
kvi.nl. 3600 IN DNSKEY 257 3 8
AwEAAb9i0ycPgnT71XuBrWg7XuvEcUcmhLsWtXsO/vmg3xpWiYR1wW15rEMvloZ7Bl7O4/42to8GlQHx0yY1r1Kx4mkFtH6Mol31QXE8vwk4JaG7dW3UJKCWAjLD2mrBhp0umzDQK5dlkE+9o
m0sjcz2aUASNAQqwh38qOl8+3jNGbfjaw9MGK1WMYRv805NGGgPnmQ1BoB/4d99nhzqAfAWLWRLCoxD2FWjbUm+cQCft+YMtzEk46Ua1H/g/0B38E/2A71fUMWfGM5tE0XuArpFc7ri81MAzEHl5gsYGgn4QnGlsg8ip0wFZns/1NndgXpnjMlSel
vp4EEC8RCBKJ7E5IM= ;{id = 61849 (ksk), size = 2048b}
kvi.nl. 3600 IN DNSKEY 256 3 8
AwEAAbuEIkm1DbfRGZVFEfJ2BfD2h1us5RD85wTAZpXI9UfHpEjj86ApLn4uctHza1/ekkNAwy4aOgsz+TxLrvAhfKLfQL17q44ty6PDw8jQcinA8LIqB9xo9umvVagCHQeTTkoTRdHjh3DLQ
Fw9ice4N+7emoi+NTtTEa5pg9r1L41X ;{id = 43923 (zsk), size = 1024b}
kvi.nl. 3600 IN RRSIG DNSKEY 8 2 3600 20160930105859
20160916140006 61849 KVI.nl.
LB2yvkZT3+8gKzLYlnlrhxbCmugYAe0R4mICsodskbBJaRDZUncObYJZv8a4ogZo6IIwswHj8EfwzofW6ZXfcrXAymNYQ
adD38Iht7Xc2S3axpAwZ2jKA/CnlBI9trB4WIwb8zLBbH1sCKrFIofa+2r8h1J2Gv6AU7hjbLHK5dCMP7MlkqO54t9ENDqC6AgvKMn6/miw7xrI+9hK6VLvxjv/zQddWa8S+EX8waYVUC9sZI2f2SYWVgS3xAkOyn0PXyr7/mZ6llssSLJ7UZ9AGB
sitpJpimw+1FqjiX5jls4tr8VsSONhsb+a7v/d8n5EoPgCwuhUT8viJxoSFcm5Iw==
kvi.nl. 86400 IN NS dns1.kvi.nl.
kvi.nl. 86400 IN NS dns2.kvi.nl.
kvi.nl. 86400 IN NS dwalin.nikhef.nl.
kvi.nl. 86400 IN RRSIG NS 8 2 86400 20160929021424
20160914141121 31771 KVI.nl.
xmrTUJo4xM9vzhah0tQ1sPoEub2KEajKEjUjgrKCXNFsdmrVge/3iP8rpcjukSxOXQ4zHTGprFKxzyBFgWtkzZRQHX9dD/DI
iLIWoJ2Wh1xKTfWSTydmrP5C3E7HR6y6fEZqJ16p6Wu/eAjbf3yPcRKHLXePWjbNFVXVrbuycw4=
The retiring keys are not present in the zone.
The retiring KSK is the old backup KSK from ods 1.4.10.
One of the retiring ZSK is the old backup ZSK from ods 1.4.10.
The other retiring ZSK is the old active ZSK.
The ready ZSK is the new ZSK. However, there is no active ZSK.
The ready ZSK is used to sign the SOA record, but the retiring ZSK 31771
is still used to sign other records, but it is not present in the zone.
So, now many of the records do not have a validated signature.
Any idea how this can be solved? Will the ready key become active at the
next transition?
Yuri Schaeffer
2016-09-22 08:24:09 UTC
Permalink
Post by Fred.Zwarts
I forced another ZSK roll-over on our test system and the same problem
popped up.
There are now two retiring ZSKs and one ready ZSK, but no active ZSK.
In the zone file, many records are still signed with the retiring ZSK.
However, this ZSK itself is no longer in the signed zone file.
To debug this I could really use your database or at the very least the
output of
ods-enforcer key list -d

//Yuri
Post by Fred.Zwarts
Could it be that the option <Standby>1</Standby> causes these problems?
unlikely.
Fred.Zwarts
2016-09-22 08:38:50 UTC
Permalink
# ods-enforcer key list -d
Keys:
Zone: Key role: DS: DNSKEY:
RRSIGDNSKEY: RRSIG: Pub: Act: Id:
40.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 956d1b9309f7db3f5dd407c5a2153d64
56.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 d4029c2ca8e7c952bd091a8c26368769
81.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 f2c94f383a7e678a6828e62f8b13ebd6
0.6.0.0.8.0.a.1.0.1.6.0.1.0.0.2.ip6.arpa KSK omnipresent
omnipresent omnipresent NA 1 1
5c7cad2b7f74ca4deb3ed7e85cef753d
37.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 ce20f66ea4acc883fd1268e06d85f379
81.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 df12be2f9455e37f72988c0ec5de9134
0.6.0.0.8.0.a.1.0.1.6.0.1.0.0.2.ip6.arpa ZSK NA
omnipresent NA omnipresent 1 1
4ecdf02db5c3fb7cc75f4ed790a48c35
KVI.nl ZSK NA hidden NA
hidden 0 0 d5104974928d9d3b962efe9cdb0d423c
56.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 5caf384ceb8704d78ba171dd26317994
40.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 0db0fe4431642f178a1130330e87420e
27.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 ea51bf3290a73b7f69593f7b971c6edb
37.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 b03d9281a6d5337b10081a0a1495efc7
27.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 c59608a2dee24c15f4b520b42e4c6c59
15.125.129.in-addr.arpa KSK omnipresent omnipresent
omnipresent NA 1 1 a1fc7f463d0418e31a00452c7f43d613
81.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 e781f75513f9f0298178acb9276b41b0
0.6.0.0.8.0.a.1.0.1.6.0.1.0.0.2.ip6.arpa ZSK NA
omnipresent NA omnipresent 1 1
35a44e60dcd7016d324cadbdb25cdd0b
KVI.nl ZSK NA omnipresent NA
unretentive 1 0 63b58e329df2a6bfa09671425146b72d
56.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 a9c86ca62462256f7a199fd54f50423a
40.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 41501fedde3f3380fa05753010f2f022
KVI.nl KSK omnipresent omnipresent
omnipresent NA 1 1 933aa76282968a1212c6dd6de92a5dae
37.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 7caf0a333030e34d5bbb80540ce4a981
27.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 30edff1e19f2d00d9896757ebbae0b71
15.125.129.in-addr.arpa ZSK NA omnipresent NA
omnipresent 1 1 b2f99fddb59f1d6190c807cccd219c09
KVI.nl ZSK NA omnipresent NA
rumoured 1 1 0ef4982714ed47c4cf84c87e62c38890
key list completed in 0 seconds.


# ods-enforcer key list --verbose --zone KVI.nl
Keys:
Zone: Keytype: State: Date of next transition:
Size: Algorithm: CKA_ID: Repository: KeyTag:
KVI.nl ZSK retire 2016-10-05 00:29:43
1024 8 d5104974928d9d3b962efe9cdb0d423c SoftHSM 30271
KVI.nl ZSK retire 2016-10-05 00:29:43
1024 8 63b58e329df2a6bfa09671425146b72d SoftHSM 20904
KVI.nl KSK active 2016-10-05 00:29:43
2048 8 933aa76282968a1212c6dd6de92a5dae SoftHSM 38854
KVI.nl ZSK ready 2016-10-05 00:29:43
1024 8 0ef4982714ed47c4cf84c87e62c38890 SoftHSM 13143
key list completed in 0 seconds.
Post by Fred.Zwarts
I forced another ZSK roll-over on our test system and the same problem
popped up.
There are now two retiring ZSKs and one ready ZSK, but no active ZSK.
In the zone file, many records are still signed with the retiring ZSK.
However, this ZSK itself is no longer in the signed zone file.
To debug this I could really use your database or at the very least the
output of
ods-enforcer key list -d

//Yuri
Post by Fred.Zwarts
Could it be that the option <Standby>1</Standby> causes these problems?
unlikely.
Fred.Zwarts
2016-09-22 10:55:05 UTC
Permalink
Sorry, I forgot the database. See attachment.
Yuri Schaeffer
2016-09-22 10:58:04 UTC
Permalink
Hi Fred,

We are currently in the process of finding out why the retired ZSK after
the migration gets unpublished to fast. It seems an issue in the
migration script. Working on it.

This issue seems unrelated. Judging from the output the old ZSK DNSKEY
is still being published in the DNSKEY set. At least what the enforcer
KVI.nl ZSK NA hidden NA hidden 0 0 d5104974928d9d3b962efe9cdb0d423c
KVI.nl ZSK NA omnipresent NA unretentive 1 0 63b58e329df2a6bfa09671425146b72d
KVI.nl ZSK NA omnipresent NA rumoured 1 1 0ef4982714ed47c4cf84c87e62c38890
KVI.nl ZSK retire 2016-10-05 00:29:43 1024 8 d5104974928d9d3b962efe9cdb0d423c SoftHSM 30271
KVI.nl ZSK retire 2016-10-05 00:29:43 1024 8 63b58e329df2a6bfa09671425146b72d SoftHSM 20904
KVI.nl ZSK ready 2016-10-05 00:29:43 1024 8 0ef4982714ed47c4cf84c87e62c38890 SoftHSM 13143
Notice the "Pub" flag on key 63b58e329df2a6bfa09671425146b72d and
0ef4982714ed47c4cf84c87e62c38890.

The signer should include both keys in the set. 2 things could be happening:

1) A bug in the enforcer where it outputs the wrong signconf. Please
check the entry for the 63b58e329df2a6bfa09671425146b72d key in the
signconf. it should have a <ZSK/> element.

2) A bug in the signer where it fails to include the DNSKEY. I find this
unlikely. Since it is explicitly told to do so and this code did not see
many changes for quite a while.

(3) I almost don't dare to mention it: The DNSKEY is overlooked in the
signed file. It looks like the above mentioned problem of the faulty
migration and having no key in the 'active' is confusing?

//Yuri
Yuri Schaeffer
2016-09-22 11:04:18 UTC
Permalink
Post by Yuri Schaeffer
1) A bug in the enforcer where it outputs the wrong signconf. Please
check the entry for the 63b58e329df2a6bfa09671425146b72d key in the
signconf. it should have a <ZSK/> element.
Oops. That should be a <Publish/> element for key
63b58e329df2a6bfa09671425146b72d.

The <ZSK/> (meaning use it for signing as ZSK) should in fact NOT be there!

//Yuri
Fred.Zwarts
2016-09-22 14:11:00 UTC
Permalink
Hi,

I have attached the signconf file for the domain and the complete signed
zone file.
In the zone file, there are a lot of records signed with the ZSK with KeyTag
30271, but there is no DNSKEY with this tag in the zone.
In the signconf file this key has no <Publish\> in its section.
Note, this is not the key starting with 63b58..., but with d5104...
I hope that this helps to find the cause of the problem.
I think option 1) is the most probable one.

"Yuri Schaeffer" schreef in bericht news:dc5e47c4-7701-f695-6207-***@nlnetlabs.nl...

Hi Fred,

We are currently in the process of finding out why the retired ZSK after
the migration gets unpublished to fast. It seems an issue in the
migration script. Working on it.

This issue seems unrelated. Judging from the output the old ZSK DNSKEY
is still being published in the DNSKEY set. At least what the enforcer
Post by Fred.Zwarts
KVI.nl ZSK NA hidden NA
hidden 0 0 d5104974928d9d3b962efe9cdb0d423c
KVI.nl ZSK NA omnipresent NA
unretentive 1 0 63b58e329df2a6bfa09671425146b72d
KVI.nl ZSK NA omnipresent NA
rumoured 1 1 0ef4982714ed47c4cf84c87e62c38890
Zone: Keytype: State: Date of next
KVI.nl ZSK retire 2016-10-05 00:29:43
1024 8 d5104974928d9d3b962efe9cdb0d423c SoftHSM 30271
KVI.nl ZSK retire 2016-10-05 00:29:43
1024 8 63b58e329df2a6bfa09671425146b72d SoftHSM 20904
KVI.nl ZSK ready 2016-10-05 00:29:43
1024 8 0ef4982714ed47c4cf84c87e62c38890 SoftHSM 13143
Notice the "Pub" flag on key 63b58e329df2a6bfa09671425146b72d and
0ef4982714ed47c4cf84c87e62c38890.

The signer should include both keys in the set. 2 things could be happening:

1) A bug in the enforcer where it outputs the wrong signconf. Please
check the entry for the 63b58e329df2a6bfa09671425146b72d key in the
signconf. it should have a <ZSK/> element.

2) A bug in the signer where it fails to include the DNSKEY. I find this
unlikely. Since it is explicitly told to do so and this code did not see
many changes for quite a while.

(3) I almost don't dare to mention it: The DNSKEY is overlooked in the
signed file. It looks like the above mentioned problem of the faulty
migration and having no key in the 'active' is confusing?

//Yuri
Yuri Schaeffer
2016-09-23 09:42:26 UTC
Permalink
Hi Fred,

Thanks for sharing the data, I now understand what has happened. The
root cause must have been an error in the migration script. I'll write
it down in detail so you can verify your part of the events.

1) Before migration there where two ZSKs in a rollover. Lets call those
ZSK1(old) and ZSK2(new).

2) migration script was executed. ZSK2 was wrongfully marked as entirely
propagated. (but in fact only some of the signatures where generated
with this key)

3) enforcer ran, concluded ZSK1 could be removed, instructed the signer
to stop publishing the DNSKEY of ZSK1. But the signer kept reusing
signatures of this key.

4) Now the user issued a rollover to ZSK3 to fix the situation. But now
we are in a situation where we still have signatures from ZSK1 and ZSK2.
Both will be replaced by signatures of ZSK3 over the course of 14 days.
(signature validity in KASP).


To come out of this situation you could issue a
ods-signer clear kvi.nl
All signatures will then be regenerated at the next sign run. All of
them with ZSK3

For us to do:
1) Fix migration script to better recognise current rollover.
2) Make sure the signer doesn't keep signatures of a key that is no
longer active or publish.

Regards,
Yuri
Fred.Zwarts
2016-09-26 09:51:03 UTC
Permalink
Hi Yuri,

I have been away a few days, so sorry for the late response.

I am not sure that your diagnosis is the whole story.

We have had two cases of this problem. In the first case your diagnosis may
apply, because it happened rather soon after the migration. However, at the
moment of the migration, there was no roll-over in progress, but there were
two KSKs (one active, one standby) and two ZSKs (one active, one standby).
Soon (two days) after the migration a scheduled ZSK roll-over started.

The second case, on a different system, however, (from which I sent you the
database) happened when ods had been running for about one month. There were
no keys left from the migration, because a KSK and a ZSK roll-over had
completed already. At that moment there was one KSK and there was one active
ZSK and one ready (standby) ZSK. Then I forced a ZSK roll-over. So, I still
think that the problem is not (only) the migration, but also the use of a
standby ZSK.

But, anyhow, it is good to make sure the signer doesn't keep signatures of a
key that is no longer active or publish.
But the question remains: what should the signer do if there are no ZSKs
active of publish?
We now have the situation with two retiring ZSKs and one ready ZSK.
How long do we have to wait, till the ready ZSK will become active?

Thanks, for your help,
Fred.Zwarts.

"Yuri Schaeffer" schreef in bericht news:2c127074-c0c2-2132-6da0-***@nlnetlabs.nl...

Hi Fred,

Thanks for sharing the data, I now understand what has happened. The
root cause must have been an error in the migration script. I'll write
it down in detail so you can verify your part of the events.

1) Before migration there where two ZSKs in a rollover. Lets call those
ZSK1(old) and ZSK2(new).

2) migration script was executed. ZSK2 was wrongfully marked as entirely
propagated. (but in fact only some of the signatures where generated
with this key)

3) enforcer ran, concluded ZSK1 could be removed, instructed the signer
to stop publishing the DNSKEY of ZSK1. But the signer kept reusing
signatures of this key.

4) Now the user issued a rollover to ZSK3 to fix the situation. But now
we are in a situation where we still have signatures from ZSK1 and ZSK2.
Both will be replaced by signatures of ZSK3 over the course of 14 days.
(signature validity in KASP).


To come out of this situation you could issue a
ods-signer clear kvi.nl
All signatures will then be regenerated at the next sign run. All of
them with ZSK3

For us to do:
1) Fix migration script to better recognise current rollover.
2) Make sure the signer doesn't keep signatures of a key that is no
longer active or publish.

Regards,
Yuri
Fred.Zwarts
2016-10-04 09:06:43 UTC
Permalink
Hi Yuri,

Is there any progress on this matter? I have a strong impression that the
problem is not (only) caused by migration problems. It seems to happen
always if a standby ZSK is configured. When after some days of changing keys
states the system enters a more stable situation, then I see that there are
two active ZSKs. Both ZSKs have the <Publish/> attribute in the signer
configuration and both of them are found in the signed zone, although only
one is used in the RRSIG records. (So, it would be better if the enforcer
would show one as active and the other one as ready, but that is a minor
problem.)
When I force a ZSK roll-over, then both ZSKs go to the retire state and a
new ZSK goes to the publish and later to the ready state. But only one of
the retiring ZSKs is still present in the signed zone and unfortunately, it
is the wrong one, the one that is not used in the RRSIG records. So, there
are then many RRSIG records using an ZSK that is no longer present in the
signed zone.

When I set standby to 0, then after some days only one ZSK is left in the
active state. If I then force a roll-over no problem is seen. The retiring
ZSK stays in the signed zone untill all RRSIG records using this ZSK have
been replaced. During the transition, there is no active ZSK, but one ZSK is
in the retire state and the new ZSK is in the publish or ready state. (I
would expect already an active state, but that is a minor problem.) During
the transition the ZSKs used in the RRSIG are always present in the signed
zone.

So, I have now set standby to 0, hoping that this will avoid further
problems.

I wonder if you can reproduce this problem with standby ZSKs?

Regards,
Fred.Zwarts.

"Fred.Zwarts" schreef in bericht news:nsar1v$2af$***@blaine.gmane.org...

Hi Yuri,

I have been away a few days, so sorry for the late response.

I am not sure that your diagnosis is the whole story.

We have had two cases of this problem. In the first case your diagnosis may
apply, because it happened rather soon after the migration. However, at the
moment of the migration, there was no roll-over in progress, but there were
two KSKs (one active, one standby) and two ZSKs (one active, one standby).
Soon (two days) after the migration a scheduled ZSK roll-over started.

The second case, on a different system, however, (from which I sent you the
database) happened when ods had been running for about one month. There were
no keys left from the migration, because a KSK and a ZSK roll-over had
completed already. At that moment there was one KSK and there was one active
ZSK and one ready (standby) ZSK. Then I forced a ZSK roll-over. So, I still
think that the problem is not (only) the migration, but also the use of a
standby ZSK.

But, anyhow, it is good to make sure the signer doesn't keep signatures of a
key that is no longer active or publish.
But the question remains: what should the signer do if there are no ZSKs
active of publish?
We now have the situation with two retiring ZSKs and one ready ZSK.
How long do we have to wait, till the ready ZSK will become active?

Thanks, for your help,
Fred.Zwarts.

"Yuri Schaeffer" schreef in bericht news:2c127074-c0c2-2132-6da0-***@nlnetlabs.nl...

Hi Fred,

Thanks for sharing the data, I now understand what has happened. The
root cause must have been an error in the migration script. I'll write
it down in detail so you can verify your part of the events.

1) Before migration there where two ZSKs in a rollover. Lets call those
ZSK1(old) and ZSK2(new).

2) migration script was executed. ZSK2 was wrongfully marked as entirely
propagated. (but in fact only some of the signatures where generated
with this key)

3) enforcer ran, concluded ZSK1 could be removed, instructed the signer
to stop publishing the DNSKEY of ZSK1. But the signer kept reusing
signatures of this key.

4) Now the user issued a rollover to ZSK3 to fix the situation. But now
we are in a situation where we still have signatures from ZSK1 and ZSK2.
Both will be replaced by signatures of ZSK3 over the course of 14 days.
(signature validity in KASP).


To come out of this situation you could issue a
ods-signer clear kvi.nl
All signatures will then be regenerated at the next sign run. All of
them with ZSK3

For us to do:
1) Fix migration script to better recognise current rollover.
2) Make sure the signer doesn't keep signatures of a key that is no
longer active or publish.

Regards,
Yuri
Berry A.W. van Halderen
2016-10-04 09:36:39 UTC
Permalink
Post by Fred.Zwarts
Hi Yuri,
Is there any progress on this matter? I have a strong impression that
the problem is not (only) caused by migration problems. It seems to
happen always if a standby ZSK is configured.
Yuri is out of the office for today, but we have some progress as we
think we have encountered a similar issue, though not quite the same.
The issue actually hit us painfully hard as we have not installed
normal safety procedures to avoid problems :-(
Post by Fred.Zwarts
I wonder if you can reproduce this problem with standby ZSKs?
However with us there are no standby keys. The problem we encountered
was really caused by the migration, and although we cannot map it
directly to your case, we have new leads to pursue the issue and are
following up onto it. Probably either migration or with you another
cause causes keys to be partially seen in a inconsistent state, but
we really need a couple more days to review and replay.

\Berry

Yuri Schaeffer
2016-09-20 13:07:10 UTC
Permalink
Hi Fred,

My colleague Hoda found the error. The SOA serial strategy is numbered
differently between 1.4 and 2.0. This is actually a problem with the
migration script not taking this in to account.

What should solve your issue is running

ods-enforcer policy import

Your kasp.xml will be reread and any differences applied.

Alternatively you could do it manually in your database (assuming
default policy):

UPDATE policy SET zoneSoaSerial=1 WHERE name = 'default';

I expect that field in your database to be 3. Which was 'datacounter' in
1.4. But maps to 'keep' in 2.0.

For us left to do is update the migration script.

Regards,
Yuri
Fred.Zwarts
2016-09-22 07:09:51 UTC
Permalink
Hi Yuri,

I have been a few days away, so I read your message now.

I am a bit confused about your reply. Does it refer to my first question, in
an earlier mail, about the refusal of the signer to sign the zone because of
the serial?
This was indeed solved with "ods-enforcer policy import".
However, a few days later we got this new problem with a ZSK roll-over,
where ods 2.0.1 completely ruined the zone. No active ZSK was left. The
retiring keys were not in the signed zone, but most of the records were
still signed with the retiring keys. Some more "ods-enforcer policy import"
did not help (of course). Only a few records were signed with the ready ZSK,
which was also in the zone. Only those records could be used with DNSsec
verification.
Finally, my collegue deleted the zone from the database.
So, I am not able to send you any other information.

Could it be that this problem was also caused by a migration problem, or is
it something else?

Regards,
Fred.Zwarts.


"Yuri Schaeffer" schreef in bericht news:0bc2193f-292a-4952-5791-***@nlnetlabs.nl...

Hi Fred,

My colleague Hoda found the error. The SOA serial strategy is numbered
differently between 1.4 and 2.0. This is actually a problem with the
migration script not taking this in to account.

What should solve your issue is running

ods-enforcer policy import

Your kasp.xml will be reread and any differences applied.

Alternatively you could do it manually in your database (assuming
default policy):

UPDATE policy SET zoneSoaSerial=1 WHERE name = 'default';

I expect that field in your database to be 3. Which was 'datacounter' in
1.4. But maps to 'keep' in 2.0.

For us left to do is update the migration script.

Regards,
Yuri
Yuri Schaeffer
2016-09-22 08:02:13 UTC
Permalink
Post by Fred.Zwarts
I am a bit confused about your reply. Does it refer to my first
question, in an earlier mail, about the refusal of the signer to sign
the zone because of the serial?
Oh yes sorry, I replied to the wrong thread.
Post by Fred.Zwarts
<cut>
Could it be that this problem was also caused by a migration problem, or
is it something else?
I don't know at this point. I'll look in to it today. But migration
would be my first suspicion.

Regards,
Yuri
Loading...