[Opendnssec-user] End-of-life OpenDNSSEC 1.3 on 2017-07-11

[Berry A.W. van Halderen]

Dear community.

With the release of OpenDNSSEC 2.0 we have reached the point to let go
of the support of one of the older, not to say elderly, OpenDNSSEC.

We will end the support on OpenDNSSEC 1.3 in accordance with our policy
in one year from now. This historic release has long been replaced by
most of not all so we regard this notice to be mostly a formal message
for most of you.

OpenDNSSEC 1.4 has been the de-facto Long Term Support version, and will
continue to be the official LTS version. This at least for over a year
and until OpenDNSSEC 2.x has matured. This will be the case as soon as
it has been around long enough to prove itself.

Please note however that we will concentrate our development on the 2.x
versions. Features and improvements that are not considered bugs are
mostly targeted at that steam of development. We believe OpenDNSSEC 2.0
is already stable to be used.

With kind regards,
The OpenDNSSEC team

[Wytze van der Raay]

Post by Berry A.W. van Halderen
With the release of OpenDNSSEC 2.0 we have reached the point to let go
of the support of one of the older, not to say elderly, OpenDNSSEC.

Congratulations with the final release of 2.0 !!

Post by Berry A.W. van Halderen
We will end the support on OpenDNSSEC 1.3 in accordance with our policy
in one year from now. This historic release has long been replaced by
most of not all so we regard this notice to be mostly a formal message
for most of you.

Most perhaps, but definitely not all -- CAcert.org is still running the
1.3.18 (LTS) release, as it is mostly doing what we need (migration to
newer algorithms would be nice though :-)).

I noticed a MIGRATION document in the 2.0 release kit, but it only talks
about migrating from 1.4 to 2.0. Can you also provide instructions for
migrating from 1.3.18 to 2.0?

Regards,
Wytze van der Raay

[Berry A.W. van Halderen]

Post by Wytze van der Raay

Post by Berry A.W. van Halderen
We will end the support on OpenDNSSEC 1.3 in accordance with our policy
in one year from now. This historic release has long been replaced by
most of not all so we regard this notice to be mostly a formal message
for most of you.

Most perhaps, but definitely not all -- CAcert.org is still running the
1.3.18 (LTS) release, as it is mostly doing what we need (migration to
newer algorithms would be nice though :-)).

We do not maintain a list of who is running what, sometimes difficult to
tell what wording to use. In any case, 1.3 has two newer alternatives
that are much better maintained.

Post by Wytze van der Raay
I noticed a MIGRATION document in the 2.0 release kit, but it only talks
about migrating from 1.4 to 2.0. Can you also provide instructions for
migrating from 1.3.18 to 2.0?

In order to provide the most tested and predictable migration, it is
advisable to perform the migration steps from 1.3 to 1.4 first, and then
perform the migration to 2.0. If we make separate migration scripts for
each of the alternatives, this will not only involve more work for us,
but more important for the users, make the migration actually less
tested.

You can however do the migration pretty simple, as you can just grap
the latest 1.4 release (1.4.10). The instructions in the migration
can just migrate you to the latest 1.4, by starting at the instructions
from 1.3 to 1.4/trunk (see the included MIGRATION file in the
distribution). You do not need to start OpenDNSSEC in any of the in
between steps. And without actually starting you can then grab
the 2.0 release and migrate to it. This ought to work. The 2.0
has a completely revised database schema.

It is actually only a few steps still, a direct migration from 1.3 to
2.0 would not be that much shorter.

With kind regards,
Berry van Halderen

Post by Wytze van der Raay
Regards,
Wytze van der Raay
_______________________________________________
Opendnssec-user mailing list
https://lists.opendnssec.org/mailman/listinfo/opendnssec-user

[Wytze van der Raay]

Post by Berry A.W. van Halderen
...
In order to provide the most tested and predictable migration, it is
advisable to perform the migration steps from 1.3 to 1.4 first, and then
perform the migration to 2.0. If we make separate migration scripts for
each of the alternatives, this will not only involve more work for us,
but more important for the users, make the migration actually less
tested.

That makes sense.

Post by Berry A.W. van Halderen
You can however do the migration pretty simple, as you can just grap
the latest 1.4 release (1.4.10). The instructions in the migration
can just migrate you to the latest 1.4, by starting at the instructions
from 1.3 to 1.4/trunk (see the included MIGRATION file in the
distribution). You do not need to start OpenDNSSEC in any of the in
between steps. And without actually starting you can then grab
the 2.0 release and migrate to it. This ought to work. The 2.0
has a completely revised database schema.
It is actually only a few steps still, a direct migration from 1.3 to
2.0 would not be that much shorter.

It seems pretty easy, so I started on this.

Executing the instructions for upgrading from 1.3 to 1.4/trunk went fine.
You did not mention this, but I've assumed that the steps for migrating
from 1.4.X to 1.4.8 should also be performed, so I did that next. Again
no problem.

However, then the 2.0 migration instructions point me to
enforcer/utils/1.4-2.0_db_convert/README.md. This instructs me
to execute the convert_sqlite script. The script starts off with
a reference SCHEMA=../../src/db/schema.sqlite, but that schema.sqlite
file does not exist anywhere in the src distro.
Should it? Or should it be generated somehow?

In any case, here I am stuck ... please advise.

Regards,
Wytze van der Raay

[Yuri Schaeffer]

Hi Wytze,

Post by Wytze van der Raay
However, then the 2.0 migration instructions point me to
enforcer/utils/1.4-2.0_db_convert/README.md. This instructs me
to execute the convert_sqlite script. The script starts off with
a reference SCHEMA=../../src/db/schema.sqlite, but that schema.sqlite
file does not exist anywhere in the src distro.
Should it? Or should it be generated somehow?

I just checked the release tarball and indeed it isn't there! Quick
workaround check out the complete source from github.

git clone https://github.com/opendnssec/opendnssec.git
-or-
https://github.com/opendnssec/opendnssec/archive/master.zip

I'll figure out later today why the scripts are absent.

//Yuri

[Yuri Schaeffer]

Alternatively I prepared a new release which includes the files:

https://dist.opendnssec.org/source/opendnssec-2.0.0-1.tar.gz

Tonight I'll test it and make it an official release.

//Yuri

Post by Yuri Schaeffer
Hi Wytze,

Post by Wytze van der Raay
However, then the 2.0 migration instructions point me to
enforcer/utils/1.4-2.0_db_convert/README.md. This instructs me
to execute the convert_sqlite script. The script starts off with
a reference SCHEMA=../../src/db/schema.sqlite, but that schema.sqlite
file does not exist anywhere in the src distro.
Should it? Or should it be generated somehow?

I just checked the release tarball and indeed it isn't there! Quick
workaround check out the complete source from github.
git clone https://github.com/opendnssec/opendnssec.git
-or-
https://github.com/opendnssec/opendnssec/archive/master.zip
I'll figure out later today why the scripts are absent.
//Yuri
_______________________________________________
Opendnssec-user mailing list
https://lists.opendnssec.org/mailman/listinfo/opendnssec-user

[Wytze van der Raay]

Hi Yuri,

Post by Yuri Schaeffer

Post by Wytze van der Raay
However, then the 2.0 migration instructions point me to
enforcer/utils/1.4-2.0_db_convert/README.md. This instructs me
to execute the convert_sqlite script. The script starts off with
a reference SCHEMA=../../src/db/schema.sqlite, but that schema.sqlite
file does not exist anywhere in the src distro.
Should it? Or should it be generated somehow?

I just checked the release tarball and indeed it isn't there! Quick
workaround check out the complete source from github.

Thanks for the quick response and all follow-up actions, including
the new release. I managed to complete the upgrade (from 1.4.10 now)
to 2.0.0-1, but things were not as smooth as I had hoped for.

1. The MIGRATION file in the 2.0.0-1 tarball is an old one, it is
missing the upgrade instructions that were present in the 2.0.0
tarball.

2. After converting kasp.db and bringing up the new software, all
zones were immediately re-signed, but the SOA in each zone was
reset to the (old) datetime value in the unsigned copy, which
is much lower than the value in the signed zonefiles produced
by 1.4.10 and earlier. Thus my secondary servers did not accept
the newly re-signed zones.

I suspect that this is related to some error messages I noted in
the logs:
Jul 16 15:28:00 ns ods-signerd: [duration] cannot create from string 50: P not
found
Jul 16 15:28:00 ns ods-signerd: [duration] cannot create from string
datecounter: P not found
Jul 16 15:28:00 ns ods-signerd: [zone] corrupted backup file zone cacert.com:
read signconf error
Jul 16 15:28:00 ns ods-signerd: [engine] unable to recover zone cacert.com
from backup, performing full sign
which seems to indicate that the new signer cannot properly parse
the old intermediate files ... but there is nothing I can do about
that as a user, or is there?

I recovered from this bad state by manually increasing the SOA in
each unsigned zonefile, and issuing a re-sign for each of them.

3. The ods-enforcerd is logging requests which I do not understand,
for example:

ods-enforcerd: [enforce_task] please submit DS with keytag 62462 for
zone cacert.org

but the referenced key in each case is a ZSK according to key list:

cacert.org ZSK ready waiting for ds-submit
1024 7 1af3ac95b060f4b65fec26006587be18 SoftHSM 62462
cacert.org ZSK rumoured omnipresent
omnipresent rumoured 1 1 1af3ac95b060f4b65fec26006587be18

Why does this happen? What should I do about it?

4. For one (fortunately unimportant) zone, the situation is worse, the
ods-enforcerd tell me:

ods-enforcerd: [enforce_task] please retract DS with keytag 2701 for
zone cacert.community
ods-enforcerd: [enforce_task] please submit DS with keytag 65382 for
zone cacert.community

but key 2701 was until now my active KSK, while 65382 is a ZSK.
A new KSK 37255 has appeared for this zone in the key list, with state
'publish'. For the outside world the zone looks broken (the present DS
records/keys do not correspond to what's in the zone). It looks like
the 2.0.0. software has forced a KSK key roll for this zone without
me even asking ... is that expected behaviour??

Regards,
-- wytze

[Yuri Schaeffer]

Hi Wytze,

I'll try to answer as much as possible with what comes to mind. Further
analysis will have to wait a bit.

Post by Wytze van der Raay
Thanks for the quick response and all follow-up actions, including
the new release. I managed to complete the upgrade (from 1.4.10 now)
to 2.0.0-1, but things were not as smooth as I had hoped for.
1. The MIGRATION file in the 2.0.0-1 tarball is an old one, it is
missing the upgrade instructions that were present in the 2.0.0
tarball.

That's odd. All I did change was a Makefile.am. There must have been
something that has gone wrong in our release process. Will investigate.

Post by Wytze van der Raay
2. After converting kasp.db and bringing up the new software, all
zones were immediately re-signed, but the SOA in each zone was
reset to the (old) datetime value in the unsigned copy, which
is much lower than the value in the signed zonefiles produced
by 1.4.10 and earlier. Thus my secondary servers did not accept
the newly re-signed zones.

I think 2.0.0 *should* be able to parse the 1.4.10 signconf files.
However I'm less sure about earlier versions. At this point I'm not
entirely sure there is an upgrade path that allows one to keep the
signconf from old versions. This may mean loosing the SOA serial. We
need to properly document/guide this I think.

Post by Wytze van der Raay
I suspect that this is related to some error messages I noted in
Jul 16 15:28:00 ns ods-signerd: [duration] cannot create from string 50: P not
found
Jul 16 15:28:00 ns ods-signerd: [duration] cannot create from string
datecounter: P not found
read signconf error
Jul 16 15:28:00 ns ods-signerd: [engine] unable to recover zone cacert.com
from backup, performing full sign
which seems to indicate that the new signer cannot properly parse
the old intermediate files ... but there is nothing I can do about
that as a user, or is there?
I recovered from this bad state by manually increasing the SOA in
each unsigned zonefile, and issuing a re-sign for each of them.
3. The ods-enforcerd is logging requests which I do not understand,
ods-enforcerd: [enforce_task] please submit DS with keytag 62462 for
zone cacert.org
cacert.org ZSK ready waiting for ds-submit
1024 7 1af3ac95b060f4b65fec26006587be18 SoftHSM 62462
cacert.org ZSK rumoured omnipresent
omnipresent rumoured 1 1 1af3ac95b060f4b65fec26006587be18
Why does this happen? What should I do about it?

This is not right at all. The key identifies as a ZSK, but looks more
like a CSK (zsk+ksk)! I recommend to initiate a rollover for that zone.
For both the ksk and zsk. Make sure your policy in kasp.xml looks good.

Post by Wytze van der Raay
4. For one (fortunately unimportant) zone, the situation is worse, the
ods-enforcerd: [enforce_task] please retract DS with keytag 2701 for
zone cacert.community
ods-enforcerd: [enforce_task] please submit DS with keytag 65382 for
zone cacert.community
but key 2701 was until now my active KSK, while 65382 is a ZSK.
A new KSK 37255 has appeared for this zone in the key list, with state
'publish'. For the outside world the zone looks broken (the present DS
records/keys do not correspond to what's in the zone). It looks like
the 2.0.0. software has forced a KSK key roll for this zone without
me even asking ... is that expected behaviour??

It should not ask that for a ZSK. Something has gone wrong with the
database conversion I guess. Do you still have the database as produced
by 1.4.10? I'd like to compare it to your 2.0.0 database.

The second part: yes. Unless you specified <ManualRollover> for the KSK
it will automatically roll as your policy prescribes. However if you
don't indicate you retracted the old DS I expect both DNSKEYS to be
published in your zone. And therefore not broken. Is this not the case?


Kind regards,
Yuri

[Wytze van der Raay]

Hi Yuri,

Post by Yuri Schaeffer
I'll try to answer as much as possible with what comes to mind. Further
analysis will have to wait a bit.

That's fine.

Post by Yuri Schaeffer
...

Post by Wytze van der Raay
2. After converting kasp.db and bringing up the new software, all
zones were immediately re-signed, but the SOA in each zone was
reset to the (old) datetime value in the unsigned copy, which
is much lower than the value in the signed zonefiles produced
by 1.4.10 and earlier. Thus my secondary servers did not accept
the newly re-signed zones.

I think 2.0.0 *should* be able to parse the 1.4.10 signconf files.
However I'm less sure about earlier versions. At this point I'm not
entirely sure there is an upgrade path that allows one to keep the
signconf from old versions. This may mean loosing the SOA serial. We
need to properly document/guide this I think.

Your explanation is probably spot-on. Although I had been running with
1.4.10 for one day, I can observe now that the signconf files had not
been rewritten yet by the 1.4.10 software, so they were still in 1.3.18
format (apparently these files are not rewritten very often?)
But yes, this definitely requires some warnings in the MIGRATION docs.

Post by Yuri Schaeffer
...
This is not right at all. The key identifies as a ZSK, but looks more
like a CSK (zsk+ksk)! I recommend to initiate a rollover for that zone.
For both the ksk and zsk. Make sure your policy in kasp.xml looks good.

I've tried this for one zone, but only appears to make things worse :-(
According to 'key list', there is now a new KSK with id 330, but
ods-enforcerd wants me to submit the DS with keytag 11318, which
is a ZSK according to key list. And when I use key export -d to
get the DS to be submitted, it only gives me DS's for 11318 (and
some retired keys). How can I get the DS for keytag 330??

Right now, it looks to me that the only way to recover is to
turn off DNSSEC for all zones, and start from scratch :-(

Post by Yuri Schaeffer
...
It should not ask that for a ZSK. Something has gone wrong with the
database conversion I guess. Do you still have the database as produced
by 1.4.10? I'd like to compare it to your 2.0.0 database.

I will send you those off-list.

Post by Yuri Schaeffer
The second part: yes. Unless you specified <ManualRollover> for the KSK
it will automatically roll as your policy prescribes. However if you
don't indicate you retracted the old DS I expect both DNSKEYS to be
published in your zone. And therefore not broken. Is this not the case?

I did *not* retract the old DS, but its DNSKEY is not published anymore in
the zone, so the zone is broken (by OpenDNSSEC 2.0 I am sad to conclude).

Regards,
-- wytze

[Wytze van der Raay]

Post by Wytze van der Raay

Post by Yuri Schaeffer
...
This is not right at all. The key identifies as a ZSK, but looks more
like a CSK (zsk+ksk)! I recommend to initiate a rollover for that zone.
For both the ksk and zsk. Make sure your policy in kasp.xml looks good.

I've tried this for one zone, but only appears to make things worse :-(
According to 'key list', there is now a new KSK with id 330, but
ods-enforcerd wants me to submit the DS with keytag 11318, which
is a ZSK according to key list. And when I use key export -d to
get the DS to be submitted, it only gives me DS's for 11318 (and
some retired keys). How can I get the DS for keytag 330??
Right now, it looks to me that the only way to recover is to
turn off DNSSEC for all zones, and start from scratch :-(

Well, after waiting a day, a somewhat friendlier solution has presented
itself. After exoiry of a timer for the newly created KSK 330, the
ods-enforcer key export -d *did* actually give me the DS records for
KSK 330 (and some other useless ones). After uploading these DS records
to the registrar, the zone did come back to life, and is basically
looking healthy now.

Still, this is not a feasible method to repair my other zones, since
I don't want to see them die DNSSEC-wise, while waiting for the timer
to expire. Only after that (many hours later) ods-enforcer key export -d
will finally give me the desired DS records.

I am wondering what criteria are applied by the code to decide which keys
to export. It looks like they are wrong most of the time. Right now, with
the zone restored to working state, it is exporting the DS for a retired
ZSK. How could that ever be useful??

FYI, I have attached the output from key list -v and key list -d for this
zone. The only exported key is now 13284 (retired ZSK).
Even when I explicitly specify "key export -t KSK", it is still giving me
this retired ZSK 13284.

Regards,
-- wytze

[Yuri Schaeffer]

Hi Wytze,

I found a couple of errors in the migration script. Causing confusion in
the enforcer about the role of a key (ksk/zsk). I would advice you to
run the migration again but since you are live that might not be feasible.

If you are adventurous we could try to patch your database? Assuming you
are. Lets do this:

- stop opendnssec entirely
- backup your kasp.db
- run the following queries on your db:

UPDATE keyData
SET dsatparent = 0
WHERE role = 2;

UPDATE keyState
SET state = 4
WHERE (keyState.type = 0 OR keyState.type = 3) AND keyDataId IN (
SELECT keyData.id
FROM keyData
WHERE keyData.role = 2);

UPDATE keyState
SET state = 4
WHERE keyState.type = 1 AND keyDataId IN (
SELECT keyData.id
FROM keyData
WHERE keyData.role = 1);

This should get rid of those pesky ds-submit messages for ZSKs. And
prevent premature rollovers.

- start ODS back up
- Make sure the enforcer processed all zones. If needed run ods-enforcer
enforce; ods-enforcer signconf (we want to make sure it writes a new
signconf even if it thinks there is nothing to do);

Post by Wytze van der Raay
Well, after waiting a day, a somewhat friendlier solution has presented
itself. After exoiry of a timer for the newly created KSK 330, the
ods-enforcer key export -d *did* actually give me the DS records for
KSK 330 (and some other useless ones). After uploading these DS records
to the registrar, the zone did come back to life, and is basically
looking healthy now.
Still, this is not a feasible method to repair my other zones, since
I don't want to see them die DNSSEC-wise, while waiting for the timer
to expire. Only after that (many hours later) ods-enforcer key export -d
will finally give me the desired DS records.

I expect after fixing the DB this will give you correct results.

Post by Wytze van der Raay
I am wondering what criteria are applied by the code to decide which keys
to export. It looks like they are wrong most of the time. Right now, with
the zone restored to working state, it is exporting the DS for a retired
ZSK. How could that ever be useful??

It isn't. Botched DB.

//Yuri

Post by Wytze van der Raay
FYI, I have attached the output from key list -v and key list -d for this
zone. The only exported key is now 13284 (retired ZSK).
Even when I explicitly specify "key export -t KSK", it is still giving me
this retired ZSK 13284.
Regards,
-- wytze

[Wytze van der Raay]

Hi Yuri,

Post by Yuri Schaeffer
I found a couple of errors in the migration script. Causing confusion in
the enforcer about the role of a key (ksk/zsk). I would advice you to
run the migration again but since you are live that might not be feasible.

Not really ... too many things have been changed already since.

Post by Yuri Schaeffer
If you are adventurous we could try to patch your database? Assuming you

OK, a little bit, so I tried this.

Post by Yuri Schaeffer
- stop opendnssec entirely
- backup your kasp.db
UPDATE keyData
SET dsatparent = 0
WHERE role = 2;
UPDATE keyState
SET state = 4
WHERE (keyState.type = 0 OR keyState.type = 3) AND keyDataId IN (
SELECT keyData.id
FROM keyData
WHERE keyData.role = 2);
UPDATE keyState
SET state = 4
WHERE keyState.type = 1 AND keyDataId IN (
SELECT keyData.id
FROM keyData
WHERE keyData.role = 1);
This should get rid of those pesky ds-submit messages for ZSKs. And
prevent premature rollovers.

Yes, the key states look a lot more reasonable now indeed. But it has
also managed to break my cacert.net zone again, just for a while I hope.
That zone is now signed by a retired ZSK, which is not in the zone file
anymore, while the formerly active ZSK is now in the 'ready' state,
which will hopefully change to 'active' at 2016-07-19 00:06:09.

Post by Yuri Schaeffer
- start ODS back up
- Make sure the enforcer processed all zones. If needed run ods-enforcer
enforce; ods-enforcer signconf (we want to make sure it writes a new
signconf even if it thinks there is nothing to do);

(I did all of that)

Post by Yuri Schaeffer

Post by Wytze van der Raay
Well, after waiting a day, a somewhat friendlier solution has presented
itself. After exoiry of a timer for the newly created KSK 330, the
ods-enforcer key export -d *did* actually give me the DS records for
KSK 330 (and some other useless ones). After uploading these DS records
to the registrar, the zone did come back to life, and is basically
looking healthy now.
Still, this is not a feasible method to repair my other zones, since
I don't want to see them die DNSSEC-wise, while waiting for the timer
to expire. Only after that (many hours later) ods-enforcer key export -d
will finally give me the desired DS records.

I expect after fixing the DB this will give you correct results.

Not really, for the cacert.net zone *nothing* is exported (might be
considered reasonable since the current KSK has already been uploaded),
but for the cacert.com zone ODS continues to export useless retired
records (all KSK now, that's a minor improvement I guess). Which means
I still have to wait until tomorrow morning before I can export the DS
of the new KSK which is still in 'publish' state ... but cannot be
published lacking the DS :-(

Regards,
-- wytze