[Opendnssec-user] Not enough keys to satisfy zsk policy for zone

Discussion:

Marc Richter

2017-12-19 11:16:06 UTC

Hi,

we are getting the following errors in our logs (zonename replaced with
<zone>):

ods-enforcerd: [ID 992331 local0.warning] Not enough keys to satisfy zsk
policy for zone: <zone>. keys_to_allocate(1) = keys_needed(2) -
(keys_available(2) - keys_pending_retirement(1))

ods-enforcerd: [ID 115111 local0.warning] Tried to allocate 1 keys, failed
on allocating key number 1

ods-enforcerd: [ID 482275 local0.warning] ods-enforcerd will create some
more keys on its next run

ods-enforcerd: [ID 363081 local0.error] Error allocating zsks to zone <zone>

According to

https://wiki.opendnssec.org/display/DOCS/Troubleshooting

as well as the error message, ods-enforcerd should create new keys on its
next run. However, that doesn't seem to happen as the messages are
repeating every time ods-enforcerd is running.

ManualKeyGeneration is not set.

This is opendnssec version 1.4.10

How do I fix this ?

Regards
Marc

Hoda Rohani

2017-12-19 12:11:12 UTC

Permalink

Hello Marc,

I would recommend to upgrade your opendnssec.
We saw similar bugs before and fixed them in 1.4.14. There was a miscalculation in getting the right number of required
keys.

Please let us know if you still see those messages after upgrading.

Regards,
Hoda

Marc Richter

2017-12-19 12:15:40 UTC

Permalink

Hi Hoda,

is there a way to fix that even with the current version ?
That would allow a proper upgrade planning instead of doing this now in a rush.

Regards
Marc

Post by Hoda Rohani
Hello Marc,
I would recommend to upgrade your opendnssec.
We saw similar bugs before and fixed them in 1.4.14. There was a miscalculation in getting the right number of required
keys.
Please let us know if you still see those messages after upgrading.
Regards,
Hoda

Post by Marc Richter
Hi,
we are getting the following errors in our logs (zonename replaced with
ods-enforcerd: [ID 992331 local0.warning] Not enough keys to satisfy zsk
policy for zone: <zone>. keys_to_allocate(1) = keys_needed(2) -
(keys_available(2) - keys_pending_retirement(1))
ods-enforcerd: [ID 115111 local0.warning] Tried to allocate 1 keys, failed
on allocating key number 1
ods-enforcerd: [ID 482275 local0.warning] ods-enforcerd will create some
more keys on its next run
ods-enforcerd: [ID 363081 local0.error] Error allocating zsks to zone <zone>
According to
https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opendnssec.org_display_DOCS_Troubleshooting&d=DwIC-g&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=wDgZv-d1RrBMzWr_7pSF_09ZAXIr59EgoXQU4ctOHMk&m=mKI6YLd07oL68W0Uhj30N_PrFQT1h-999YDxiqHNv2M&s=DK0eg6GmdauHR_8RwJZtzemEEgDtM2u6rMEEfsd9uyI&e=
as well as the error message, ods-enforcerd should create new keys on its
next run. However, that doesn't seem to happen as the messages are
repeating every time ods-enforcerd is running.
ManualKeyGeneration is not set.
This is opendnssec version 1.4.10
How do I fix this ?
Regards
Marc
_______________________________________________
Opendnssec-user mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.opendnssec.org_mailman_listinfo_opendnssec-2Duser&d=DwIC-g&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=wDgZv-d1RrBMzWr_7pSF_09ZAXIr59EgoXQU4ctOHMk&m=mKI6YLd07oL68W0Uhj30N_PrFQT1h-999YDxiqHNv2M&s=YBLPkwcnP77lIvJQpsyVEXt9X3llX1ohP3PQBr8aJ-c&e=

--
Marc Richter
Engr IV Cslt-Ntwk Eng&Ops | Server & Services Management International
Global Operations | Verizon Wireline Network

Sebrathweg 20
44149 Dortmund - Germany

O +49 231 972 1293
F +49 231 972 2587
E ***@de.verizon.com

Hoda Rohani

2017-12-19 13:35:54 UTC

Permalink

Hello Marc,

This fix is in our software and there is no other way to solve it without migrating to 1.4.14.

1.4.14 contains some fixes for some minor issues as well. Migrating from 1.4.10 to 1.4.14 is very trivial. There is no
change in database.

Regards,
Hoda

Post by Marc Richter
Hi Hoda,
is there a way to fix that even with the current version ?
That would allow a proper upgrade planning instead of doing this now in a rush.
Regards
Marc

Yuri Schaeffer

2017-12-20 09:10:20 UTC

Permalink

Post by Marc Richter
is there a way to fix that even with the current version ?

What Hoda said, the upgrade is the fix.

However a workaround might be possible. If I remember correctly the
issue was that the enforcer during key generation would calculate the
wrong number of ZSKs. It only happens in the case where your KSK and ZSK
have the same key length. It would add the number of KSKs to the number
of ZSKs and concluded it has enough ZSKs and doesn't need to generate more.

A short term workaround:
use "ods-ksmutil key generate --period PERIOD" to generate more keys.
For PERIOD choose something bigger than the value from the conf. Say
twice. Make sure the lifetime of the ZSK is shorter than the KSK or
you'll probably hit the same problem.

Long term workaround:
Use a different key length for ZSK than KSK.

None of this is tested.
//Yuri

Marc Richter

2017-12-20 12:42:50 UTC

Permalink

Hi Yuri, Hi Hoda,

Post by Yuri Schaeffer

Post by Marc Richter
is there a way to fix that even with the current version ?

What Hoda said, the upgrade is the fix.

I have restored a backup of the database, SoftHSM and the signconf files
onto a development server that runs 1.4.10 as well.

I saw the same error messages when starting ODS on that development server,
so I could reproduce the issue.

I then shutdown ODS, upgraded to 1.4.14 and restarted ODS, but the error is
still reported.

So the upgrade did not fix the issue, apparently.
Do you have any advice what do check next ?

Post by Yuri Schaeffer
use "ods-ksmutil key generate --period PERIOD" to generate more keys.
For PERIOD choose something bigger than the value from the conf. Say
twice. Make sure the lifetime of the ZSK is shorter than the KSK or
you'll probably hit the same problem.

I guess you mean "key generate --interval" instead of "key generate --period" ?

A --period switch does not seem to exist.

Post by Yuri Schaeffer
Use a different key length for ZSK than KSK.

We already do. KSK length is 2048, ZSK 1024.

Regards
Marc

Yuri Schaeffer

2017-12-20 13:57:46 UTC

Permalink

Post by Marc Richter
I guess you mean "key generate --interval" instead of "key generate --period" ?

indeed.

Post by Marc Richter

Post by Yuri Schaeffer
Use a different key length for ZSK than KSK.

We already do. KSK length is 2048, ZSK 1024.

Then you have a different problem. Please check which user OpenDNSSEC
runs as and make sure that your HSM allows that user write access.

//Yuri

Marc Richter

2017-12-20 14:21:25 UTC

Permalink

Hi,

Post by Yuri Schaeffer
Then you have a different problem. Please check which user OpenDNSSEC
runs as and make sure that your HSM allows that user write access.

I checked that already. The user that runs ODS does have read/write access
to the SoftHSM storage directory (actually the token directory is owned by
the user that runs ODS).
I could successfully create and delete files with that userid in the HSM
storage directory.

Using that userid I can also run ods-hsmutil to list keys etc.

Regards
Marc

Yuri Schaeffer

2017-12-21 11:16:36 UTC

Permalink

Post by Marc Richter
ods-enforcerd: [ID 992331 local0.warning] Not enough keys to satisfy zsk
policy for zone: <zone>. keys_to_allocate(1) = keys_needed(2) -
(keys_available(2) - keys_pending_retirement(1))
ods-enforcerd: [ID 115111 local0.warning] Tried to allocate 1 keys, failed
on allocating key number 1
ods-enforcerd: [ID 482275 local0.warning] ods-enforcerd will create some
more keys on its next run
ods-enforcerd: [ID 363081 local0.error] Error allocating zsks to zone <zone>

These warning are emitted when the enforcer tries to reserve a key for a
zone but the key isn't available. It is normal to see these warnings
sometimes.

The actual generation of the key fails but that happens at slightly
different time. Could you provide more log output?

//Yuri

Marc Richter

2017-12-21 11:41:11 UTC

Permalink

Hi Yuri,

Post by Yuri Schaeffer
The actual generation of the key fails but that happens at slightly
different time. Could you provide more log output?

how would a log message look like when new keys are generated ?
I searched the log (already at verbosity 10) but did not find any messages
that would indicate that ODS is generating, or trying to generate but
failing, any new keys.

The only thing I found is in the startup messages, where it says that "No
new ZSKs need to be created".
See logs below (config filenames and DB information have been removed from
the log messages):

ods-enforcerd: [ID 676094 daemon.info] opendnssec starting...
ods-enforcerd: [ID 326049 local0.info] HSM connection open.
ods-enforcerd: [ID 442419 local0.info] Reading config
ods-enforcerd: [ID 321401 local0.info] Reading config schema
ods-enforcerd: [ID 779269 local0.info] Communication Interval: 900
ods-enforcerd: [ID 166010 local0.info] Rollover Notification Interval: 604800
ods-enforcerd: [ID 796646 local0.info] Using command: to submit DS records
ods-enforcerd: [ID 646761 local0.info] MySQL database schema set to:
ods-enforcerd: [ID 950666 local0.info] MySQL database user set to:
ods-enforcerd: [ID 130658 local0.info] MySQL database password set
ods-enforcerd: [ID 517519 local0.info] Log User set to: local0
ods-enforcerd: [ID 399845 local0.info] Pidfile set to:
ods-enforcerd: [ID 599916 local0.info] Switched log facility to: local0
ods-enforcerd: [ID 813082 local0.info] Connecting to Database...
ods-enforcerd: [ID 799338 local0.info] Policy default found.
ods-enforcerd: [ID 792314 local0.info] Key sharing is On
ods-enforcerd: [ID 931102 local0.info] 86 zone(s) found on policy "default"
ods-enforcerd: [ID 970822 local0.info] No new KSKs need to be created.
ods-enforcerd: [ID 193721 local0.info] No new ZSKs need to be created.
ods-enforcerd: [ID 630891 local0.info] NOTE: keys generated in repository
SoftHSM will not become active until they have been backed up
ods-enforcerd: [ID 685651 local0.debug] Purging keys...

Regards
Marc

Yuri Schaeffer

2017-12-21 12:06:57 UTC

Permalink

Hi Marc,

Post by Marc Richter
The only thing I found is in the startup messages, where it says that "No
new ZSKs need to be created".

This is a useful hint. We have two options. Either the number of 'still
good' keys from the database is counted wrong. Or the keys in the
database are in a strange state.

Perhaps a little bit of both. I'd like to dig in your database to see
which one. Can you send me a database dump? Your kasp.xml would also be
useful.

//Yuri

Yuri Schaeffer

2017-12-21 13:26:16 UTC

Permalink

Post by Marc Richter
ods-enforcerd: [ID 630891 local0.info] NOTE: keys generated in repository
SoftHSM will not become active until they have been backed up

We think you have <RequireBackup/> in your conf but did not indicate to
OpenDNSSEC that you actually backed them up. Therefore it isn't allowed
to use the keys.

So try backing up your keys or stop requiring it.

//Yuri

Marc Richter

2017-12-21 14:10:42 UTC

Permalink

Hi,

Post by Yuri Schaeffer

Post by Marc Richter
ods-enforcerd: [ID 630891 local0.info] NOTE: keys generated in repository
SoftHSM will not become active until they have been backed up

I don't think this is the issue. We are doing a key backup multiple times
per day using "ods-ksmutil backup prepare" as the first step and
"ods-ksmutil backup commit" as the last step of the process.

So a key that was freshly generated should become active shortly after that.

I also just did this manually and no keys were marked during prepare or commit:

# ods-ksmutil backup prepare
There were no keys to mark

# ods-ksmutil backup commit
There were no keys to mark

Regards
Marc

Yuri Schaeffer

2017-12-21 15:42:12 UTC

Permalink

Hi Marc,

I'm looking at your database and can't find anything obvious so far.
What value have you for AutomaticKeyGenerationPeriod in the conf.xml?

also, did you set a key count limit on your HSM in the conf?

//Yuri

Marc Richer

2017-12-21 17:20:26 UTC

Permalink

Hi,

Post by Yuri Schaeffer
What value have you for AutomaticKeyGenerationPeriod in the conf.xml?

none. This is 1.4 and if I read the documentation correctly that option
was introduced in 2.0.

Post by Yuri Schaeffer
also, did you set a key count limit on your HSM in the conf?

No, we have not set the Capacity option in the Repositiy section (which
I think is what you were referring to).

Regards
Marc