[Opendnssec-user] ods-signerd: adapter failed (Unable to open file)

Discussion:

Randy Bush

2018-05-30 17:20:38 UTC

freebsd 11.1-RELEASE-p10
opendnssec version 1.4.14

ods-signerd: [tools] unable to write zone lr: adapter failed (Unable to open file)
ods-signerd: [worker[1]] CRITICAL: failed to sign zone lr: Unable to open file

how do i find out which file/dir is the issue? is there a nice perms
checker i can run? can i have a pony? :)

thanks!

randy

Jake Zack

2018-05-30 17:47:19 UTC

Permalink

Hey Randy,

Never done it under FreeBSD, but my guesses would be the files/directories mentioned in:

# grep File /etc/opendnssec/zonelist.xml
<Adapter type="File">/var/opendnssec/unsigned/ca</Adapter>
<Adapter type="File">/var/opendnssec/signed/ca</Adapter>

# grep -i working conf.xml
<WorkingDirectory>/var/opendnssec/tmp</WorkingDirectory>

...as I don't know that ods-signerd writes much else (the rest done by enforcer or such).

-jake

-----Original Message-----
From: Opendnssec-user <opendnssec-user-***@lists.opendnssec.org> On Behalf Of Randy Bush
Sent: May-30-18 1:21 PM
To: opendnssec-user <opendnssec-***@lists.opendnssec.org>
Subject: [Opendnssec-user] ods-signerd: adapter failed (Unable to open file)

freebsd 11.1-RELEASE-p10
opendnssec version 1.4.14

ods-signerd: [tools] unable to write zone lr: adapter failed (Unable to open file)
ods-signerd: [worker[1]] CRITICAL: failed to sign zone lr: Unable to open file

how do i find out which file/dir is the issue? is there a nice perms checker i can run? can i have a pony? :)

thanks!

randy

Randy Bush

2018-05-30 18:03:31 UTC

Permalink

jake,

Post by Jake Zack
Never done it under FreeBSD, but my guesses would be the
# grep File /etc/opendnssec/zonelist.xml
<Adapter type="File">/var/opendnssec/unsigned/ca</Adapter>
<Adapter type="File">/var/opendnssec/signed/ca</Adapter>

# ls -l /usr/local/var/opendnssec/unsigned/2001.0418.3807
-rw-r--r-- 1 opendnssec staff 2581 Sep 17 2017 /usr/local/var/opendnssec/unsigned/2001.0418.3807
# ls -ld /usr/local/var/opendnssec/unsigned
drwxrwxr-x 2 opendnssec opendnssec 1024 May 30 15:33 /usr/local/var/opendnssec/unsigned/

# ls -ld /usr/home/dns/primary
drwxrwsr-x 3 bind bind 2048 May 27 15:10 /usr/home/dns/primary/
# ls -l /usr/home/dns/primary/2001.0418.8006
-rw-r--r-- 1 opendnssec bind 82491 May 27 15:10 /usr/home/dns/primary/2001.0418.8006

Post by Jake Zack
# grep -i working conf.xml
<WorkingDirectory>/var/opendnssec/tmp</WorkingDirectory>

Randy Bush

2018-06-01 03:04:00 UTC

Permalink

is it looking for system (math etc) libraries in /usr/local?

82235 ods-signerd NAMI "/usr/local/lib/libz.so.6"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL openat(AT_FDCWD,0x8006732a9,0x100000<O_RDONLY|O_CLOEXEC>)
82235 ods-signerd NAMI "/var/run/ld-elf.so.hints"
82235 ods-signerd RET openat 3
82235 ods-signerd CALL read(0x3,0x80087ab68,0x80)
82235 ods-signerd GIO fd 3 read 128 bytes
0x0000 4568 6e74 0100 0000 8000 0000 7900 0000 |Ehnt........y...|
0x0010 0000 0000 7800 0000 0000 0000 0000 0000 |....x...........|
0x0020 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0030 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0040 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0050 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0060 0000 0000 0000 0000 0000 0000 0000 0000 |................|
0x0070 0000 0000 0000 0000 0000 0000 0000 0000 |................|
82235 ods-signerd RET read 128/0x80
82235 ods-signerd CALL fstat(0x3,0x7fffffffd5d0)
82235 ods-signerd STRU struct stat {dev=152, ino=1605131, mode=0100444, nlink=1, uid=0, gid=0, rdev=3217083, atime=1527757792.707291000, mtime=1527694092.438438000, ctime=1527694092.438478000, birthtime=1527694092.438367000, size=249, blksize=32768, blocks=8, flags=0x0 }
82235 ods-signerd RET fstat 0
82235 ods-signerd CALL lseek(0x3,0x80,SEEK_SET)
82235 ods-signerd RET lseek 128/0x80
82235 ods-signerd CALL read(0x3,0x80067c300,0x79)
82235 ods-signerd GIO fd 3 read 121 bytes
"/lib:/usr/lib:/usr/local/lib:/usr/lib/compat:/usr/local/lib/mysql:/usr\
/local/lib/perl5/5.26/mach/CORE:/usr/local/lib/pth\0"
82235 ods-signerd RET read 121/0x79
82235 ods-signerd CALL close(0x3)
82235 ods-signerd RET close 0
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)
82235 ods-signerd NAMI "/lib/libz.so.6"
82235 ods-signerd RET access 0
82235 ods-signerd CALL openat(AT_FDCWD,0x80067b240,0x300000<O_RDONLY|O_CLOEXEC|O_VERIFY>)
82235 ods-signerd NAMI "/lib/libz.so.6"
82235 ods-signerd RET openat 3
82235 ods-signerd CALL fstat(0x3,0x7fffffffd908)
82235 ods-signerd STRU struct stat {dev=150, ino=1284251, mode=0100444, nlink=1, uid=0, gid=0, rdev=2582560, atime=1527757765.277929000, mtime=1527431524.821526000, ctime=1527431524.821567000, birthtime=1527431524.821369000, size=102024, blksize=32768, blocks=200, flags=0x0 }
82235 ods-signerd RET fstat 0
82235 ods-signerd CALL mmap(0,0x1000,0x1<PROT_READ>,0x40002<MAP_PRIVATE|MAP_PREFAULT_READ>,0x3,0)
82235 ods-signerd RET mmap 34366562304/0x800682000
82235 ods-signerd CALL mmap(0,0x219000,0<PROT_NONE>,0x21002<MAP_PRIVATE|MAP_ANON|MAP_NOCORE>,0xffffffff,0)
82235 ods-signerd RET mmap 34374868992/0x800e6e000
82235 ods-signerd CALL mmap(0x800e6e000,0x18000,0x5<PROT_READ|PROT_EXEC>,0x60012<MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ>,0x3,0)
82235 ods-signerd RET mmap 34374868992/0x800e6e000
82235 ods-signerd CALL mmap(0x801086000,0x1000,0x3<PROT_READ|PROT_WRITE>,0x40012<MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ>,0x3,0x18000)
82235 ods-signerd RET mmap 34377064448/0x801086000
82235 ods-signerd CALL munmap(0x800682000,0x1000)
82235 ods-signerd RET munmap 0
82235 ods-signerd CALL close(0x3)
82235 ods-signerd RET close 0
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)
82235 ods-signerd NAMI "/usr/local/lib/liblzma.so.5"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)
82235 ods-signerd NAMI "/lib/liblzma.so.5"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)
82235 ods-signerd NAMI "/usr/lib/liblzma.so.5"
82235 ods-signerd RET access 0
82235 ods-signerd CALL openat(AT_FDCWD,0x80067b280,0x300000<O_RDONLY|O_CLOEXEC|O_VERIFY>)
82235 ods-signerd NAMI "/usr/lib/liblzma.so.5"
82235 ods-signerd RET openat 3
82235 ods-signerd CALL fstat(0x3,0x7fffffffd908)
82235 ods-signerd STRU struct stat {dev=155, ino=177132688, mode=0100444, nlink=1, uid=0, gid=0, rdev=353892904, atime=1527757700.733788000, mtime=1527431525.449343000, ctime=1527431525.449409000, birthtime=1527431525.449081000, size=168728, blksize=32768, blocks=336, flags=0x0 }
82235 ods-signerd RET fstat 0
82235 ods-signerd CALL mmap(0,0x1000,0x1<PROT_READ>,0x40002<MAP_PRIVATE|MAP_PREFAULT_READ>,0x3,0)
82235 ods-signerd RET mmap 34366562304/0x800682000
82235 ods-signerd CALL mmap(0,0x229000,0<PROT_NONE>,0x21002<MAP_PRIVATE|MAP_ANON|MAP_NOCORE>,0xffffffff,0)
82235 ods-signerd RET mmap 34377068544/0x801087000
82235 ods-signerd CALL mmap(0x801087000,0x28000,0x5<PROT_READ|PROT_EXEC>,0x60012<MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ>,0x3,0)
82235 ods-signerd RET mmap 34377068544/0x801087000
82235 ods-signerd CALL mmap(0x8012af000,0x1000,0x3<PROT_READ|PROT_WRITE>,0x40012<MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ>,0x3,0x28000)
82235 ods-signerd RET mmap 34379329536/0x8012af000
82235 ods-signerd CALL munmap(0x800682000,0x1000)
82235 ods-signerd RET munmap 0
82235 ods-signerd CALL close(0x3)
82235 ods-signerd RET close 0
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)
82235 ods-signerd NAMI "/usr/local/lib/libm.so.5"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)

Berry A.W. van Halderen

2018-06-01 08:11:42 UTC

Permalink

Post by Randy Bush
is it looking for system (math etc) libraries in /usr/local?
82235 ods-signerd NAMI "/usr/local/lib/libz.so.6"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL openat(AT_FDCWD,0x8006732a9,0x100000<O_RDONLY|O_CLOEXEC>)
82235 ods-signerd NAMI "/var/run/ld-elf.so.hints"
82235 ods-signerd RET openat 3

....

Post by Randy Bush
82235 ods-signerd GIO fd 3 read 121 bytes
"/lib:/usr/lib:/usr/local/lib:/usr/lib/compat:/usr/local/lib/mysql:/usr\
/local/lib/perl5/5.26/mach/CORE:/usr/local/lib/pth\0"

....

Post by Randy Bush
82235 ods-signerd NAMI "/usr/local/lib/libm.so.5"
82235 ods-signerd RET access -1 errno 2 No such file or directory
82235 ods-signerd CALL access(0x80067e800,0<F_OK>)

Apart from the PKCS#11 library, explicitly mentioned in the conf.xml,
the signer does not load the shared libraries themselves. I think
this is just the operating system (or rather ld.so) hunting for
the shared libraries like libm. It is normal that it will get a lot
of No such file or directory in there, before it finds the right one.
If any of them would fail, any wouldn't even try to start. You
can use the program ldd to which which shared library is actually
used (if not dynamically loaded).

\Berry

Berry A.W. van Halderen

2018-06-01 08:14:21 UTC

Permalink

Dear Randy,

Post by Randy Bush

Post by Jake Zack
# grep File /etc/opendnssec/zonelist.xml
<Adapter type="File">/var/opendnssec/unsigned/ca</Adapter>
<Adapter type="File">/var/opendnssec/signed/ca</Adapter>

These are only used for reading, so these wouldn't get the earlier
problem.

Post by Randy Bush
# ls -ld /usr/home/dns/primary
drwxrwsr-x 3 bind bind 2048 May 27 15:10 /usr/home/dns/primary/
# ls -l /usr/home/dns/primary/2001.0418.8006
-rw-r--r-- 1 opendnssec bind 82491 May 27 15:10 /usr/home/dns/primary/2001.0418.8006

I guess you mention this because it is either specified in the <Adaptor>
part that indicated where the output file should go. If this is indeed
the case, this will probably be the issue. Even through
/usr/home/dns/primary/2001.0418.8006 is writable by the opendnssec user
(I guess you are running the signer as the opendnssec user id),
the directory in which it is contained is not.

The /usr/home/dns/primary needs to be writable as well. The signer need
to write a new file (with .tmp appended) and then move this new
file over the old file. Hence it will need write permissions on the
directory. This procedure is necessary to make the action atomic as
the consumer of the file (bind or NSD) might decide to read file file
mid-way and that would be wrong.

If the above assumptions are not correct it might be related to
either the user id that the signer is running as or some other
setting in your conf.xml, for which I would then need more info.
But I suspect the above explanation.

\Berry

Post by Randy Bush

Post by Jake Zack
# grep -i working conf.xml
<WorkingDirectory>/var/opendnssec/tmp</WorkingDirectory>

# grep -i working /usr/local/etc/opendnssec/conf.xml
<WorkingDirectory>/usr/local/var/opendnssec/tmp</WorkingDirectory>
<WorkingDirectory>/usr/local/var/opendnssec/tmp</WorkingDirectory>
# ls -ld /usr/local/var/opendnssec/tmp
drwxr-xr-x 3 opendnssec opendnssec 2560 May 30 15:54 /usr/local/var/opendnssec/tmp/
randy
_______________________________________________
Opendnssec-user mailing list
https://lists.opendnssec.org/mailman/listinfo/opendnssec-user

Randy Bush

2018-06-01 10:32:44 UTC

Permalink

Post by Berry A.W. van Halderen

<doh>

i saw and let it pass as it had not changed. but something clearly did.

and staring at the ktrace was not going anywhere for some reason
# kdump -f ktrace.out | grep primary
#

thank you!

randy

Randy Bush

2018-06-01 10:50:28 UTC

Permalink

Post by Randy Bush

Post by Berry A.W. van Halderen
The /usr/home/dns/primary needs to be writable as well.

i saw and let it pass as it had not changed. but something clearly did.

found. what changed was, in the move from freebsd 10.3 to 11.1,
the /etc/group uid entry for bind lost the opendnssec uid.

thanks again.

randy