[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API timeout for GET /api/v1/servers/localhost/zones in PDNS 4.9 - It even crashes OS #14720

Open
julgonwimg opened this issue Sep 27, 2024 · 5 comments
Labels

Comments

@julgonwimg
Copy link
julgonwimg commented Sep 27, 2024

We migrated from PDNS (Authorative) v4.1.X to v4.9.

We noticed that the PDNS API is not working properly with /api/v1/servers/localhost/zones
it just times out.

Running the same curl command in the old version (4.1) works as expected taking under 1 minute.
We ran other api calls just to be sure is not a connectivity issue and they worked as expected.

This makes PDA unusable, as it relays on GET localhost/zones, to sync its UI...

So to summarize:

  • On PDNS v4.1
    • curl -H 'X-API-Key: <apikey>' http://127.0.0.1:8081/api/v1/servers/localhost/zones/domain1.com --> OK
    • curl -H 'X-API-Key: <apikey>' http://127.0.0.1:8081/api/v1/servers/localhost/zones --> OK (under 1 minute)
  • On PDNS v4.9
    • curl -H 'X-API-Key: <apikey>' http://127.0.0.1:8081/api/v1/servers/localhost/zones/domain1.com --> OK
    • curl -H 'X-API-Key: <apikey>' http://127.0.0.1:8081/api/v1/servers/localhost/zones --> ERROR (Times out after 10 minutes, also sometimes crashes the whole VM)

We have around 300k domains managed by this PDNS server.

Conducted several more tests:
Test 1:
On the Rocky Linux 9 -->

  • Downgraded from 4.9 down to 4.7 within rocky linux OS.
  • Pointed to the migrated DB and a snapshot before the schema change.
    It didn't work in any case

Test 2:
On the CentOS 7 -->

  • Installed PDNS 4.9. Pointed to the migrated DB and a snapshot before the schema change. It didn't work in any case- - Downgraded PDNS from 4.9 down to 4.2. None minor version worked
  • Installed PDNS 4.1.14 --> with this one worked (both pointing to the old/non-schema-migrated and new database)

Environment

  • Installed through yum (https://repo.powerdns.com/repo-files/centos-auth-49.repo)
  • OS: PDNS4.9 in Rocky Linux 9, PDNS 4.1 in Centos 7
  • Backend used: Mysql 8 (RDS aurora.mysql specifically). We ran the old version where the /zones call works connecting to same DB after the DB schema was upgraded and it worked too
  • HW Specs: 2vCPU, 4gb RAM (aws - t3.medium) | 2gb SWAP, same specs for both pdns servers (4.9 and 4.1)

Issue Type and Service

  • Program: Authoritative
  • Issue type: Bug report

Expected behaviour

for the PDNS api to call to http://127.0.0.1:8081/api/v1/servers/localhost/zones to respond under 5 minutes

Actual behaviour

The request timesout (sometimes even crashes the VM)

Other information

This worked perfectly on version 4.1

@zeha
Copy link
Collaborator
zeha commented Sep 27, 2024

Some initial questions:

  • How much memory do you have for pdns_server?
  • Which backend is in use?
  • Which OS?

@zeha zeha added the auth label Sep 27, 2024
@julgonwimg
Copy link
Author

Thanks for the followup, sorry for the missing info.
Updating the post with that context

@rgacogne
Copy link
Member

ERROR (Times out after 10 minutes, also sometimes crashes the whole VM)

Would you by any chance be able to check how much memory is used during that test? Do you know if the operating system reports an out-of-memory condition? Knowing how much the same operation uses with version 4.1 would be very useful as well.

@julgonwimg
Copy link
Author
julgonwimg commented Sep 27, 2024

We upscale the instance to a xlarge (4vcpu 16gb ram) assuming it was related to this.
But the same behaviour showed up.
Also I posted "10Minutes timeout" before, but is because we kill it before it can actually finishes.

We can check the OS yes. But not right away sadly

@julgonwimg
Copy link
Author
julgonwimg commented Sep 27, 2024

Found a solid lead
running in 4.9 with dnssec=false it works.
curl -H 'X-API-Key: <key>' --max-time 120 'http://127.0.0.1:8081/api/v1/servers/localhost/zones?dnssec=false'

This change, cascades into other issues within PDA so is not a fix for us.

Could this help in pinpointing the cause of it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants