Tech and Work at Hopbox

from Sahil Dhiman

Managing authoritative and recursive name servers (i.e. DNS servers) at hopbox, has shown us a bunch of quirks and insights of domain name system.

Here's a list of them in no particular order:

trace full name recursion

+trace option with dig does full recursion showing each stage of name server delegation from root name server to TLD name server to domain authoritative name server:

# dig +trace hopbox.net
;; communications error to 9.9.9.9#53: timed out

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> +trace hopbox.net
;; global options: +cmd
.                       7256    IN      NS      f.root-servers.net.
.                       7256    IN      NS      g.root-servers.net.
.                       7256    IN      NS      h.root-servers.net.
.                       7256    IN      NS      i.root-servers.net.
.                       7256    IN      NS      j.root-servers.net.
.                       7256    IN      NS      k.root-servers.net.
.                       7256    IN      NS      l.root-servers.net.
.                       7256    IN      NS      m.root-servers.net.
.                       7256    IN      NS      a.root-servers.net.
.                       7256    IN      NS      b.root-servers.net.
.                       7256    IN      NS      c.root-servers.net.
.                       7256    IN      NS      d.root-servers.net.
.                       7256    IN      NS      e.root-servers.net.
.                       7256    IN      RRSIG   NS 8 0 518400 20250114170000 20250101160000 26470 . Dpc2m6PNj2gdfiO0RGvDAONDSPlPpfc+BYy947k/Nr4bPDiWTgNAxjAT bpU9nNUmnp91Catsz2hW6eSInVeboF3Ngu2YJSyOsp7J18ZWJ1HlRZmi d7kIY0/qqLzKv8FELtN9utLqr3JnT7asGe1AaJiqmsVaKaCjd6bEpE1v W/ouA91IFeSmFfPOBCBtSm7cRlbrwPQ5Nk+KZ+xbwJiJLg4hfHz+GGqZ bev18kbHG+TKL9lgD/RFc0QjMDA6sEvX2KEsU4YSnHOvDTK4/YxDKBs2 OZN+HWQKUaU5jlfAYUbiMlHH6fDhcqFSnv47kfirOZ/EegRsE4JdfQ2g xD0vVA==
;; Received 525 bytes from 9.9.9.9#53(9.9.9.9) in 288 ms

net.                    172800  IN      NS      e.gtld-servers.net.
net.                    172800  IN      NS      a.gtld-servers.net.
net.                    172800  IN      NS      i.gtld-servers.net.
net.                    172800  IN      NS      d.gtld-servers.net.
net.                    172800  IN      NS      j.gtld-servers.net.
net.                    172800  IN      NS      f.gtld-servers.net.
net.                    172800  IN      NS      b.gtld-servers.net.
net.                    172800  IN      NS      k.gtld-servers.net.
net.                    172800  IN      NS      c.gtld-servers.net.
net.                    172800  IN      NS      g.gtld-servers.net.
net.                    172800  IN      NS      h.gtld-servers.net.
net.                    172800  IN      NS      l.gtld-servers.net.
net.                    172800  IN      NS      m.gtld-servers.net.
net.                    86400   IN      DS      37331 13 2 2F0BEC2D6F79DFBD1D08FD21A3AF92D0E39A4B9EF1E3F4111FFF2824 90DA453B
net.                    86400   IN      RRSIG   DS 8 1 86400 20250115050000 20250102040000 26470 . N893DyeJ8VpUSOFqR8D5md0pYbjuu5osyYCSFU/rbJcD14e8nbLLgbP0 P23kQu5t75HknIBgujBPYCPu5gvoNJqal9k2TbbAfCaJYQL1Nyn7d+7X 7SW8rQnBZgNGgJ0OeBStF1MIcVmxmro581AmAt8vi8Zt/qhuR7j57djz 47dt2EYHcPWArM+MoiepAZF3sCmAOToNt/AXC7RphpPxKsw+csvnsGQB ln/Cs8exOT6ZzTFPCnJKA8MUIZAcUf7A3gmH2sbBrSZWD53S9ix3ALmz NFnve/Es+dv3dt6Gh5ezq1ENtlzicUKBkuAyAfCjQb3hCDR6rvBXttnK GIsBYg==
;; Received 1167 bytes from 2001:500:2f::f#53(f.root-servers.net) in 76 ms

hopbox.net.             172800  IN      NS      ns-146-a.gandi.net.
hopbox.net.             172800  IN      NS      ns-17-b.gandi.net.
hopbox.net.             172800  IN      NS      ns-241-c.gandi.net.
A1RT98BS5QGC9NFI51S9HCI47ULJG6JH.net. 900 IN NSEC3 1 1 0 - A1RTLNPGULOGN7B9A62SHJE1U3TTP8DR NS SOA RRSIG DNSKEY NSEC3PARAM
A1RT98BS5QGC9NFI51S9HCI47ULJG6JH.net. 900 IN RRSIG NSEC3 13 2 900 20250107030726 20241231015726 31059 net. GwLCxxuQ4CZwSQDd0s2kdMTqPFovriFLpLPpn68iueesfX5aRciPVNju xdQ/yDfj2hRAjG50vhQILBQQ2m/86g==
EGMA2VHP2FPMA6D3EIFRUKQ94J3FTVCB.net. 900 IN NSEC3 1 1 0 - EGMBDPOV3HTH4B18NND99572B5V8VU8H NS DS RRSIG
EGMA2VHP2FPMA6D3EIFRUKQ94J3FTVCB.net. 900 IN RRSIG NSEC3 13 2 900 20250106030031 20241230015031 31059 net. lO4ZFHTfS3MFScIk27f5rZm0rVX7BHspNek365M91J7hTCNr3wS62QQw 9yyT1OeXpOmDSwyCU+Jb+K+yIvvtpA==
;; Received 602 bytes from 192.31.80.30#53(d.gtld-servers.net) in 164 ms

hopbox.net.             10800   IN      A       101.53.132.190
;; Received 55 bytes from 213.167.230.18#53(ns-17-b.gandi.net) in 24 ms

Notice in ;; Received ... in each step of the process, that tells which server is responding for that step of the process and their response before that.

Let's go through each step of the response:

In first level:

;; Received 525 bytes from 9.9.9.9#53(9.9.9.9) in 288 ms

my recursive resolver i.e. 9.9.9.9 responded for list of all root servers.

In second level:

;; Received 1167 bytes from 2001:500:2f::f#53(f.root-servers.net) in 76 ms

f.root-server.net responded with all .net. TLD name servers.

In third level:

;; Received 602 bytes from 192.31.80.30#53(d.gtld-servers.net) in 164 ms

d.gtld-servers.net responded with list of name servers for hopbox.net

In fourth level:

;; Received 55 bytes from 213.167.230.18#53(ns-17-b.gandi.net) in 24 ms

Long CNAME chain insanity

ecs.office.com is a Microsoft Office domain which seem to be behind 4 level of CNAME(ing):

# dig ecs.office.com

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> ecs.office.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41779
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ecs.office.com.                        IN      A

;; ANSWER SECTION:
ecs.office.com.         143     IN      CNAME   ecs.office.trafficmanager.net.
ecs.office.trafficmanager.net. 19 IN    CNAME   s-0005-office.config.skype.com.
s-0005-office.config.skype.com. 6644 IN CNAME   ecs-office.s-0005.s-msedge.net.
ecs-office.s-0005.s-msedge.net. 214 IN  CNAME   s-0005.s-msedge.net.
s-0005.s-msedge.net.    214     IN      A       52.113.194.132

;; Query time: 284 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Thu Jan 02 11:16:02 IST 2025
;; MSG SIZE  rcvd: 198

Would add name resolution latency penalty, but they would have their reasons.

The same behavior is observed in IPv6/AAAA record as well.

Too many A records

android.googleapis.com has atleast 15 A records:

# dig android.googleapis.com A

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> android.googleapis.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40939
;; flags: qr rd ra; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;android.googleapis.com.                IN      A

;; ANSWER SECTION:
android.googleapis.com. 219     IN      A       142.250.183.10
android.googleapis.com. 219     IN      A       142.250.66.10
android.googleapis.com. 219     IN      A       142.250.183.138
android.googleapis.com. 219     IN      A       142.250.70.42
android.googleapis.com. 219     IN      A       142.250.70.74
android.googleapis.com. 219     IN      A       142.250.182.202
android.googleapis.com. 219     IN      A       142.251.42.106
android.googleapis.com. 219     IN      A       142.250.70.106
android.googleapis.com. 219     IN      A       142.250.199.170
android.googleapis.com. 219     IN      A       142.250.199.138
android.googleapis.com. 219     IN      A       142.251.42.74
android.googleapis.com. 219     IN      A       142.250.183.170
android.googleapis.com. 219     IN      A       142.250.77.74
android.googleapis.com. 219     IN      A       142.250.183.202
android.googleapis.com. 219     IN      A       142.250.71.106
android.googleapis.com. 219     IN      A       142.250.182.234

;; Query time: 324 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Thu Jan 02 11:49:42 IST 2025
;; MSG SIZE  rcvd: 307

For AAAA records, we observed only 4 records though:

# dig android.googleapis.com AAAA

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> android.googleapis.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23104
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;android.googleapis.com.                IN      AAAA

;; ANSWER SECTION:
android.googleapis.com. 203     IN      AAAA    2404:6800:4009:81e::200a
android.googleapis.com. 203     IN      AAAA    2404:6800:4009:81f::200a
android.googleapis.com. 203     IN      AAAA    2404:6800:4009:820::200a
android.googleapis.com. 203     IN      AAAA    2404:6800:4009:81d::200a

;; Query time: 284 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Thu Jan 02 11:50:38 IST 2025
;; MSG SIZE  rcvd: 163

Short TTL of 5 seconds

AWS S3 seems to have started using TTLs of as low as 5 seconds for some of the domains like

  • s3.ap-south-1.amazonaws.com
  • s3-r-w.ap-south-1.amazonaws.com and
  • s3-w.ap-south-1.amazonaws.com

The IPs seem to be rotating quite frequently as well:

# dig s3-r-w.ap-south-1.amazonaws.com 

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> s3-r-w.ap-south-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 931
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;s3-r-w.ap-south-1.amazonaws.com. IN    A

;; ANSWER SECTION:
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       52.219.158.194
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       3.5.213.197
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       3.5.213.172
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       52.219.62.71
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       3.5.211.148
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       16.12.36.18
s3-r-w.ap-south-1.amazonaws.com. 5 IN   A       52.219.160.198

;; Query time: 292 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Thu Jan 02 11:52:33 IST 2025
;; MSG SIZE  rcvd: 172

No AAAA/IPv6 records, though.

This behavior can be observed across AWS regions endpoints.

The aaaaaaaaaaaaaaaaa domains in prod

Following are valid subdomains used in production by hotstar.com, an OTT provider in India owned by Disney:

  • abh2p32aaaaaaaamaaaaaaaaaaaaa.live-ssai-cf-mum-ace.cdn.hotstar.com
  • aba5q2faaaaaaaamaaaaaaaaaaaaa.hses7-vod-cf-ace.cdn.hotstar.com
  • aba5q2faaaaaaaamaaaaaaaaaaaaa.hses7-vod-cf-ace.cdn.hotstar.com

Name server domain diversity

After he.net domain was put on hold causing this single domain issue to bring large part of their infrastructure down, having name server domain across diverse domains does help.

Amazon seems to have most domain diversity in their name servers.

For example, see S3 name servers:

# dig ns s3.amazonaws.com

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> ns s3.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4181
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;s3.amazonaws.com.              IN      NS

;; ANSWER SECTION:
s3.amazonaws.com.       12769   IN      NS      ns-1300.awsdns-34.org.
s3.amazonaws.com.       12769   IN      NS      ns-1579.awsdns-05.co.uk.
s3.amazonaws.com.       12769   IN      NS      ns-63.awsdns-07.com.
s3.amazonaws.com.       12769   IN      NS      ns-771.awsdns-32.net.

;; Query time: 284 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Thu Jan 02 13:42:57 IST 2025
;; MSG SIZE  rcvd: 181

Let's expand on this:

  • ccTLD diversity – .com, .org, .uk and .net is used (Side note – whole TLDs can go down too, see .bd example and .ru outage)
  • second and third level domain diversity

As we keep on stumbling on more observations in domain name system, we'll keep on posting more of DNS chronicles.

 
Read more...

from Sahil Dhiman

We at Hopbox have been Free Software users and contributors for more than a decade now. We were looking to contribute more to the community in a different from usual way. Downloads from GNU FTP mirrors were slow in India and there existed only 1 mirror (when we decided) for whole of India. As of 4th April 2024, we’re only GNU mirror in India.

GNU is an extensive collection of software packages like Bash, Emacs, Octave and various coreutils. So after a quick discussion, we started hosting a GNU software mirror on our locally hosted Chromebox. The mirror is available on both http://mirrors.hopbox.net/gnu/ and https://mirrors.hopbox.net/gnu/.

As most Free Software don’t have commercial backing and require heavy downloads, the concept of software download mirrors helps take the traffic load off of the primary server, leading to geographical redundancy, higher availability and faster download in general.

At Hopbox, we regularly build a in-house customized version of OpenWrt, a GNU/Linux distribution for routers and embedded devices, which in turn uses GNU coreutils for compilation. Having a local mirror gives us speeds faster than 1 Gbps which is many times faster than our internet speeds, as traffic gets served locally.

The mirror is running on a Dell Optiplex 3020 solely dedicated for mirroring purposes behind a Hopbox APU gateway. The mirror right now is running on base Debian 12 bookworm.

Presently the mirror is serving 30-50 GB of traffic daily since becoming the sole GNU FTP mirror in India with Emacs and Octave fighting for top downloaded package. We generally see traffic from India and neighboring countries.

 
Read more...

from Sahil Dhiman

Basic function of Domain Name System (DNS) is to translate domain names to IP and vice-versa. It’s the phonebook of the computer. Why you may ask, because human’s remember domain names easily, wikipedia.org is easier than 103.102.166.224 . The IP is then used by the computer to fetch required resource for us.

But, DNS also contains other information like Mail Exchanger (MX) records, Nameserver (NS) records, Text (TXT) records amongst other information.

DNS records

DNS systems serves various information through various records. List of major DNS record type: * A record – points to IPv4 address where application is hosted. Like wikipedia.org A record points to 103.102.166.224. Run nslookup domain-name to find it. * AAAA record – points to IPv6 address where application is hosted. wikipedia.org’s AAAA record points to 2620:0:862:ed1a::1. Run nslookup domain-name to find it. * MX – Mail exchange, points to domain’s email server. For wikipedia.org mail is handled by mx1001.wikimedia.org. Run host domain-name to find MX record. * CNAME – canonical name, points to another A record. * NS – points to name server for the domain and sub-domains. Like wikipedia.org’s NS records point to ns2.wikimedia.org. Run dig NS domain-name to find it. * PTR – serves IP to hostname ie domain name * SOA – Start of authority record, serves important information about domain  * TXT – text records for verification and other purposes. * CAA – certificate authority records. * SRV – service record. Used to define hostname and port number for specific services.

Note – All these are GNU/Linux specific commands.

DNS request flow

Each DNS request has to traverse the whole tree once for getting the DNS response. Subsequently, responses are cached at various levels to speed up the process and not to overload the chain of servers.

The flow of request if as follow – Application → Stub resolver (query cache, answer not found) → Recursive resolver (query cache, answer not found) → DNS Root servers → Recursive resolver (populate cache for TLD NS) → Authoritative nameserver TLD → Recursive resolver (populate cache for domain name server) → domain nameserver → Recursive resolver (populates cache for answer) → Stub resolver (populates cache for answer) → Application

DNS request flow

  1. Application to stub resolver – Where is wikipedia.org?
  2. Stub resolver to recursive resolver – Where is wikipedia.org?
  3. Recursive resolver to root DNS server – Where is wikipedia.org?
  4. Root DNS server to recursive resolver – Ask top level domain authoritative name server for .org TLD like b2.org.afilias-net.org. (bunch of different name servers for redundancy).
  5. Recursive resolver to TLD authoritive name server – Where is wikipedia.org?
  6. TLD authoritive name server to recursive resolver – Ask domain name server for wikipedia.org – ns1.wikimedia.org (bunch of different name servers for redundancy).
  7. Recursive resolver to domain name server – Where is wikipedia.org?
  8. Domain name server to recursive resolver – Here’s the A record for wikipedia.org – 103.102.166.224
  9. Recursive resolver to stub resolver – wikipedia.org can be found at 103.102.166.224
  10.  Stub resolver to application – wikipedia.org can be found at 103.102.166.224

Explainers for each applications

  •  Application – Any application which needs to resolve a domain name like browser (Mozilla Firefox), mail client (Thunderbird) etc.
  • Stub resolver – A local, system-wide cache like systemd-resolved. dnsmasq etc. They hold the first level of system cache for DNS.
  •  Recursive resolver – A dedicate program like BIND, knot, etc which does the actual domain tree traversal ie recursion. It can be running on local or remote system.
  •  DNS root server – 13 DNS root servers. Their IP address is pre-programmed in DNS revolvers to start DNS query from. Complete list can be found at https://www.iana.org/domains/root/servers .
  •  TLD authoritative name server – Each top level domain like .org has a authoritative name server set which holds name servers for each domain. Run dig NS org. to find them for .org TLD.
  •  Domain name server – For each domain, a name server is defined which responses for each query for

DNS Message

To find more details about DNS, we can use a utility called dig. Run dig wikipedia.org to do a DNS query. Response for which would look something like this:

; <<>> DiG 9.18.12-1-Debian <<>> wikipedia.org 
;; global options: +cmd 
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23667 
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232 
; COOKIE: d8a190b00c8d01c201000000646f2e418b9d49b5081970dc (good) 
;; QUESTION SECTION: 
;wikipedia.org.                 IN      A 

;; ANSWER SECTION: 
wikipedia.org.          600     IN      A       103.102.166.224 

;; Query time: 523 msec 
;; SERVER: 10.23.0.30#53(10.23.0.30) (UDP) 
;; WHEN: Thu May 25 15:15:37 IST 2023 
;; MSG SIZE  rcvd: 86

There are 5 sections to the query – *  Header – shares status of query/resposne, flags and other info. *  Question section – what is being asked/queried – in our case A record for wikipedia.org *  Answer section – response received – in our case we received A record response of type IN (Internet) for wikipedia.org with a time to live (TTL) with IP 103.102.166.224 *  Authority section – SOA response, only avail in authoritative name server response like root-servers.net. Also see authority response in header section as 0. *  Additional info section – resolver info, time, size etc.

DNSSEC

A public-private key method to digitally sign the zone being served. Each domain name server signs the zone they’re serving with their private key. The public key is also shared with the zone.

The trust of public key comes from signature of public key by parent zone. For wikipedia.org, their public key would be signed by .org TLD authoritative name server’s private key, who’s public key would in turn be signed by DNS root servers private key. All recursive resolvers have already pre-defined to trust root’s public key’ making the trust anchor.

Read more

 
Read more...

from Sahil Dhiman

Domain Name System (DNS) can be an effective tool to block access to certain domains in an enterprise setting. This can also help prevent intended or unintended access to known malicious domains.

At Hopbox, we wanted to incorporate customer specific DNS based blocking while also protecting against ransomware, malware, and other newly discovered threats for 900+ locations across customer networks.

Getting the response actually takes multiple back and forth in DNS server hierarchy. A DNS recursive resolver traverses the DNS hierarchy starting from root DNS servers to TLD authoritative servers to domain authoritative servers to get DNS response.

We settled on BIND 9 as our DNS recursive resolver, as it's tried and tested and developed by ISC and used by many of the Internet root DNS servers.

Response Policy Zones

Initially, we tried serving a redirect but that essentially isn't possible without breaking HTTPs and end client trusting our custom TLS certificates, so we choose to serve NXDOMAIN to queries. For routing/serving NXDOMAIN on intended queries, Response Policy Zones (RPZ) were used to collate block lists and allow lists for BIND 9. Most block list providers nowadays provide block lists in RPZ format, which we directly feed into BIND.

Sample RPZ file:

$TTL 3600
@       IN      SOA     localhost.      root.localhost. (
                        1               ; serial
                        1h              ; refresh
                        1h              ; retry
                        1w              ; expiry
                        1h )            ; minimum
 
        IN      NS      localhost.
 
example.org    CNAME    IN    rpz-passthru.

Views

We also wanted customer specific allow and block lists. In DNS terms, we wanted to serve a custom zone for each customer. Concept of Views was according to our requirement. View essentially allowed us to serve custom zones on matching clients source IP (customer networks). We created one View for each customer with their specific allow and block lists.

Secondary DNS setup

For redundancy, we setup-ed multiple secondary server which sync zone files. The primary server sends notification and zone transfer whenever its zones are updated. The Secondary DNS server automatically reloads the updated zone on receiving a full zone transfer.

DNS Architecture

On primary, the following was added in BIND 9 config:

allow-trasfer { SECONDARY_SERVER_IP;  };
also-notify { SECONDARY_SERVER_IP; };

For each zone definition, the following was added:

master yes;
notify yes;
allow-transfer { SECONDARY_SERVER_IP; }

On secondary, following was added in BIND 9 config, for each zone definition, following was added:

type slave;
masters { PRIMARY_SERVER_IP; };

Deployment

As DNS server is added in dnsmasq in our OpenWrt based Hopbox gateway device, all DNS queries were routed to our DNS recursive resolver installation. If a user searches for a blocked domain, NXDOMAIN would be served for the query, essentially blocking access. Additionally, dnsmasq then caches the response on Hopbox gateway, thus speeding up the process. Ansible was used to push changes to Hopbox. Right now, the DNS server is functional for 650+ customer locations, serving 240k requests/hour and blocking 4.5k requests/hour on the network.

 
Read more...