Tech and Work at Hopbox

We at Hopbox have been Free Software users and contributors for more than a decade now. We were looking to contribute more to the community in a different from usual way. Downloads from GNU FTP mirrors were slow in India and there existed only 1 mirror (when we decided) for whole of India. As of 4th April 2024, we’re only GNU mirror in India.

GNU is an extensive collection of software packages like Bash, Emacs, Octave and various coreutils. So after a quick discussion, we started hosting a GNU software mirror on our locally hosted Chromebox. The mirror is available on both http://mirrors.hopbox.net/gnu/ and https://mirrors.hopbox.net/gnu/.

As most Free Software don’t have commercial backing and require heavy downloads, the concept of software download mirrors helps take the traffic load off of the primary server, leading to geographical redundancy, higher availability and faster download in general.

At Hopbox, we regularly build a in-house customized version of OpenWrt, a GNU/Linux distribution for routers and embedded devices, which in turn uses GNU coreutils for compilation. Having a local mirror gives us speeds faster than 1 Gbps which is many times faster than our internet speeds, as traffic gets served locally.

The mirror is running on a Dell Optiplex 3020 solely dedicated for mirroring purposes behind a Hopbox APU gateway. The mirror right now is running on base Debian 12 bookworm.

Presently the mirror is serving 30-50 GB of traffic daily since becoming the sole GNU FTP mirror in India with Emacs and Octave fighting for top downloaded package. We generally see traffic from India and neighboring countries.

Basic function of Domain Name System (DNS) is to translate domain names to IP and vice-versa. It’s the phonebook of the computer. Why you may ask, because human’s remember domain names easily, wikipedia.org is easier than 103.102.166.224 . The IP is then used by the computer to fetch required resource for us.

But, DNS also contains other information like Mail Exchanger (MX) records, Nameserver (NS) records, Text (TXT) records amongst other information.

DNS records

DNS systems serves various information through various records. List of major DNS record type: * A record – points to IPv4 address where application is hosted. Like wikipedia.org A record points to 103.102.166.224. Run nslookup domain-name to find it. * AAAA record – points to IPv6 address where application is hosted. wikipedia.org’s AAAA record points to 2620:0:862:ed1a::1. Run nslookup domain-name to find it. * MX – Mail exchange, points to domain’s email server. For wikipedia.org mail is handled by mx1001.wikimedia.org. Run host domain-name to find MX record. * CNAME – canonical name, points to another A record. * NS – points to name server for the domain and sub-domains. Like wikipedia.org’s NS records point to ns2.wikimedia.org. Run dig NS domain-name to find it. * PTR – serves IP to hostname ie domain name * SOA – Start of authority record, serves important information about domain  * TXT – text records for verification and other purposes. * CAA – certificate authority records. * SRV – service record. Used to define hostname and port number for specific services.

Note – All these are GNU/Linux specific commands.

DNS request flow

Each DNS request has to traverse the whole tree once for getting the DNS response. Subsequently, responses are cached at various levels to speed up the process and not to overload the chain of servers.

The flow of request if as follow – Application → Stub resolver (query cache, answer not found) → Recursive resolver (query cache, answer not found) → DNS Root servers → Recursive resolver (populate cache for TLD NS) → Authoritative nameserver TLD → Recursive resolver (populate cache for domain name server) → domain nameserver → Recursive resolver (populates cache for answer) → Stub resolver (populates cache for answer) → Application

1. Application to stub resolver – Where is wikipedia.org?
2. Stub resolver to recursive resolver – Where is wikipedia.org?
3. Recursive resolver to root DNS server – Where is wikipedia.org?
4. Root DNS server to recursive resolver – Ask top level domain authoritative name server for .org TLD like b2.org.afilias-net.org. (bunch of different name servers for redundancy).
5. Recursive resolver to TLD authoritive name server – Where is wikipedia.org?
6. TLD authoritive name server to recursive resolver – Ask domain name server for wikipedia.org – ns1.wikimedia.org (bunch of different name servers for redundancy).
7. Recursive resolver to domain name server – Where is wikipedia.org?
8. Domain name server to recursive resolver – Here’s the A record for wikipedia.org – 103.102.166.224
9. Recursive resolver to stub resolver – wikipedia.org can be found at 103.102.166.224
10.  Stub resolver to application – wikipedia.org can be found at 103.102.166.224

Explainers for each applications

•  Application – Any application which needs to resolve a domain name like browser (Mozilla Firefox), mail client (Thunderbird) etc.
• Stub resolver – A local, system-wide cache like systemd-resolved. dnsmasq etc. They hold the first level of system cache for DNS.
•  Recursive resolver – A dedicate program like BIND, knot, etc which does the actual domain tree traversal ie recursion. It can be running on local or remote system.
•  DNS root server – 13 DNS root servers. Their IP address is pre-programmed in DNS revolvers to start DNS query from. Complete list can be found at https://www.iana.org/domains/root/servers .
•  TLD authoritative name server – Each top level domain like .org has a authoritative name server set which holds name servers for each domain. Run dig NS org. to find them for .org TLD.
•  Domain name server – For each domain, a name server is defined which responses for each query for

DNS Message

To find more details about DNS, we can use a utility called dig. Run dig wikipedia.org to do a DNS query. Response for which would look something like this:

; <<>> DiG 9.18.12-1-Debian <<>> wikipedia.org 
;; global options: +cmd 
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23667 
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232 
; COOKIE: d8a190b00c8d01c201000000646f2e418b9d49b5081970dc (good) 
;; QUESTION SECTION: 
;wikipedia.org.                 IN      A 

;; ANSWER SECTION: 
wikipedia.org.          600     IN      A       103.102.166.224 

;; Query time: 523 msec 
;; SERVER: 10.23.0.30#53(10.23.0.30) (UDP) 
;; WHEN: Thu May 25 15:15:37 IST 2023 
;; MSG SIZE  rcvd: 86			

There are 5 sections to the query – *  Header – shares status of query/resposne, flags and other info. *  Question section – what is being asked/queried – in our case A record for wikipedia.org *  Answer section – response received – in our case we received A record response of type IN (Internet) for wikipedia.org with a time to live (TTL) with IP 103.102.166.224 *  Authority section – SOA response, only avail in authoritative name server response like root-servers.net. Also see authority response in header section as 0. *  Additional info section – resolver info, time, size etc.

DNSSEC

A public-private key method to digitally sign the zone being served. Each domain name server signs the zone they’re serving with their private key. The public key is also shared with the zone.

The trust of public key comes from signature of public key by parent zone. For wikipedia.org, their public key would be signed by .org TLD authoritative name server’s private key, who’s public key would in turn be signed by DNS root servers private key. All recursive resolvers have already pre-defined to trust root’s public key’ making the trust anchor.

Read more

• A warm welcome to DNS – https://powerdns.org/hello-dns/
• Chapter 4 – How DNS work from DNS and Bind, 4th Edition – https://docstore.mik.ua/orelly/networking_2ndEd/dns/index.htm
• Understanding linux dig command – https://www.howtouselinux.com/post/linux-dig-dns

Domain Name System (DNS) can be an effective tool to block access to certain domains in an enterprise setting. This can also help prevent intended or unintended access to known malicious domains.

At Hopbox, we wanted to incorporate customer specific DNS based blocking while also protecting against ransomware, malware, and other newly discovered threats for 900+ locations across customer networks.

Getting the response actually takes multiple back and forth in DNS server hierarchy. A DNS recursive resolver traverses the DNS hierarchy starting from root DNS servers to TLD authoritative servers to domain authoritative servers to get DNS response.

We settled on BIND 9 as our DNS recursive resolver, as it's tried and tested and developed by ISC and used by many of the Internet root DNS servers.

Response Policy Zones

Initially, we tried serving a redirect but that essentially isn't possible without breaking HTTPs and end client trusting our custom TLS certificates, so we choose to serve NXDOMAIN to queries. For routing/serving NXDOMAIN on intended queries, Response Policy Zones (RPZ) were used to collate block lists and allow lists for BIND 9. Most block list providers nowadays provide block lists in RPZ format, which we directly feed into BIND.

Sample RPZ file:

$TTL 3600
@       IN      SOA     localhost.      root.localhost. (
                        1               ; serial
                        1h              ; refresh
                        1h              ; retry
                        1w              ; expiry
                        1h )            ; minimum
 
        IN      NS      localhost.
 
example.org    CNAME    IN    rpz-passthru.

Views

We also wanted customer specific allow and block lists. In DNS terms, we wanted to serve a custom zone for each customer. Concept of Views was according to our requirement. View essentially allowed us to serve custom zones on matching clients source IP (customer networks). We created one View for each customer with their specific allow and block lists.

Secondary DNS setup

For redundancy, we setup-ed multiple secondary server which sync zone files. The primary server sends notification and zone transfer whenever its zones are updated. The Secondary DNS server automatically reloads the updated zone on receiving a full zone transfer.

On primary, the following was added in BIND 9 config:

allow-trasfer { SECONDARY_SERVER_IP;  };
also-notify { SECONDARY_SERVER_IP; };

For each zone definition, the following was added:

master yes;
notify yes;
allow-transfer { SECONDARY_SERVER_IP; }

On secondary, following was added in BIND 9 config, for each zone definition, following was added:

type slave;
masters { PRIMARY_SERVER_IP; };

Deployment

As DNS server is added in dnsmasq in our OpenWrt based Hopbox gateway device, all DNS queries were routed to our DNS recursive resolver installation. If a user searches for a blocked domain, NXDOMAIN would be served for the query, essentially blocking access. Additionally, dnsmasq then caches the response on Hopbox gateway, thus speeding up the process. Ansible was used to push changes to Hopbox. Right now, the DNS server is functional for 650+ customer locations, serving 240k requests/hour and blocking 4.5k requests/hour on the network.