The SPF lookup limit explained

An SPF policy may not require more than 10 terms that require additional DNS lookups to evaluate. If the limit is exceeded, an email message may fail SPF inspection which can cause deliverability issues, and may hurt domain reputation.

In this post we'll explain what this lookup limit is, how it can affect your email deliverability, and how to reduce the number of required lookups.

What is SPF?

By default, any computer connected to the internet can send email to any email inbox with any sender name. This means that, without additional countermeasures, anyone could send an email as president@whitehouse.gov.

The Sender Policy Framework (SPF) is a standard that is part of the email ecosystem that aims at preventing this form of email identity fraud. SPF is also used as one of the factors in detecting spam messages.

An SPF policy is a list of senders (hosts) that are allowed to send email on behalf of a domain. The policy is published as a DNS record under the domain it applies to.

When an email message is received by an email service, the receiver uses SPF to determine if the computer that sent the message was allowed to do so. If the sender does not pass SPF validation, the message is likely to be rejected, or flagged as spam or fraud.

We have a longer explanation of SPF in our knowledge base.

The SPF policy format

An SPF policy consists of multiple terms separated by whitespace. A term can be a modifier (such as v or redirect), or a matching mechanism (such as a, mx, include, etc.).

A matching term has the following format:

[prefix][mechanism][:value]

The prefix determines the SPF validation outcome that the receiver should apply to the message if the sender matches the term. Allowed values are + (pass), ? (neutral), ~ (soft fail) or - (fail). The prefix is optional, if no prefix is defined, it defaults to pass (+).

The mechanism determines how to match an IP address against the term, supported values are a, ipv4, ipv6, mx, ptr, include, exists and all.

The value portion of a term is optional, and depends on the used mechanism. For most mechanisms the value allows you to point to other domains, and if omitted it defaults to the current domain.

SPF term matching

Email services communicate using IP addresses, not domain names. If an SMTP service receives an email, it uses SPF to determine if the IP-address of the sender matches one of the terms in the SPF record.

The receiver iterates the terms in the SPF policy from left-to-right, looking for a term that matches the sender IP address using the specified mechanism. Once a match is found, iteration stops, and the receiver applies the action as defined in the prefix value of the matching term.

If no match can be found, the result of SPF validation is 'neutral', meaning no SPF validation is used in spam detection. This is why SPF policies usually end with a term using the all mechanism, which always matches. The all mechanism is typically combined with the ~ (soft-fail) prefix. (why Mailhardener recommends SPF softfail over fail).

Additional DNS queries

Most SPF mechanisms require the validator to perform additional DNS queries to match the IP address against it.

For example: the SPF a mechanism means: match if the IP address equals any of the DNS A records of this domain.

So, in order to match against a term with an a mechanism, the validator must first perform an A (or AAAA) DNS query on the domain. SPF policies with multiple terms can require more DNS lookups. Some mechanisms require more than one additional lookup.

Most mechanisms, except for ip4, ip6 and all will require the validator to perform additional lookups.

Additionally, the redirect modifier will also cause an additional lookup.

Example

Consider the following SPF record that is published under example.com:

v=spf1 a -all

The example policy above has three terms.

• The first term is the v modifier, which is required at the beginning of an SPF record.
• The second term is a matching term that uses the a mechanism, no prefix is set, so it defaults to + (pass). No value is set, so it defaults to the domain where the SPF is published. So the full format of this term is +a:example.com.
• The third term is the all mechanism, which matches any IP address, it has a - (fail) prefix.

This term means: SPF validation should pass if the sender matches any of the DNS A records of example.com and fail on any other IP address.

This SPF policy requires the receiver to perform 1 additional SPF lookup (example.com A) to fully evaluate.

The lookup limit

Performing DNS queries costs the validator resources (bandwidth, time, CPU, memory). So to avoid 'unreasonable load' on the validator, RFC7208 section 4.6.4 states that evaluation of an SPF policy may not exceed 10 terms that require DNS lookups. The DNS query for the SPF policy record itself does not count towards this limit.

According to the RFC, a validator (the receiving email system) must not proceed after 10 lookups, and reject the SPF validation with a permerror error.

Additionally, the RFC states that a DNS query of a hostname found in an MX record must not yield more than 10 A or AAAA records.

If a DNS PTR query (reverse-DNS lookup) yields more than 10 results, only the first 10 results are to be used. Please note that the use of the SPF ptr mechanism is strongly discouraged, and should not be used.

Note that RFC7208 section 4.6.4 states that the limit is 10 terms (that require additional DNS lookups), not necessarily 10 DNS queries. This has caused is a lot of confusion, debate and misinformation about the SPF lookup limit. An SPF record can actually require more than 10 DNS queries to evaluate, but still be within the 10 term limit and thus be valid.

 

For example: the SPF mx mechanism will require the evaluator to query the DNS for MX, then query A or AAAA records for each of the hosts. So the number of required DNS queries is 1+N, but the mx SPF mechanism still only counts as one term towards the limit.

Exceeding the limit

If a receiver exceeds the DNS lookup limit while evaluating the SPF policy, it must fail the SPF validation for that message with a permerror. This error can be observed when using DMARC monitoring.

The resulting action of the permerror failure is for the receiver to decide. Some receivers will reject (bounce) the email completely. Some receivers give the email a 'neutral' SPF result (as if no SPF is used), while other receivers will set the SPF result to 'fail' or 'softfail'. Usually there are multiple other factors such as DMARC, DKIM, spam rating, etc. that the receiver uses to determine if the message should be delivered to the recipient's inbox.

A permerror during SPF validation reduces the likelihood that the message is delivered at all. It increases the chance of the message being flagged as spam or potential fraud. If the receiver utilizes a domain or sender rating system, a permerror will negatively impact the rating.

Remember that validators evaluate the terms in the SPF policy from left-to-right. As soon as a match on the sender IP address is found, evaluation stops. So depending on the sender, a validator may not always reach the lookup limit, even if the policy requires more than 10 lookups to fully evaluate. This makes SPF lookup limit related deliverability issues particularly difficult to identify.

Note that there are more reasons for a validator to return a permerror, not just the DNS lookup limit. So whenever you see a permerror as the SPF validation result in a DMARC report, you may have a DNS lookup limit problem, but it could also be a different issue with your SPF policy (such as a malformed record). You can use our free SPF record validator to check if your DNS policy record is valid, it will also report the maximum required lookups.

How to reduce the number of required lookups

The limit of 10 additional lookups is quite low. The way that organizations now use email is quite different from what it used to be in 2006 when the first SPF standard was initially finalized in RFC4408 (now obsoleted by RFC7208). organizations may use various cloud based email services with a single domain.

It is quite common to see SPF policies exceeding the SPF lookup limit. For some domains, it may be quite challenging to stay within the 10 lookup limit.

Here are some tips to follow to reduce the number of required lookups:

Remove unused services

The most basic step is to check your SPF record and remove any services that you may no longer use. Check your records for any include or other mechanism that points to a domain of a service that is no longer in use.

Remove the default SPF values

Most hosting services set a 'default' SPF policy whenever a new domain is provided. The default is usually something like v=spf1 a mx. Most A/AAAA DNS records are used for web servers that may not send email, so the a mechanism may not be needed. The mx mechanism may not be needed, as mx is for receiving email, not necessarily for sending.

Do not use of the ptr mechanism

The ptr mechanism is strongly discouraged by the current SPF RFC and should not be used due to various security and reliability issues. The ptr mechanism can cause a big increase in required lookups, that you cannot control.

Avoid the use of the mx mechanism

You may not need to have mx in your policy. Remember that MX (Mail eXchange) records are used for receiving email, not necessarily sending. If you use a cloud based email service such as G-Suite or Office 365, the include mechanism should be used, and the mx mechanism omitted.

Use ip4 or ip6 mechanism, if suitable

The ip4 and ip6 mechanisms require no additional lookups, and are thus 'free' to use. When your organization manages their own email services, you may want to use ip4 and/or ip6 mechanisms to set the IP addresses of those services directly. Be aware that IP addresses are subjective to change, thus may require more maintenance on the policy. The ip4 and ip6 mechanisms are therefore prone to errors if not kept up-to-date.

Use a dynamic SPF policy service

As a last resort, you may use a 'dynamic' SPF policy service such as autoSPF. In general, we wouldn't recommend using such services as it increases complexity and adds failure points to the email infrastructure.

Do NOT flatten records

'Flattening' of SPF records is sometimes suggested on various internet forums as a means of reducing SPF lookups. Some go as far as claiming that the shorter the policy, the better your domain's 'reputation' will become. We have absolutely no reason to believe that this is true, and strongly discourage this practice. Flattening SPF records is prone to errors, and requires constant maintenance. We even wrote a dedicated article on the subject.

Validate your record after you make changes

To prevent deliverability issues, always validate your SPF records when making changes, to assure the SPF policy does not allow for more than 10 lookups.

Conclusion

The SPF standard RFC7208 mandates that an SPF policy may not take more than 10 additional DNS lookups to fully evaluate. When a receiver has to perform more than 10 lookups to evaluate the SPF policy, the email message fails SPF validation with a permerror status, which may prevent the email message from being delivered.

The SPF DNS lookup limit is an often overlooked, but essential factor in email deliverability.

The limit of 10 lookups is a bit outdated for the way that email is used nowadays. With the advance of cloud based email services and marketing platforms, the limit is easily exceeded. Care must be taken to prevent exceeding the lookup limit.

When in doubt, validate your SPF records to assure the SPF policy does not allow for more than 10 lookups.