Optimizing VPN Algorithms and Ciphers for FortiGate Firewalls

Summary

Configuring a secure and efficient VPN on FortiGate firewalls requires careful selection of algorithms, ciphers, and settings. In this post, I’ll share recommendations for VPN configurations, explain the reasoning behind key choices like AES-GCM over AES-CBC, and provide insights into Diffie-Hellman (DH) groups, key lifetimes, and hardware performance.

Note: This guide assumes compatibility of the proposed settings on the remote peer. If this is another FortiGate, it can be easily mirrored. For other vendors/solutions, you'll need to check cipher/algorithms that are supported on both sides, and make your choice from there.

Recommended VPN Settings for FortiGate

For most deployments, I recommend one of two configurations:

OPTION 1: High Security.

OPTION 2: Balanced Security / Performance.

These settings leverage FortiGate’s hardware acceleration and modern cryptographic standards.

Option 1 (High Security)

• Phase 1 Encryption: AES256-GCM

• Phase 1 PRF: PRFSHA512

• Phase 2 Encryption: AES256-GCM

• DH Group: 21 (521-bit ECP)

• IKE Version: IKEv2

Option 2 (Balanced Security and Performance)

• Phase 1 Encryption: AES128-GCM

• Phase 1 PRF: PRFSHA256

• Phase 2 Encryption: AES128-GCM

• DH Group: 19 (256-bit ECP)

• IKE Version: IKEv2

Reasoning

Why AES-GCM Over AES-CBC?

When choosing between AES-GCM (Galois/Counter Mode) and AES-CBC (Cipher Block Chaining), AES-GCM should be preferred for VPNs.

Here’s why:

• Performance: AES-GCM combines encryption and integrity checks in one step, making it faster than AES-CBC, which requires a separate HMAC for authentication.

• Security: AES-GCM’s built-in integrity protection reduces the risk of misconfiguration errors. AES-CBC, while secure when paired with proper HMAC, is more susceptible to attacks like padding oracle exploits if not implemented correctly.

• Hardware Support: FortiGate’s network processors (like the NP6XLite, NP6, NP7Lite and NP7) accelerate AES-GCM, allowing for minimal CPU overhead.

DH Group Selection for Each Option

• DH Group 21 (521-bit ECP) suits Option 1 (AES256-GCM), delivering ~256-bit security that complements the high-security profile use of AES256-GCM. It’s ideal for prioritizing maximum protection.

• DH Group 19 (256-bit ECP) is preferred for Option 2 (AES128-GCM), providing ~128-bit security that aligns with AES128-GCM’s strength. This choice balances security with performance, avoiding unnecessary overhead whilst maintaining good protection.

Why Does a 256-bit ECP Group Provide ~128-bit Security?

In elliptic curve cryptography (ECC), security strength doesn’t scale linearly with key size like it does in symmetric encryption (e.g., AES). Instead, the security level is roughly half the key size. A 256-bit ECP group offers ~128-bit security because the computational effort to solve the discrete logarithm problem on a 256-bit elliptic curve is equivalent to breaking a 128-bit symmetric key. This makes DH Group 19 an efficient and secure match for AES128-GCM, while DH Group 21’s ~256-bit security fits AES256-GCM’s higher requirements.

Why IKEv2?

I recommend IKEv2 over IKEv1 because it’s faster, more reliable, and better suited for modern networks. It handles NAT traversal seamlessly, supports quicker rekeying, and performs well in unstable conditions, making it the smarter choice.

Section References

• Fortinet Documentation: FortiOS IPsec VPN Documentation (fortinet.com) for supported ciphers and Hardware Acceleration Guides (fortinet.com) for hardware acceleration details.
• NIST SP 800-57 Part 1: Recommendation for Key Management (nist.gov) for AES-GCM vs. AES-CBC comparisons.
• RFC 7296: Internet Key Exchange Protocol Version 2 (IKEv2) (ietf.org) for IKEv2 benefits.
• RFC 7748: Elliptic Curves for Security (ietf.org) for ECC security strength.

Diffie-Hellman (DH) Groups – Security and Performance

The Diffie-Hellman (DH) group you choose determines the strength of your VPN’s key exchange. Below is a table summarizing the DH group options available on FortiGate, their security levels, and my recommendations for AES128 and AES256 based options.

DH Groups Summary

DH Group	Type	Key Size	Security Level	Recommendation
1	MODP	768-bit	Weak	Avoid
2	MODP	1024-bit	Weak	Avoid
5	MODP	1536-bit	Weak	Avoid
14	MODP	2048-bit	Acceptable (~112-bit)	Minimum for AES128
15	MODP	3072-bit	Strong (~128-bit)	Good for AES128
16	MODP	4096-bit	Very Strong (~256-bit)	Suitable for AES256
19	ECP	256-bit	Strong (~128-bit)	Good for AES128
20	ECP	384-bit	Very Strong (~192-bit)	Suitable for AES256
21	ECP	521-bit	Very Strong (~256-bit)	Ideal for AES256
27	ECP	192-bit	Moderate (~96-bit)	Avoid (weaker than 21)
28	ECP	224-bit	Moderate (~112-bit)	Avoid (weaker than 21)
29	ECP	256-bit	Strong (~128-bit)	Suitable for AES128
30	ECP	384-bit	Very Strong (~192-bit)	Suitable for AES256 (like 20)
31	ECP	521-bit	Very Strong (~256-bit)	Suitable for AES256 (like 21)
32	ECP	256-bit (Brainpool)	Strong (~128-bit)	Suitable for AES128

Choosing the Right DH Group

• Weak Groups (1, 2, 5): These offer insufficient security for modern standards - avoid them entirely.

• Group 14: A baseline for AES128, but it’s on the edge of acceptability (~112-bit security).

• Groups 15, 19: Strong options for AES128 with ~128-bit security.

• Groups 16, 20, 21: Ideal for AES256, with Group 21 (521-bit ECP) being my top pick due to its efficiency and ~256-bit security.

• Groups 27–28: Weaker than Group 21; skip these unless required for compatibility.

Performance Considerations

FortiGate’s network processors accelerate both MODP and ECP DH groups, but ECP groups (like 19, 20, 21) are more efficient due to their smaller key sizes for equivalent security. Hardware acceleration ensures fast exchange of strong keys without encumbering the CPU.

In case you were wondering about our outlier option, ChaCha20-Poly1305 isn’t hardware-accelerated on FortiGate. This leads to higher CPU usage and lower performance compared to AES-GCM. It also doesn’t offer any meaningful security advantage over AES-GCM in this context, so I recommend against its use.

DH Group Consistency and Security Priority in IPsec VPNs

When configuring an IPsec VPN, the Diffie-Hellman (DH) group is a key component that determines the strength of the cryptographic keys used in both Phase 1 (IKE Security Association) and Phase 2 (IPsec Security Association). I'm adding this in as I've had some prior questions as to whether you need to use the same DH group across both phases and which phase’s security is more critical.

Do We Need the Same DH Group for Phase 1 and Phase 2?

• Not Mandatory: You don’t have to use the same DH group for both phases. Phase 1 establishes the secure channel for key exchange, while Phase 2 (if Perfect Forward Secrecy, or PFS, is enabled) generates keys for the actual data encryption. Different groups can be used to meet specific security or performance needs.

• Practical Choice: That said, using the same DH group simplifies configuration and ensures compatibility between devices. In terms of FortiGate, hardware acceleration often makes the performance impact negligible, so sticking with one strong DH group (e.g., Group 14 or higher) across both phases is a common and effective approach.

• Flexibility: If performance is a concern, you could opt for a stronger DH group in Phase 1 and a slightly less intensive one in Phase 2. Just avoid weak groups like 1, 2, or 5 in either phase. Unless your firewall is extremely loaded however, this is generally unnecessary.

Which Is More Important: Stronger Phase 1 or Phase 2?

• Phase 1 Is Critical: A strong DH group in Phase 1 is more important because it secures the initial key exchange that sets up the VPN tunnel. If an attacker compromises Phase 1, they could potentially intercept or manipulate the keys used in Phase 2, jeopardizing the entire VPN’s security.

• Phase 2 Adds Protection: A strong DH group in Phase 2 (with PFS) enhances security by ensuring each IPsec session has unique keys. This provides forward secrecy - meaning a compromised key doesn’t affect past or future sessions - but its immediate security impact is less critical since Phase 1 already protects the key exchange process.

Recommendation

• Prioritize Phase 1: Always use a robust DH group (e.g., Group 14, 16, or higher) in Phase 1 to safeguard the VPN’s foundation.

• Phase 2 Options: You can use the same strong group in Phase 2 for simplicity and maximum security, or a slightly weaker (but still secure) group if performance optimization is needed.

• Avoid Weakness: Never compromise on weak DH groups in either phase, as this undermines the VPN’s overall security.

Section References

• Fortinet Documentation: Hardware Acceleration Guides (fortinet.com) for hardware acceleration details.
• RFC 4753: ECP Groups for IKE and IKEv2 (ietf.org) for DH group definitions.
• NIST SP 800-56A: Recommendation for Pair-Wise Key Establishment (nist.gov) for DH security levels.
• Imperfect Forward Secrecy (weakdh.org) for weak DH group vulnerabilities.

IPsec Key Lifetimes – Best Practices

Key lifetimes control how often IPsec Security Associations (SAs) are renegotiated, impacting both security and stability. Here’s what I’ve found works best on FortiGate:

• Phase 1 (IKE SA) Lifetime: Set this longer, e.g., 86400 seconds (24 hours).

• Phase 2 (IPsec SA) Lifetime: Keep this shorter or equal to Phase 1, e.g., 43200 seconds (12 hours).

Key Recommendations

• Phase 2 ≤ Phase 1: Phase 2 must renegotiate within a valid Phase 1 SA to avoid disruptions. For example, if Phase 1 is 24 hours, Phase 2 should be at minimum the same, or ideally less.

• No Need for Divisibility: The lifetimes don’t have to be mathematically divisible - just ensure Phase 2 doesn’t outlive Phase 1.

FortiGate defaults (86400s for Phase 1, 43200s for Phase 2) are a solid starting point for most setups, balancing security and simplicity.

Section References

• Fortinet Documentation: FortiOS IPsec VPN Documentation (fortinet.com) for default lifetimes.
• RFC 7296: IKEv2 (ietf.org) for SA lifetime best practices.
• Cisco: IPsec VPN Design Guide (cisco.com) for general lifetime recommendations.

Configuration Examples

Option 1 (Highest Security)

Phase 1

config vpn ipsec phase1-interface
    edit "option1"
        set ike-version 2
        set proposal aes256gcm-prfsha512
        set dhgrp 21
    next
end

Phase 2

config vpn ipsec phase2-interface
    edit "option1"
        set phase1name "option1"
        set proposal aes256gcm
        set dhgrp 21
        set auto-negotiate enable
    next
end

Option 2 (Balanced)

Phase 1

config vpn ipsec phase1-interface
    edit "option2"
        set ike-version 2
        set proposal aes128gcm-prfsha256
        set dhgrp 19
    next
end

Phase 2

config vpn ipsec phase2-interface
    edit "option2"
        set phase1name "option2"
        set proposal aes128gcm
        set dhgrp 19
        set auto-negotiate enable
    next
end

Key Takeaways

• For top-tier security, go with AES256-GCM, DH Group 21, and IKEv2 - (OPTION 1).

• For a balanced approach, AES128-GCM with DH Group 19 or 21 (OPTION 2) works well.

• Avoid ChaCha20-Poly1305 and anything that isn't offloaded due to its lack of hardware acceleration, and stick with AES-GCM to leverage the FortiGate NP hardware.

• Set Phase 2 lifetimes shorter than or equal to Phase 1 (e.g., 43200s and 86400s) for stability

• Leverage the "auto-negotiate" option to keep tunnels active, even in the absence of interesting traffic (unless something like idle data usage is a concern on metered circuits).