Optimizing VPN Algorithms and Ciphers for FortiGate Firewalls
Summary
Configuring a secure and efficient VPN on FortiGate firewalls requires careful selection of algorithms, ciphers, and settings. In this post, I’ll share recommendations for VPN configurations, explain the reasoning behind key choices like AES-GCM over AES-CBC, and provide insights into Diffie-Hellman (DH) groups, key lifetimes, and hardware performance.
Note: This guide assumes compatibility of the proposed settings on the remote peer. If this is another FortiGate, it can be easily mirrored. For other vendors/solutions, you'll need to check cipher/algorithms that are supported on both sides, and make your choice from there.
Recommended VPN Settings for FortiGate
For most deployments, I recommend one of two configurations:
OPTION 1: High Security.
OPTION 2: Balanced Security / Performance.
These settings leverage FortiGate’s hardware acceleration and modern cryptographic standards.
Option 1 (High Security)
Phase 1 Encryption: AES256-GCM
Phase 1 PRF: PRFSHA512
Phase 2 Encryption: AES256-GCM
DH Group: 21 (521-bit ECP)
IKE Version: IKEv2
Option 2 (Balanced Security and Performance)
Phase 1 Encryption: AES128-GCM
Phase 1 PRF: PRFSHA256
Phase 2 Encryption: AES128-GCM
DH Group: 19 (256-bit ECP)
IKE Version: IKEv2
Reasoning
Why AES-GCM Over AES-CBC?
When choosing between AES-GCM (Galois/Counter Mode) and AES-CBC (Cipher Block Chaining), AES-GCM should be preferred for VPNs.
Here’s why:
Performance: AES-GCM combines encryption and integrity checks in one step, making it faster than AES-CBC, which requires a separate HMAC for authentication.
Security: AES-GCM’s built-in integrity protection reduces the risk of misconfiguration errors. AES-CBC, while secure when paired with proper HMAC, is more susceptible to attacks like padding oracle exploits if not implemented correctly.
Hardware Support: FortiGate’s network processors (like the NP6XLite, NP6, NP7Lite and NP7) accelerate AES-GCM, allowing for minimal CPU overhead.
DH Group Selection for Each Option
DH Group 21 (521-bit ECP) suits Option 1 (AES256-GCM), delivering ~256-bit security that complements the high-security profile use of AES256-GCM. It’s ideal for prioritizing maximum protection.
DH Group 19 (256-bit ECP) is preferred for Option 2 (AES128-GCM), providing ~128-bit security that aligns with AES128-GCM’s strength. This choice balances security with performance, avoiding unnecessary overhead whilst maintaining good protection.
Why Does a 256-bit ECP Group Provide ~128-bit Security?
In elliptic curve cryptography (ECC), security strength doesn’t scale linearly with key size like it does in symmetric encryption (e.g., AES). Instead, the security level is roughly half the key size. A 256-bit ECP group offers ~128-bit security because the computational effort to solve the discrete logarithm problem on a 256-bit elliptic curve is equivalent to breaking a 128-bit symmetric key. This makes DH Group 19 an efficient and secure match for AES128-GCM, while DH Group 21’s ~256-bit security fits AES256-GCM’s higher requirements.
Why IKEv2?
I recommend IKEv2 over IKEv1 because it’s faster, more reliable, and better suited for modern networks. It handles NAT traversal seamlessly, supports quicker rekeying, and performs well in unstable conditions, making it the smarter choice.
Section References
Diffie-Hellman (DH) Groups – Security and Performance
The Diffie-Hellman (DH) group you choose determines the strength of your VPN’s key exchange. Below is a table summarizing the DH group options available on FortiGate, their security levels, and my recommendations for AES128 and AES256 based options.
DH Groups Summary
DH Group
Type
Key Size
Security Level
Recommendation
1
MODP
768-bit
Weak
Avoid
2
MODP
1024-bit
Weak
Avoid
5
MODP
1536-bit
Weak
Avoid
14
MODP
2048-bit
Acceptable (~112-bit)
Minimum for AES128
15
MODP
3072-bit
Strong (~128-bit)
Good for AES128
16
MODP
4096-bit
Very Strong (~256-bit)
Suitable for AES256
19
ECP
256-bit
Strong (~128-bit)
Good for AES128
20
ECP
384-bit
Very Strong (~192-bit)
Suitable for AES256
21
ECP
521-bit
Very Strong (~256-bit)
Ideal for AES256
27
ECP
192-bit
Moderate (~96-bit)
Avoid (weaker than 21)
28
ECP
224-bit
Moderate (~112-bit)
Avoid (weaker than 21)
29
ECP
256-bit
Strong (~128-bit)
Suitable for AES128
30
ECP
384-bit
Very Strong (~192-bit)
Suitable for AES256 (like 20)
31
ECP
521-bit
Very Strong (~256-bit)
Suitable for AES256 (like 21)
32
ECP
256-bit (Brainpool)
Strong (~128-bit)
Suitable for AES128
Choosing the Right DH Group
Weak Groups (1, 2, 5): These offer insufficient security for modern standards - avoid them entirely.
Group 14: A baseline for AES128, but it’s on the edge of acceptability (~112-bit security).
Groups 15, 19: Strong options for AES128 with ~128-bit security.
Groups 16, 20, 21: Ideal for AES256, with Group 21 (521-bit ECP) being my top pick due to its efficiency and ~256-bit security.
Groups 27–28: Weaker than Group 21; skip these unless required for compatibility.
Performance Considerations
FortiGate’s network processors accelerate both MODP and ECP DH groups, but ECP groups (like 19, 20, 21) are more efficient due to their smaller key sizes for equivalent security. Hardware acceleration ensures fast exchange of strong keys without encumbering the CPU.
In case you were wondering about our outlier option, ChaCha20-Poly1305 isn’t hardware-accelerated on FortiGate. This leads to higher CPU usage and lower performance compared to AES-GCM. It also doesn’t offer any meaningful security advantage over AES-GCM in this context, so I recommend against its use.
DH Group Consistency and Security Priority in IPsec VPNs
When configuring an IPsec VPN, the Diffie-Hellman (DH) group is a key component that determines the strength of the cryptographic keys used in both Phase 1 (IKE Security Association) and Phase 2 (IPsec Security Association). I'm adding this in as I've had some prior questions as to whether you need to use the same DH group across both phases and which phase’s security is more critical.
Do We Need the Same DH Group for Phase 1 and Phase 2?
Not Mandatory: You don’t have to use the same DH group for both phases. Phase 1 establishes the secure channel for key exchange, while Phase 2 (if Perfect Forward Secrecy, or PFS, is enabled) generates keys for the actual data encryption. Different groups can be used to meet specific security or performance needs.
Practical Choice: That said, using the same DH group simplifies configuration and ensures compatibility between devices. In terms of FortiGate, hardware acceleration often makes the performance impact negligible, so sticking with one strong DH group (e.g., Group 14 or higher) across both phases is a common and effective approach.
Flexibility: If performance is a concern, you could opt for a stronger DH group in Phase 1 and a slightly less intensive one in Phase 2. Just avoid weak groups like 1, 2, or 5 in either phase. Unless your firewall is extremely loaded however, this is generally unnecessary.
Which Is More Important: Stronger Phase 1 or Phase 2?
Phase 1 Is Critical: A strong DH group in Phase 1 is more important because it secures the initial key exchange that sets up the VPN tunnel. If an attacker compromises Phase 1, they could potentially intercept or manipulate the keys used in Phase 2, jeopardizing the entire VPN’s security.
Phase 2 Adds Protection: A strong DH group in Phase 2 (with PFS) enhances security by ensuring each IPsec session has unique keys. This provides forward secrecy - meaning a compromised key doesn’t affect past or future sessions - but its immediate security impact is less critical since Phase 1 already protects the key exchange process.
Recommendation
Prioritize Phase 1: Always use a robust DH group (e.g., Group 14, 16, or higher) in Phase 1 to safeguard the VPN’s foundation.
Phase 2 Options: You can use the same strong group in Phase 2 for simplicity and maximum security, or a slightly weaker (but still secure) group if performance optimization is needed.
Avoid Weakness: Never compromise on weak DH groups in either phase, as this undermines the VPN’s overall security.
Section References
IPsec Key Lifetimes – Best Practices
Key lifetimes control how often IPsec Security Associations (SAs) are renegotiated, impacting both security and stability. Here’s what I’ve found works best on FortiGate:
Phase 1 (IKE SA) Lifetime: Set this longer, e.g., 86400 seconds (24 hours).
Phase 2 (IPsec SA) Lifetime: Keep this shorter or equal to Phase 1, e.g., 43200 seconds (12 hours).
Key Recommendations
Phase 2 ≤ Phase 1: Phase 2 must renegotiate within a valid Phase 1 SA to avoid disruptions. For example, if Phase 1 is 24 hours, Phase 2 should be at minimum the same, or ideally less.
No Need for Divisibility: The lifetimes don’t have to be mathematically divisible - just ensure Phase 2 doesn’t outlive Phase 1.
FortiGate defaults (86400s for Phase 1, 43200s for Phase 2) are a solid starting point for most setups, balancing security and simplicity.
Section References
Cisco: IPsec VPN Design Guide (cisco.com) for general lifetime recommendations.
Configuration Examples
Option 1 (Highest Security)
Phase 1
Phase 2
Option 2 (Balanced)
Phase 1
Phase 2
Key Takeaways
For top-tier security, go with AES256-GCM, DH Group 21, and IKEv2 - (OPTION 1).
For a balanced approach, AES128-GCM with DH Group 19 or 21 (OPTION 2) works well.
Avoid ChaCha20-Poly1305 and anything that isn't offloaded due to its lack of hardware acceleration, and stick with AES-GCM to leverage the FortiGate NP hardware.
Set Phase 2 lifetimes shorter than or equal to Phase 1 (e.g., 43200s and 86400s) for stability
Leverage the "auto-negotiate" option to keep tunnels active, even in the absence of interesting traffic (unless something like idle data usage is a concern on metered circuits).
Last updated
Was this helpful?