OpenStack: Fix “Missing” External IPs in Neutron
Have you ever assigned a “floating” IP on your external network to an OpenStack VM…but not been able to ping it? Read our article for info on how we workaround the problem!
Here’s the scenario: you’ve just created a VM and you’ve just allocated a “floating” IP from your external network. You see that the external IP is properly associated:
[l.abruce@co1]$ nova --os-tenant-name=foobar list +--------------------------------------+------+--------+-------------------------------------+ | ID | Name | Status | Networks | +--------------------------------------+------+--------+-------------------------------------+ | 39612446-a2b9-4b6b-ab25-08af0d3a0f85 | vm1 | ACTIVE | foobar-net=10.0.0.40, 172.20.132.33 | | a288bdce-a0c4-4ba8-b6d7-28274ca47fa1 | vm2 | ACTIVE | foobar-net=10.0.0.3, 172.20.132.23 | +--------------------------------------+------+--------+-------------------------------------+
In the above, we see that the vm2 has internal IP of 10.0.0.40 and external IP 172.20.132.33.
The problem? You try to ping that IP address…and, nothing. Nada. Zilch. Zero.
The solution? Normally, a lot of frantic searching on Google, mass restarts of Neutron services, and serious hand-wringing.
But no more! While the *root cause* is that Neutron notification via AMQP is failing to catch the changed IP, we won’t solve that now (or World Hunger, for that matter). But we will get you back up and runnning; no need to restart services, interrupt clients, etc.
Here’s the process:
- Find the Router ID. For the tenant in question, get the ID of the actual Router object you created:
[l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ neutron --os-tenant-name=foobar router-list | grep foobar | 418f9a8d-b994-44eb-86b2-c5fa0ea9d732 | foobar-router | {"network_id": "cab649b6-c4b2-4da5-af58-15319d244abf", "enable_snat": true} |The Router ID is
418f9a8d-b994-44eb-86b2-c5fa0ea9d732. - Check the Router Configuration on your Neutron Controller. We logon to our system, become
root, and check out the current settings using these commands.- First, verify you can hit the VM on its internal IP. If you can’t…this article won’t help you (you have a different issue or DHCP failure):
[root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ping -c 4 10.0.0.40 PING 10.0.0.40 (10.0.0.40) 56(84) bytes of data. 64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=1.80 ms 64 bytes from 10.0.0.40: icmp_seq=2 ttl=64 time=0.761 ms 64 bytes from 10.0.0.40: icmp_seq=3 ttl=64 time=0.618 ms 64 bytes from 10.0.0.40: icmp_seq=4 ttl=64 time=0.643 ms --- 10.0.0.40 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 0.618/0.956/1.802/0.491 ms
- Assuming that you have connectivity to the internal IP, now get all of the *external* IP addresses bound to that Router:
[root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a 13: qg-ad0b02aa-38:
The above shows that we have two “external” addresses (172.20.132.8 and 172.20.132.23) bound to this Router. The 172.20.132.8 is the IP assigned to the Router object itself, the 172.20.132.23 is for another running VM…so – we are missing 172.20.132.33!
- Next, get the
iptablesinformation for that Router:[root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -S -t nat -P PREROUTING ACCEPT -P POSTROUTING ACCEPT -P OUTPUT ACCEPT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-POSTROUTING -N neutron-l3-agent-PREROUTING -N neutron-l3-agent-float-snat -N neutron-l3-agent-snat -N neutron-postrouting-bottom -A PREROUTING -j neutron-l3-agent-PREROUTING -A POSTROUTING -j neutron-l3-agent-POSTROUTING -A POSTROUTING -j neutron-postrouting-bottom -A OUTPUT -j neutron-l3-agent-OUTPUT -A neutron-l3-agent-OUTPUT -d 172.20.132.23/32 -j DNAT --to-destination 10.0.0.3 -A neutron-l3-agent-POSTROUTING ! -i qg-ad0b02aa-38 ! -o qg-ad0b02aa-38 -m conntrack ! --ctstate DNAT -j ACCEPT -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 -A neutron-l3-agent-PREROUTING -d 172.20.132.23/32 -j DNAT --to-destination 10.0.0.3 -A neutron-l3-agent-float-snat -s 10.0.0.3/32 -j SNAT --to-source 172.20.132.23 -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat -A neutron-l3-agent-snat -s 10.0.0.0/24 -j SNAT --to-source 172.20.132.8 -A neutron-postrouting-bottom -j neutron-l3-agent-snat
Look for the values associated with the external IPs, we’ll add a few more entries.
- First, verify you can hit the VM on its internal IP. If you can’t…this article won’t help you (you have a different issue or DHCP failure):
- Add the missing external IP address to your Router object:
[root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a add 172.20.132.33/32 broadcast 172.20.132.33 scope global dev qg-ad0b02aa-38
You get no output from that command unless you have a failure. Let’s verify:
[root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a show 13: qg-ad0b02aa-38:
We have our missing external IP address…now let’s setup routing for it.
- To setup routing, we issue a few simple commands in the Router object’s context.
- First, map (NAT) outgoing calls from the internal IP address to the external IP:
ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-OUTPUT -d 172.20.132.33/32 -j DNAT --to-destination 10.0.0.40
- Next, map incoming calls to the external IP to the internal IP address:
ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-PREROUTING -d 172.20.132.33/32 -j DNAT --to-destination 10.0.0.40
- Finally, setup source NAT from the internal IP to the external IP:
ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-float-snat -s 10.0.0.40/32 -j SNAT --to-source 172.20.132.33
- First, map (NAT) outgoing calls from the internal IP address to the external IP:
And – that is it! Now let’s verify we can get to the external IP from any other node (like your local laptop):
[l.abruce@co1]$ ping -c 4 172.20.132.33 PING 172.20.132.33 (172.20.132.33) 56(84) bytes of data. 64 bytes from 172.20.132.33: icmp_seq=1 ttl=62 time=5.90 ms 64 bytes from 172.20.132.33: icmp_seq=2 ttl=62 time=11.2 ms 64 bytes from 172.20.132.33: icmp_seq=3 ttl=62 time=13.0 ms 64 bytes from 172.20.132.33: icmp_seq=4 ttl=62 time=2.17 ms --- 172.20.132.33 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3006ms rtt min/avg/max/mdev = 2.175/8.077/13.035/4.300 ms
Success!
Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!
I have been looking for this solution for months! See OpenStack Questions.
Many thanks for a very clear and “obvious” solution. I was on the point of removing OpenStack and trying DevStack, in the faint hope …
Go not gentle in to the cold, dark night
– D. Thomass —- Happpy 100 Dylan.