Get started with 40GbE SDN with Microsoft Azure SONiC for under $1K (2024)

Set up the IP Addresses

As it is installed in build #270, SONiC defines each of the 32 ports with a with an EthernetX device and a 10.0.0.Y address on port N range 1→32, where X = (N-1) * 4 and Y = (N-1) * 2. These are the IP addresses assigned to ports 17 and 18:

root@arista:~# ip addr show | egrep 'inet.*Ethernet6[48]'inet 10.0.0.32/31 brd 255.255.255.255 scope global Ethernet64inet 10.0.0.34/31 brd 255.255.255.255 scope global Ethernet68

The first 16 ports use Internal BGP, the second 16 ports are External BGP. Since External BGP distributes the routes across the cluster and I am not connecting BGP to the Internet, I used the ports 17→32. I plan on creating a Layer 2 LANs with the first 16 ports.

The addresses assigned to the hosts are:

Since the layer 2 network set up using this config is confined to the link between the switch and the port, the host has to route through the switch in order to connect with the other ports otherwise any IP address other than the link pair will try to route to the wrong network. I did this by setting up my router as the default-gateway, which propagates default route through the switch to the hosts. My initial 6 host config looks like this:

Install quagga and start zebra.

root@router:~# yum install quaggaroot@router:~# systemctl start zebra

My 40gbe devices are named fo0 and fo1.

root@router:~# vtyshrouter# config termrouter(config)# log file /var/log/quagga/quagga.logrouter(config)# interface fo0router(config-if)# ip address 10.0.0.49/31router(config-if)# no shutdownrouter(config-if)# endrouter# writeBuilding Configuration...Configuration saved to /etc/quagga/zebra.conf

You should be able to ping from the host to 10.0.0.48, which is port 25 on the switch, in this case, my Fedora 26 Router.

root@router:~# ping -c 3 10.0.0.48PING 10.0.0.48 (10.0.0.48) 56(84) bytes of data.64 bytes from 10.0.0.48: icmp_seq=1 ttl=64 time=0.316 ms64 bytes from 10.0.0.48: icmp_seq=2 ttl=64 time=0.188 ms64 bytes from 10.0.0.48: icmp_seq=3 ttl=64 time=0.156 ms--- 10.0.0.48 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 1999msrtt min/avg/max/mdev = 0.059/0.062/0.066/0.003 ms

One other thing I did on the router was route the 10.0.0.0/26 range through the switch:

root@router:~# vtyshrouter# config termrouter(config)# ip route 10.0.0.0/26 fo0router(config-if)# endrouter# write

Repeat the above for each host connected, except for the last line.

Set up BGP Routing

Each of my hosts has a docker0 device on it using a unique IP address range. BGP is going to distribute the routes for the docker0 bridge across the cluster. This is the Docker dameon config file that causes docker to create a
default bridge at 192.168.37.1/24.

root@host:~# cat /etc/docker/daemon.json{ "dns": ["192.168.255.1"], "iptables": false, "ip-forward": false, "ip-masq": false, "storage-driver": "btrfs", "bip": "192.168.37.1/24", "fixed-cidr": "192.168.37.0/24", "mtu": 1500}root@host:~# systemctl start dockerroot@host:~# brctl showbridge name bridge id STP enabled interfacesdocker0 8000.0242deb9f0e5 noroot@host:~# ip -4 addr show dev docker08: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN inet 192.168.37.1/24 scope global docker0 valid_lft forever preferred_lft forever

The DNS entry is a caching named on the router. I set up a loopback device with this IP address and distributed it with BGP.

root@router:~# cat /etc/sysconfig/network-scripts/ifcfg-lo:1DEVICE=lo:1IPADDR=192.168.255.1NETMASK=255.255.255.0NETWORK=192.168.255.0BROADCAST=192.168.255.255ONBOOT=yes

Here is the /etc/quagga/bgpd.conf for the above Docker bridge on the CentOS 7 server plugged into port 21.

root@host:~# cat /etc/quagga/bgpd.confhostname myhostpassword zebralog file /var/log/quagga/bgp.logrouter bgp 64005 bgp router-id 10.0.0.41 network 192.168.37.0/24 neighbor 10.0.0.40 remote-as 65100

Start bgpd to communicate with the BGP running on switch.

root@host:~# systemctl start bgpd

The 64005 AS number above is unique for each port from 17 → 32. These are configured be on the switch in the BGP container in /etc/quagga/bgpd.conf via the Redis database.

Sinced the above host is in port 21, the AS number is 64005 (AS is a BGP acronym for Autonomous system). After starting bgpd on each of the hosts, you should see the routes populated by BGP with the protocol zebra. On the switch it shows this:

root@arista:/# ip route show | grep zebraip route show | grep zebradefault via 10.0.0.49 dev Ethernet96 proto zebra src 10.1.0.32192.168.74.0/24 via 10.0.0.33 dev Ethernet64 proto zebra src 10.1.0.32192.168.111.0/24 via 10.0.0.37 dev Ethernet72 proto zebra src 10.1.0.32192.168.37.0/24 via 10.0.0.41 dev Ethernet80 proto zebra src 10.1.0.32192.168.120.0/24 via 10.0.0.45 dev Ethernet88 proto zebra src 10.1.0.32192.168.255.0/24 via 10.0.0.49 dev Ethernet96 proto zebra src 10.1.0.32192.168.66.0/24 via 10.0.0.51 dev Ethernet96 proto zebra src 10.1.0.32

All the hosts are set up the same, except for the router. The router has this in the /etc/quagga/bgpd.conf:

hostname routerpassword zebralog file /var/log/quagga/bgp.logrouter bgp 64009 bgp router-id 10.0.0.49 network 192.168.255.0/24 neighbor 10.0.0.48 remote-as 65100 neighbor 10.0.0.48 default-originate

Note the “default-originate” line. This propagates the default route to the hosts and the switch.

I also block the BGP port from the internet side:

root@host:~# iptables -A INPUT -i te0 -p udp -m udp --dport 179 -j DROProot@host:~# iptables -A INPUT -i te0 -p tcp -m tcp --dport 179 -j DROP

Set up Layer 2 Vlans

In order to use a port in normal ethernet mode, a Vlan has to be created and ports need to be added to it. There is no DEFAULT_VLAN.

The location of the SONiC config file is on the switch here:
/etc/sonic/sonic_db.json

When the switch boots, it loads this into the Redis database. The switch daemons that are running in Docker monitor the database through pub/sub subscriptions and update the operating state. The VLAN part that becomes the Debian ifupdown configuration uses a Python templating language called Jinja2.

To add a Vlan, there are two components of the json file that need to be updated, the “VLAN” field and the “VLAN_INTERFACE” field. The following is the first 10 ports and the last two ports (1→10, 31→32) configured for the Vlan1000. I am using the 10.1.1.0/24 network for this VLAN.

 "VLAN": { "Vlan1000": { "members": [ "Ethernet0", "Ethernet4", "Ethernet8", "Ethernet12", "Ethernet16", "Ethernet20", "Ethernet24", "Ethernet28", "Ethernet32", "Ethernet36", "Ethernet120", "Ethernet124" ], "vlanid": "1000" } }, "VLAN_INTERFACE": { "Vlan1000|10.1.1.1/24": {} },

The template interfaces.j2 generates the file /etc/network/interfaces from the above json, which looks like this:

root@arista:/# tail -15 /etc/network/interfacesallow-hotplug Ethernet124#iface Ethernet124 inet manual pre-up ifconfig Ethernet124 up mtu 9100 post-up brctl addif Vlan1000 Ethernet124 || true post-down ifconfig Ethernet124 down## Vlan interfacesauto Vlan1000iface Vlan1000 inet static bridge_ports none hwaddress ether 44:4c:a8:12:87:c6 address 10.1.1.1 netmask 255.255.255.0#

After startup, you can verify that the ports configured are indeed in the Vlan1000 bridge:

root@arista:/# brctl showbridge name bridge id STP enabled interfacesVlan1000 8000.444ca81287c6 no Ethernet0 Ethernet12 Ethernet120 Ethernet124 Ethernet16 Ethernet20 Ethernet24 Ethernet28 Ethernet32 Ethernet36 Ethernet4 Ethernet8docker0 8000.0242d26920e5 no

I also removed these ports from the other sections of the configuration file, specifically the “BGP_NEIGHBOR”, “DEVICE_NEIGHBOR”, and “INTERFACE”.

Since my router is populating the default route for the BGP network, I also set the management address to be static, otherwise the DHCP default route overrides the BGP. This configured in the “MGMT_INTERFACE” section of the json file:

 "MGMT_INTERFACE": { "eth0|10.3.2.63/24": {} },