\n \n

\n\n## Build\n\n**This plugin requires Go 1.8 to build.**\n\nGo 1.5 users will need to set `GO15VENDOREXPERIMENT=1` to get vendored dependencies. This flag is set by default in 1.6.\n\n```\n#./build\n```\n## Work flow\n

\n \n

\n## Network configuration reference\n\n* `name` (string, required): the name of the network\n* `type` (string, required): \"multus\"\n* `kubeconfig` (string, optional): kubeconfig file for the out of cluster communication with kube-apiserver, Refer the doc\n* `delegates` (([]map,required): number of delegate details in the Multus, ignored in case kubeconfig is added.\n* `masterplugin` (bool,required): master plugin to report back the IP address and DNS to the container\n\n## Usage with Kubernetes CRD/TPR based Network Objects\n\nKubelet is responsible for establishing the network interfaces for each pod; it does this by invoking its configured CNI plugin. When Multus is invoked, it recovers pod annotations related to Multus, in turn, then it uses these annotations to recover a Kubernetes custom resource definition (CRD), which is an object that informs which plugins to invoke and the configuration needing to be passed to them. The order of plugin invocation is important as is the identity of the primary plugin.\n\nPlease refer the Kubernetes Network SIG - Multiple Network PoC proposal for more details refer the link - [K8s Multiple Network proposal](https://docs.google.com/document/d/1TW3P4c8auWwYy-w_5afIPDcGNLK3LZf0m14943eVfVg/edit)\n\n

\n \n

\n\n### Creating “Network” third party resource in kubernetes\n\nMultus is compatible to work with both CRD/TPR. Both CRD/TPR based network object api self link is same.\n\n#### CRD based Network objects\n\n1. Create a Third party resource “crdnetwork.yaml” for the network object as shown below\n\n```\napiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\nmetadata:\n # name must match the spec fields below, and be in the form: .\n name: networks.kubernetes.com\nspec:\n # group name to use for REST API: /apis//\n group: kubernetes.com\n # version name to use for REST API: /apis//\n version: v1\n # either Namespaced or Cluster\n scope: Namespaced\n names:\n # plural name to be used in the URL: /apis///\n plural: networks\n # singular name to be used as an alias on the CLI and for display\n singular: network\n # kind is normally the CamelCased singular type. Your resource manifests use this.\n kind: Network\n # shortNames allow shorter string to match your resource on the CLI\n shortNames:\n - net\n```\n\n2. kubectl create command for the Custom Resource Definition\n\n```\n# kubectl create -f ./crdnetwork.yaml\ncustomresourcedefinition \"network.kubernetes.com\" created\n```\n\n3. kubectl get command to check the Network CRD creation\n\n```\n# kubectl get CustomResourceDefinition\nNAME KIND\nnetworks.kubernetes.com CustomResourceDefinition.v1beta1.apiextensions.k8s.io\n```\n\n4. Save the below following YAML to flannel-network.yaml\n\n```\napiVersion: \"kubernetes.com/v1\"\nkind: Network\nmetadata:\n name: flannel-networkobj\nplugin: flannel\nargs: '[\n {\n \"delegate\": {\n \"isDefaultGateway\": true\n }\n }\n]'\n```\n5. create the custom resource definition \n```\n# kubectl create -f customCRD/flannel-network.yaml\nnetwork \"flannel-networkobj\" created\n```\n```\n# kubectl get network\nNAME KIND ARGS PLUGIN\nflannel-networkobj Network.v1.kubernetes.com [ { \"delegate\": { \"isDefaultGateway\": true } } ] flannel\n```\n6. Get the custom network object details\n```\n# kubectl get network flannel-networkobj -o yaml\napiVersion: kubernetes.com/v1\nargs: '[ { \"delegate\": { \"isDefaultGateway\": true } } ]'\nkind: Network\nmetadata:\n clusterName: \"\"\n creationTimestamp: 2017-07-11T21:46:52Z\n deletionGracePeriodSeconds: null\n deletionTimestamp: null\n name: flannel-networkobj\n namespace: default\n resourceVersion: \"6848829\"\n selfLink: /apis/kubernetes.com/v1/namespaces/default/networks/flannel-networkobj\n uid: 7311c965-6682-11e7-b0b9-408d5c537d27\nplugin: flannel\n```\n\nBoth TPR and CRD will have same selfLink : **/apis/kubernetes.com/v1/namespaces/default/networks/**\n\nif you are using 1.7 or planning to use 1.8 kubernetes, you can use CRD itself. There is no need to change any thing in Multus. For Kubernetes user < 1.7 use TPR based network objects as follows\n\n#### TPR based Network objects\n\n1. Create a Third party resource “tprnetwork.yaml” for the network object as shown below\n```\napiVersion: extensions/v1beta1\nkind: ThirdPartyResource\nmetadata:\n name: network.kubernetes.com\ndescription: \"A specification of a Network obj in the kubernetes\"\nversions:\n- name: v1\n```\n2. Run kubectl create command for the Third Party Resource\n```\n# kubectl create -f ./tprnetwork.yaml\nthirdpartyresource \"network.kubernetes.com\" created\n```\n3. Run kubectl get command to check the Network TPR creation\n```\n# kubectl get thirdpartyresource\nNAME DESCRIPTION VERSION(S)\nnetwork.kubernetes.com A specification of a Network obj in the kubernetes v1\n```\n##### Creating “Custom Network objects” third party resource in kubernetes\n1. After the ThirdPartyResource object has been created you can create network objects. Network objects should contain network fields. These fields are in JSON format. In the following example, a plugin and args fields are set to the object of kind Network. The kind Network is derived from the metadata.name of the ThirdPartyResource object we created above.\n\n2. Save the below following YAML to flannel-network.yaml\n```\napiVersion: \"kubernetes.com/v1\"\nkind: Network\nmetadata:\n name: flannel-conf\nplugin: flannel\nargs: '[\n {\n \"delegate\": {\n \"isDefaultGateway\": true\n }\n }\n]'\n```\n2. Run kubectl create command for the TPR - Network object\n```\n# kubectl create -f ./flannel-network.yaml \nnetwork \"flannel-conf\" created\n```\n3. Manage the Network objects using kubectl.\n```\n# kubectl get network\nNAME KIND\nflannel-conf Network.v1.kubernetes.com\n```\n4. You can also view the raw JSON data. Here you can see that it contains the custom plugin and args fields from the yaml you used to create it:\n```\n# kubectl get network flannel-conf -o yaml\napiVersion: kubernetes.com/v1\nargs: '[ { \"delegate\": { \"isDefaultGateway\": true } } ]'\nkind: Network\nmetadata:\n creationTimestamp: 2017-06-28T14:20:52Z\n name: flannel-conf\n namespace: default\n resourceVersion: \"5422876\"\n selfLink: /apis/kubernetes.com/v1/namespaces/default/networks/flannel-conf\n uid: fdcb94a2-5c0c-11e7-bbeb-408d5c537d27\nplugin: flannel\n```\n4. The plugin field should be the name of the CNI plugin and args should have the flannel args, it should be in the JSON format as shown above. **User can create network objects for Calico, Weave, Romana, & Cilium and test the multus.** \n5. Save the below following YAML to sriov-network.yaml. Refer [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) or contact @kural in [Intel-Corp Slack](https://intel-corp.herokuapp.com/) for running the DPDK based workloads in Kubernetes\n```\napiVersion: \"kubernetes.com/v1\"\nkind: Network\nmetadata:\n name: sriov-conf\nplugin: sriov\nargs: '[\n {\n \"if0\": \"enp12s0f1\",\n \"ipam\": {\n \"type\": \"host-local\",\n \"subnet\": \"10.56.217.0/24\",\n \"rangeStart\": \"10.56.217.171\",\n \"rangeEnd\": \"10.56.217.181\",\n \"routes\": [\n { \"dst\": \"0.0.0.0/0\" }\n ],\n \"gateway\": \"10.56.217.1\"\n }\n }\n]'\n```\n6. Save the below following YAML to sriov-vlanid-l2enable-network.yaml\n```\napiVersion: \"kubernetes.com/v1\"\nkind: Network\nmetadata:\n name: sriov-vlanid-l2enable-conf\nplugin: sriov\nargs: '[\n {\n \"if0\": \"enp2s0\",\n \"vlan\": 210,\n \"if0name\": \"north\",\n \"l2enable\": true\n }\n]'\n```\n7. Follows the step 2 to create the network object “sriov-vlanid-l2enable-conf” and “sriov-conf”\n8.\tManage the Network objects using kubectl.\n```\n# kubectl get network\nNAME KIND\nflannel-conf Network.v1.kubernetes.com\nsriov-vlanid-l2enable-conf Network.v1.kubernetes.com\nsriov-conf Network.v1.kubernetes.com\n```\n### Configuring Multus to use the kubeconfig\n1.\tCreate Multus CNI configuration file /etc/cni/net.d/multus-cni.conf with below content in minions. Use only the absolute path to point to the kubeconfig file (it may change depending upon your cluster env) and make sure all CNI binary files are in `\\opt\\cni\\bin` dir\n```\n{\n \"name\": \"minion-cni-network\",\n \"type\": \"multus\",\n \"kubeconfig\": \"/etc/kubernetes/node-kubeconfig.yaml\"\n}\n```\n2.\tRestart kubelet service\n```\n# systemctl restart kubelet\n```\n### Configuring Multus to use the kubeconfig and also default networks\n1.\tMany user want default networking feature along with Network object. Refer [#14](https://github.com/Intel-Corp/multus-cni/issues/14) & [#17](https://github.com/Intel-Corp/multus-cni/issues/17) issues for more information. In this following config, Weave act as the default network in the absence of network field in the pod metadata annotation.\n```\n{\n \"name\": \"minion-cni-network\",\n \"type\": \"multus\",\n \"kubeconfig\": \"/etc/kubernetes/node-kubeconfig.yaml\",\n \"delegates\": [{\n \"type\": \"weave-net\",\n \"hairpinMode\": true,\n \"masterplugin\": true\n }]\n}\n```\n2.\tRestart kubelet service\n```\n# systemctl restart kubelet\n```\n\n\n### Configuring Pod to use the TPR Network objects\n1. \tSave the below following YAML to pod-multi-network.yaml. In this case flannel-conf network object act as the primary network. \n```\n# cat pod-multi-network.yaml \napiVersion: v1\nkind: Pod\nmetadata:\n name: multus-multi-net-poc\n annotations:\n networks: '[ \n { \"name\": \"flannel-conf\" },\n { \"name\": \"sriov-conf\"},\n { \"name\": \"sriov-vlanid-l2enable-conf\" } \n ]'\nspec: # specification of the pod's contents\n containers:\n - name: multus-multi-net-poc\n image: \"busybox\"\n command: [\"top\"]\n stdin: true\n tty: true\n```\n3.\tCreate Multiple network based pod from the master node\n```\n# kubectl create -f ./pod-multi-network.yaml\npod \"multus-multi-net-poc\" created\n```\n4.\tGet the details of the running pod from the master\n```\n# kubectl get pods\nNAME READY STATUS RESTARTS AGE\nmultus-multi-net-poc 1/1 Running 0 30s\n```\n### Verifying Pod network\n1.\tRun “ifconfig” command inside the container:\n```\n# kubectl exec -it multus-multi-net-poc -- ifconfig\neth0 Link encap:Ethernet HWaddr 06:21:91:2D:74:B9 \n inet addr:192.168.42.3 Bcast:0.0.0.0 Mask:255.255.255.0\n inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link\n UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1\n RX packets:0 errors:0 dropped:0 overruns:0 frame:0\n TX packets:8 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:0 \n RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)\n\nlo Link encap:Local Loopback \n inet addr:127.0.0.1 Mask:255.0.0.0\n inet6 addr: ::1/128 Scope:Host\n UP LOOPBACK RUNNING MTU:65536 Metric:1\n RX packets:0 errors:0 dropped:0 overruns:0 frame:0\n TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:1 \n RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)\n\nnet0 Link encap:Ethernet HWaddr D2:94:98:82:00:00 \n inet addr:10.56.217.171 Bcast:0.0.0.0 Mask:255.255.255.0\n inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link\n UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1\n RX packets:2 errors:0 dropped:0 overruns:0 frame:0\n TX packets:8 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:1000 \n RX bytes:120 (120.0 B) TX bytes:648 (648.0 B)\n\nnorth Link encap:Ethernet HWaddr BE:F2:48:42:83:12 \n inet6 addr: fe80::bcf2:48ff:fe42:8312/64 Scope:Link\n UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1\n RX packets:1420 errors:0 dropped:0 overruns:0 frame:0\n TX packets:1276 errors:0 dropped:0 overruns:0 carrier:0\n collisions:0 txqueuelen:1000 \n RX bytes:95956 (93.7 KiB) TX bytes:82200 (80.2 KiB)\n```\nInterface name | Description\n------------ | -------------\nlo | loopback\neth0@if41 | Flannel network tap interface\nnet0 | VF0 of NIC 1 assigned to the container by [Intel - SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin\nnorth | \tVF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin\n2.\tCheck the vlan ID of the NIC 2 VFs\n```\n# ip link show enp2s0\n20: enp2s0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000\n link/ether 24:8a:07:e8:7d:40 brd ff:ff:ff:ff:ff:ff\n vf 0 MAC 00:00:00:00:00:00, vlan 210, spoof checking off, link-state auto\n vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto\n vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto\n vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto\n```\n\n## Using Multus Conf file\n\nGiven the following network configuration:\n\n```\n# tee /etc/cni/net.d/multus-cni.conf <<-'EOF'\n{\n \"name\": \"multus-demo-network\",\n \"type\": \"multus\",\n \"delegates\": [\n {\n \"type\": \"sriov\",\n #part of sriov plugin conf\n \"if0\": \"enp12s0f0\", \n \"ipam\": {\n \"type\": \"host-local\",\n \"subnet\": \"10.56.217.0/24\",\n \"rangeStart\": \"10.56.217.131\",\n \"rangeEnd\": \"10.56.217.190\",\n \"routes\": [\n { \"dst\": \"0.0.0.0/0\" }\n ],\n \"gateway\": \"10.56.217.1\"\n }\n },\n {\n \"type\": \"ptp\",\n \"ipam\": {\n \"type\": \"host-local\",\n \"subnet\": \"10.168.1.0/24\",\n \"rangeStart\": \"10.168.1.11\",\n \"rangeEnd\": \"10.168.1.20\",\n \"routes\": [\n { \"dst\": \"0.0.0.0/0\" }\n ],\n \"gateway\": \"10.168.1.1\"\n }\n },\n {\n \"type\": \"flannel\",\n \"masterplugin\": true,\n \"delegate\": {\n \"isDefaultGateway\": true\n }\n }\n ]\n}\nEOF\n\n```\n## Testing the Multus CNI ##\n### Multiple Flannel Network\nGithub user [YYGCui](https://github.com/YYGCui) has used Multiple flannel network to work with Multus CNI plugin. Please refer this [closed issue](https://github.com/Intel-Corp/multus-cni/issues/7) for Multiple overlay network support with Multus CNI.\n\n### docker\nMake sure that the multus, [sriov](https://github.com/Intel-Corp/sriov-cni), [flannel](https://github.com/containernetworking/cni/blob/master/Documentation/flannel.md), and [ptp](https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md) binaries are in the `/opt/cni/bin` directories and follow the steps as mention in the [CNI](https://github.com/containernetworking/cni/#running-a-docker-container-with-network-namespace-set-up-by-cni-plugins)\n\n### Kubernetes\nRefer the Kubernetes User Guide and network plugin\n* [Single Node](https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/)\n* [Multi Node](https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/)\n* [Network Plugin](https://kubernetes.io/docs/admin/network-plugins/)\n\nKubelet must be configured to run with the CNI `--network-plugin`, with the following configuration information. \nEdit `/etc/default/kubelet` file and add `KUBELET_OPTS`:\n```\nKUBELET_OPTS=\"...\n--network-plugin-dir=/etc/cni/net.d\n--network-plugin=cni\n\"\n```\nRestart the kubelet\n```\n# systemctl restart kubelet.service\n```\n#### Launching workloads in Kubernetes\nLaunch the workload using yaml file in the kubernetes master, with above configuration in the multus CNI, each pod should have multiple interfaces.\n> Note:\tTo verify whether Multus CNI plugin is working fine create a pod containing one “busybox” container and execute “ip link” command to check if interfaces management follows configuration.\n\n1. Create “multus-test.yaml” file containing below configuration. Created pod will consist of one “busybox” container running “top” command.\n```\napiVersion: v1\nkind: Pod\nmetadata:\n name: multus-test\nspec: # specification of the pod's contents\n restartPolicy: Never\n containers:\n - name: test1\n image: \"busybox\"\n command: [\"top\"]\n stdin: true\n tty: true\n\n```\n2. Create pod using command:\n```\n# kubectl create -f multus-test.yaml\npod \"multus-test\" created\n```\n3. Run “ip link” command inside the container:\n```\n# 1: lo: mtu 65536 qdisc noqueue qlen 1\n link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n3: eth0@if41: mtu 1500 qdisc noqueue\n link/ether 26:52:6b:d8:44:2d brd ff:ff:ff:ff:ff:ff\n20: net0: mtu 1500 qdisc mq qlen 1000\n link/ether f6:fb:21:4f:1d:63 brd ff:ff:ff:ff:ff:ff\n21: net1: mtu 1500 qdisc mq qlen 1000\n link/ether 76:13:b1:60:00:00 brd ff:ff:ff:ff:ff:ff \n```\n\nInterface name | Description\n------------ | -------------\nlo | loopback\neth0@if41 | Flannel network tap interface\nnet0 | VF assigned to the container by [SR-IOV CNI](https://github.com/Intel-Corp/sriov-cni) plugin\nnet1 | ptp localhost interface\n\n## Contacts\nFor any questions about Multus CNI, please reach out on github issue or feel free to contact the developer @kural in our [Intel-Corp Slack](https://intel-corp.herokuapp.com/)\n\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":22,"_83":11,"_84":11},"read","write","admin","user","storage_size",319699053,"orgData",{"_90":91,"_92":93,"_94":95,"_96":33,"_97":98,"_99":98,"_100":98,"_101":98,"_102":103,"_104":98,"_105":98,"_106":98,"_107":22},"id","bcd2e4d385224efe9aa041c1411e5fae","uuid","bcd2e4d3-8522-4efe-9aa0-41c1411e5fae","type","organization","orgname","full_name","","location","company","profile_url","date_joined","2015-07-17T09:13:36.373649Z","gravatar_url","gravatar_email","badge","is_active","recentTags",[110],{"_111":44,"_112":113,"_114":115,"_90":116,"_117":118,"_50":135,"_136":113,"_137":33,"_138":56,"_57":139,"_140":141,"_142":126,"_143":135,"_144":66,"_145":22},"content_type","creator",373333,"full_size",319690793,19096104,"images",[119],{"_120":121,"_122":123,"_124":98,"_125":126,"_127":128,"_129":130,"_131":98,"_132":-5,"_133":115,"_63":66,"_134":-5},"architecture","amd64","digest","sha256:e0fb27367323381055f1f8fe753bbf3bb1005aae0d518e81b3daf14439b7cb44","features","last_pulled","2026-10-29T00:00:07.716345037Z","last_pushed","2017-12-07T16:10:57Z","os","linux","os_features","os_version","size","variant","2017-12-07T16:10:56.182156Z","last_updater","last_updater_username","media_type","latest","repository",2276860,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxRunInCloud","showSbxAgenticPlatformBadge","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":161,"_168":169,"_155":178,"_179":11},{"_40":162,"_42":163,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":164,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":165,"_79":22,"_80":167,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":166},[78],{"_82":22,"_83":11,"_84":11},"tags",{"_170":64,"_171":-5,"_172":-5,"_173":174},"count","next","previous","results",[175],{"_111":44,"_112":113,"_114":115,"_90":116,"_117":176,"_50":135,"_136":113,"_137":33,"_138":56,"_57":139,"_140":141,"_142":126,"_143":135,"_144":66,"_145":22},[177],{"_120":121,"_122":123,"_124":98,"_125":126,"_127":128,"_129":130,"_131":98,"_132":-5,"_133":115,"_63":66,"_134":-5},[],"canDeleteTags","actionData","errors"]

travelping/multus-cni

Tags for travelping/multus-cni