There is no way they can avoid this kind of public notice.

The seeds of this outcome were planted years ago when sales comp plans changed. When a sales rep can hit their target by simply converting the way an existing customer gets billed, none of them look for new business. Don't need new leads. Don't need to win competitive deals. But finding new customers and losing opportunities are the only things that signal/drive innovation. But from a budgeting perspective, why increase investment in a product that already hits/exceeds their sales targets?

Over time sales targets get met, but the product doesn't advance. By the time all existing customers that can convert have converted, the product is no longer competitive. Like bankruptcy, it comes gradually, then suddenly.

But this is the problem that most designs are trying to solve. Large L2s are notoriously fragile. 1,000 nodes, 50-100 pods/node is a lot of ARPs. And sometimes you want partitions between endpoints for security/isolation.

I agree with you about static assignment of addresses. But that's why (most) CNIs work with a controller of some kind for IPAM.

IMO, the problem complexity is hard to compress. You need to distribute/manage MAC addresses, routes, and/or state. Different designs would favor one over another.

To the extent your requirement match theirs, this could be a good alternative. The most significant in my mind is that it's meant to be used in conjunction with Envoy. Envoy itself has its own set of design tradeoffs as well.

For example, Lyft currently uses 'service-assigned EC2 instances'. Not hard to see how this starting point would influence the design. The Envoy/Istio model of proxy per pod also reflects this kind of workload partitioning. Obviously, a design for a small number of pods (each with their own proxy) per instance is going to be very different from one that needs to handle 100 pods (and their IPs), or more, per instance.

Another is that k8s network policy can't be applied since the 'Kubernetes Services see connections from a node’s source IP instead of the Pod’s source IP'. But I don't think this CNI is intended to work with any other network policy API enforcement mechanism. Romana (the project I work on) and the other CNI providers that use iptables to enforce network policy rely on seeing the pod's source IP.

Again, this might be fine if you're running Envoy. On the other hand, L3 filtering on the host might be important.

Also, this design requires that 'CNI plugins communicate with AWS networking APIs to provision network resources for Pods'. This may or may not be something you want your instances to do.

FWIW, Romana lets you build clusters larger than 50 nodes without an overlay or more 'exotic networking techniques' or 'massive' complexity. It does it via simple route aggregation, completely standard networking.

Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common.

Using overlapping IPs puts you right back into the overlay model.

>Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix.

Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.

The net effect is that you can build large clusters without running out of VPC routes and no overlay is needed when traffic crosses AZs.

When a route is used to forward traffic for multiple instances, the target instance acts as router and forwards traffic to the final destination instance. This works because instances within an AZ have routes installed on them to the pod CIDRs on the other instances in the zone, so any one of them can perform this forwarding function.

Romana only piggybacks routes when there are no more VPC routes available, so for small cluster it's just like kubenet. For large clusters routes it uses all the instances to forward traffic so that none of them become a bottleneck.

Comments URL: https://news.ycombinator.com/item?id=12582163

Points: 4

# Comments: 1

The nice thing about this is that nothing has to happen for a new pod to be reachable. No /32 route distribution or BGP (or etcd) convergence, no VXLAN ID (VNID) distribution for the overlay. At some scale, route and/or VNID distribution is going to limit the speed at which new pods can be launched.

One other thing not mentioned in the blog post or in any of these comments is network policy and isolation. Kubernetes v1.3 includes the new network APIs that let you isolate namespaces. This can only be achieved with a back end network solution like Romana or Calico (some others as well).

[1] romana.io

The results of these benchmarks do not surprise me at all. To me, they all fall in to the category of 'less (overhead) is more (performance)'. With VXLAN encap being the obvious example of greatest overhead.

I think its also worth mentioning that k8s networking is being enhanced in v1.2 to support isolation and multi-tenancy through ThirdParty resources (back end network solutions). The alternatives included in the benchmarks aren't going to be able to support these kinds of network policy as is.

And, unfortunately, things get a more complicated when you want to provide more than simple reachability (which is all that k8s asks for today). The tradeoff is to be able to control the packets with the lowest overhead possible. VXLANs will give you isolation, but at the cost of encapsulation. Stacking bridges and tunnels and distributing VNIDs/routes not only introduces more latency, but becomes another multi-host coordination problem (matching tunnel IDs, etc).

We're working on a new way to build cloud native networks that avoids the encap, but still lets you control all the packets.

You can learn more at http://romana.io if you're interested.

Comments URL: https://news.ycombinator.com/item?id=2403281

Points: 1

# Comments: 0