1.First of all, the key problems we wanted to tackle:

Handling hardcoded values

Handling remote state dependencies

Handling intricate modules (public + private)

we knew that it was possible to do it without adding additional frameworks, by going through the Terraform State Files.

2.Key Assumptions:

Your infra is a graph. To model the infrastructure accurately, we used Neo4j to capture relationships between resources, states, and modules.

All the information you need is within your cloud and code: By parsing both, we could recreate the chain of dependencies and insights without additional overhead.

Our goal was to build a digital twin of the infrastructure. Encompassing code, state, and cloud information to surface and prevent issues early.

3.Our solution:

To handle downstream dependencies we are :

Creating a digital twin of the infra with all the dependencies between IaC code and cloud

For each PR, querying this graph with Cypher (Neo4J query language) to retrieve those dependencies

-> Build an up-to-date Cloud-to-Code graph

i - Understanding Terraform Stat Files

Terraform state files are super rich in term of information, way more than the files. They hold the exact state of deployed resources, including:

Resource types

Unique identifiers

Relationships between modules and their resources

By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities. By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities.

ii- Building this graph using Neo4J

Neo4j allowed us to model complex relationships natively. Unlike relational databases, graph databases are better suited for interconnected data like infrastructure resources.

We modeled infrastructure as nodes (e.g., EC2 instances, VPCs) and relationships (e.g., "CONNECTED_TO," "IN_REGION"). For example:

Nodes: Represent resources like an EC2 instance or a Security Group.

Relationships: Define how resources interact, such as an EC2 instance being attached to a Security Group.

iii- Extracting and Reconciling Data

We developed services to parse state files from multiple repositories, extracting relevant data like resource definitions, unique IDs, and relationships. Once extracted, we reconciled:

Resources from code with resources in the cloud.

Dependencies across repositories, resolving naming conflicts and overlaps.

We also labeled nodes to differentiate between sources (e.g., TF_CODE, TF_STATE) for a clear picture of infrastructure intent vs. reality.

-> Query this graph to retrieve the dependencies before a change

Once the graph is built, we use Cypher, Neo4j's query language, to answer questions about the infrastructure downstream dependencies.

Step 1 : Make a change

We make a change on resource or a module. For instance expanding an IP range in a VPC CIDR.

Step 2 : Cypher query

We're going query the graph of dependencies through different cypher queries to see which downstream dependencies will be affected by this change, potentially in other IaC repositories. For instance this change can affect 2 ECS and 1 security group.

Step 3 : Give back the info in the PR

4. Current limitations:

To handle all the use cases, we are limited by the Cypher queries we define. We want to make it as generic as possible.

It only works with Terraform, and not other IaC frameworks (could work with Pulumi though)

Comments URL: https://news.ycombinator.com/item?id=42364276

Points: 1

# Comments: 1

For now we use weight & biases but personally I'm not a huge fan of their UX.

Do you have any other tools that you recommend ?

Thanks!

Comments URL: https://news.ycombinator.com/item?id=38865088

Points: 2

# Comments: 0

We have real issues defining the exact same environment, even if we have the same code/ use the same args with weight & biaises. We try to do our best with some versioning of the image datasets but it seems that the issue can also come from CUDA versions and the different libs (I also read it can come from the GPU being used).

If you have some tools that could help us replicable the same training environment that would be great. Not sure exactly what's wrong in our config.

Thanks!

Comments URL: https://news.ycombinator.com/item?id=38732606

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=38693235

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=38601307

Points: 3

# Comments: 3

I want to remove the hustle to clone repositories and install dependencies just for trying a project specific function.

The idea would be to host open source projects and allow developers to test any function in 2 sec in a similar way as with an IDE "debug" mode.

I am searching for beta testers ! Would love to have your feedbacks on that :)

Comments URL: https://news.ycombinator.com/item?id=38287678

Points: 5

# Comments: 2

I am thinking about solving a challenge that I faced recently when dealing with OS projects. I was struggling to implement some of the latest transformers and thought that there is a lack of a common format to navigate and test these projects without having to delve deep into the code.

So I am thinking of an open tool to attempt to 'standardize' the different OS projects. I though of something with a standard format, potentially built around modularizable blocks, that would allow contributors to easily plug into different OS repositories. The main objective would be to accelerate the learning curve/contribution to various projects.

What do you think?

I am still trying to figure out the different ways to implement it (your feedbacks are most welcome! especially if you have already faced such struggle and in what technical way) and if people would be ready to test a solution like that once implemented. I really think this is an issue but still trying to figure out the best way to solve it.

Thank for your time ! and really eager to hear your thoughts about that :)

Comments URL: https://news.ycombinator.com/item?id=36517873

Points: 4

# Comments: 2