Comments URL: https://news.ycombinator.com/item?id=48435371

Points: 146

# Comments: 203

Comments URL: https://news.ycombinator.com/item?id=48169521

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47412845

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47112183

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47016102

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46935307

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46027850

Points: 2

# Comments: 0

---

    class Merge(LLMDataModel):
        indexes: list[int] = Field(description="The indices of the clusters that are being merged.")
        relations: list[SubClassRelation] = Field(
            description="A list of superclass-subclass relations chosen from the existing two clusters in such a way that they merge."
        )
    
        @field_validator("indexes")
        @classmethod
        def is_binary(cls, v):
            if len(v) != 2:
                raise ValueError(
                    f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
                )
            return v
    
    
    class Bridge(LLMDataModel):
        indexes: list[int] = Field(description="The indices of the clusters that are being bridged.")
        relations: list[SubClassRelation] = Field(
            description="A list of new superclass-subclass relations used to bridge the two clusters from the ontology."
        )
    
        @field_validator("indexes")
        @classmethod
        def is_binary(cls, v):
            if len(v) != 2:
                raise ValueError(
                    f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
                )
            return v
    
    
    class Prune(LLMDataModel):
        indexes: list[int] = Field(description="The indices of the clusters that are being pruned.")
        classes: list[str] = Field(description="A list of classes that are being pruned from the ontology.")
    
        @field_validator("indexes")
        @classmethod
        def is_unary(cls, v):
            if len(v) > 1:
                raise ValueError(
                    f"Unary op error: Invalid number of clusters: {len(v)}. The prune operation requires exactly one cluster."
                )
            return v
    
    
    class Operation(LLMDataModel):
        type: Merge | Bridge | Prune = Field(description="The type of operation to perform.")

For instance, with temperature 1 there *could be* a path that satisfies your instruction which otherwise gets missed. There's interesting work here at the intersection of generative grammars and LLMs, where you can cast the problem as an FSM/PA automaton such that you only sample from that grammar with the LLM (you use something like logits_bias to turn off unwanted tokens and keep only those that define the grammar). You can define grammars with libs like lark or parsimonious, and this was how people solved JSON format with LLMs -- JSON is a formal grammar.

Contracts alleviate some of this through post validation, *as long as* you find a way to semantically encode your deterministic constraint.