I think if you assume perf is normally distributed, you can still get some of the math to work out. But I will need to think more about this... if I ever choose this adventure, I'll post an update on https://github.com/hauntsaninja/git_bayesect/issues/25

(I really enjoy how many generalisations there are of this problem :-) )

One idea: if you always spend time testing equal to your constant overhead, I think you're guaranteed to be not more than 2x off optimal.

(and agreed with ajb on "just use ccache" in practice!)

This script in the repo https://github.com/hauntsaninja/git_bayesect/blob/main/scrip... will show you that a) the confidence level is calibrated, b) how quickly you get to that confidence level (on average, p50 and p95)

For the failure rates you describe, calibration.py shows that you should see much higher accuracy at 300 tests

IIUC the way you'd do that right now is just repeatedly recording the individual observations on a single commit, which effectively gives it a probability + the number of trials to do the Beta update. I don't yet have a CLI entrypoint to record a batch observation of (probability, num_trials), but it would be easy to add one

But ofc part of the magic is that git_bayesect's commit selection tells you how to be maximally sample efficient, so you'd only want to do a batch record if your test has high constant overhead

In addition to the repo linked in the title, I also wrote up a little bit of the math behind it here: https://hauntsaninja.github.io/git_bayesect.html

Comments URL: https://news.ycombinator.com/item?id=47557921

Points: 337

# Comments: 43

(I work at OpenAI)

Unknown is a pyright specific term for inferred Any that is used as the basis for enabling additional diagnostics prohibiting gradual typing.

Notably, this is quite different from TypeScript’s unknown, which is type safe.

A shared spec for this is important because if you write a Python library, you don’t want to have to write a different set of types for each Python type checker

Here are some things the spec has nothing to say about:

Inference

You don’t want to annotate every expression in your program. Type checkers have a lot of leeway here and this makes a huge difference to what it feels like to use a type checker.

For instance, mypy will complain about the following, but pyright will not (because it infers the types of unannotated collections as having Any):

  x = []
  x.append(1)
  x[0] + "oops"

Diagnostics

The spec has very little to say about what a type checker should do with all the information it has. Should it complain about unreachable code? Should it complain if you did `if foo` instead of `if foo()` because it’s always true? The line between type checker and linter is murky. Decisions here have nothing to do with “what does this annotation mean”, so are mostly out of scope from the spec.

Configuration

This makes a huge difference when adapting huge codebases to different levels of type checking. Also the defaults really matter, which can be tricky when Python type checkers serve so many different audiences.

Other things the spec doesn’t say anything about: error messages quality, editor integration, speed, long tail of UX issues, implementation of new type system features, plugins, type system extensions or special casing

And then of course there are things we would like to spec but haven’t yet!

The spec mostly concerns itself with the semantics of annotations, not diagnostics or inference. I don't really recommend using it as the basis for choosing a type checker.

(I was on the Python Typing Council and helped put together the spec, the conformance test suite, etc)

(I was on the Python Typing Council and helped put together the spec, the conformance test suite, etc)

(I work at OpenAI, I helped build this page and helped train text-davinci-001)