htvend is a tool to help you capture any internet dependencies needed in order to perform a task.

It builds a manifest of internet assets needed, which you can check-in with your project.

The idea being that this serves as an upstream package lock file for any asset type, and that you can re-use this to rebuild your application if the upstream assets are removed, or if you are without internet connectivity.

Has an experimental GitHub action to integrate within your GitHub build, archiving assets to S3.

[1] https://github.com/continusec/htvend

The intent was to support basic build systems accessing package eco-systems that tend to always serve the same response for the same URL.

Docker registries do this reasonably well, as do Maven repos.

It wasn't intended to be a full on proper archiving proxy (and I'll admit I hadn't heard that term - I'll look into it and see what else exists in that space).

The main use-case I had in mind for this is private projects, that are developed on workstations which have internet access, but are deployed to other environments using CI/CD systems with less network access. If both systems have access to a common blob store, then that can be populated with htvend build on a workstation and replayed at build time with htvend offline.

For that, there's no need to capture additional request information, as the focus wasn't to support getting a manifest file and being able to reliably re-download all the blobs from internet (and often those responses may have changed in the interim). And for the same reason, would expect to only need to one response per URL, per assets.json file.

Does that make sense?

That is, in a way that tightly controls which assets that they pull in so that things can be easily rebuilt without needing internet access, for situations including air-gapped networks, or simply a desire to not inadvertently bring in upstream changes while trying to make a small tweak to a private script.

It works by starting a local HTTP/HTTPS proxy server, then starting a subprocess with appropriate environment variables and certificate files set. It has special support for passing these into the RUN context for building of images, so that existing Dockerfiles can be used without modification.

Let me know what you think.

Comments URL: https://news.ycombinator.com/item?id=44775660

Points: 3

# Comments: 2

    func foo() (retErr error) {
        f, err := os.Create("out.txt")
        if err != nil {
            return fmt.Errorf("error opening file: %w", err)
        }
        defer func() {
            err := f.Close()
            if err != nil && retErr == nil {
                retErr = fmt.Errorf("error closing file: %w", err)
            }
        }()
        _, err = f.Write([]byte("hello world!"))
        return err
    }

Our team managed to screw-up some pretty major DNS due to a valid terraform plan that looked OK, but in reality then deleted a bunch of records, before failing (for some reason I can't remember) before it could create new ones.

And of course, we forgot that although we had shortened TTL on our records, the TTL on the parent records that I think get hit when no records are found were much longer, so we had a real bad afternoon. :)

To calculate ground speed (as required for navigation and fuel planning) you need true airspeed (not indicated) as well as wind direction and speed.

See some discussion here: https://www.quora.com/In-aviation-what-is-the-difference-bet...

Comments URL: https://news.ycombinator.com/item?id=19141788

Points: 2

# Comments: 0

ie if your code itself is split into modules, they won't work (as they are imported by their full path, not relatively), and anything in your vendor dir is also ignored when used outside of a GOPATH entry.

See RFC6962 Certificate Transparency logs and their consistency proofs for a widely used example.

There are ways to work around this, for example objecthash[0] describes a small modification that prepends the input data with 32 bytes of random data before hashing in order to prevent this.

[0] https://github.com/benlaurie/objecthash#redactability

One "trend", or rather bad habit that I've noticed a lot in discussion with other developers recently, and this doc also falls into, is that there seems to more focus on "input sanitisation" rather than "output escaping".

Regardless of what's been done to input, if the result is that you have a string that you need to embed into another string, then you need to know how to escape that appropriately for the context in which it's being used. Whether the data is user generated, or taken from your database, always assume that it's trying to break your app, and always escape it on output.

The trick is to be confident that you're getting the same hash as everyone else - and that's what requiring a proof that it be added to a CT logs gives you some level of assurance about.

[0] https://security.googleblog.com/2015/09/improved-digital-cer... [1] http://searchsecurity.techtarget.com/news/450411573/Certific... [2]

If FF is already doing any log inclusion proofs for certificates, then I think including one more (for the FF release itself) would be pretty much line noise.

I think an interesting question arises as to how well with the CT logs themselves would scale to handle the same kinds of certificates for all binaries, if this ends up taking off as a good idea in general. They've had to handle quite an explosion in X509 certificates over the past year or two due to Let's Encrypt. Some of Google's logs now show more than 80,000,000 certificates [0] in there - IIRC 2 years ago it was a low single digit million.

[0] https://crt.sh/monitored-logs

In that manner they can piggy-back on top of the CT ecosystem (including existing logs, including existing search / monitoring tools, and presumably gossip if/when that's solved).

This seems like a really cool hack! The state of binary software distribution is really pretty scary when you think about it - techniques like this have the potential to restore a lot of confidence.

[0] http://www.certificate-transparency.org/

It describes a consistent way to hash an object without defining a new format.

[0] https://www.certificate-transparency.org/