We don't really have that many constraints, despite that it should be cost efficient and save us from loosing all our data.

Object versioning: our data is mostly written only once, so we plan to enable object versioning to prevent overwrite attacks (or accidental overwrites).

Regional redundancy: we are considering moving all data to multi-regional buckets. This would help protect against natural disasters, fires, etc. It would not protect against systematic errors in Google Cloud Storage, though, so we are also considering an offsite backup (for example to AWS S3 Deep Glacier). Both options would come with significant upfront cost (15_000$ each). We are unsure whether we should choose both of these options, or only one (and which one).

Archival for external backup: about half of our data consists of frames extracted from videos (approx 300KiB each), and the other half are the original videos. Some of the frames have additional processing applied to them and reproduction from the videos is not fully deterministic, so we would prefer to keep the extracted files in the backup instead of recreating them if lost. However, the huge amount of small files (currently 200 Mio) causes significant cost for operations during backup, so we are considering adding a compression worker. This worker would take all the frames for a video and put them in a shared archive, which would then be sent to the backup bucket. This mechanism introduces an additional risk of failure, though, so we are also unsure about doing this.

Syncing between Google Cloud buckets and AWS S3: if we decide to go with backups to AWS S3, we would have to put in place a mechanism that keeps the original buckets in sync with the mirror on S3. Are there any good tools for this out there? If we had to roll our own, we would probably go for some sort of event-triggered serverless function to copy files when they are created / changed on the original bucket, combined with a regular full sync of files (similar to gsutil rsync).

Finally, disaster recovery should be quick but doesn't need to be instant (e.g. within 24 hours).

We're thankful for any input based on your experience!

Comments URL: https://news.ycombinator.com/item?id=39343620

Points: 1

# Comments: 5

When those changes include non-reversible database migrations, for example, it is better not to integrate that code into "mainline" before you've settled on a specific implementation.

I would still consider it very nice of them to offer this free service that let's you break free from your dynamic IP, crappy router and at the same time giving you protection that you couldn't set up yourself.

I think many people just want to be able to make their device available from the internet - this type of liberty is not really important for many people.

Regarding security: you might want to have a look at cloudflared. This is a (free for small projects) service by Cloudflare where your server (Raspberry Pi) connects to cloudflare, and all HTTP traffic is proxied through Cloudflare. This has multiple advantages:

- you don't need to open a port on your router that forwards to your Pi, which is good for security and simplicity (some routers don't even have this option)

- you get all the cloudflare protection with the click of a button, including HTTPS with automatic Cloudflare certificates, DDoS protection, etc

- you don't need DynDNS to point to your dynamically changing IP

Also, a lot of research in ML is done in Python, and there are quite a lot of startups doing ML, so Python is a natural choice. Same for data science.

A Harry Potter fan-fiction where Harry is a master of rational thinking and decision making. The plot is incredibly clever and exciting, and at the same time you learn a lot about how to think properly, but also a bit about science in general.

In the first chapters, I find Harry a bit annoying sometimes, but I beg you to ignore that and continue a bit, it gets so good!

Oh my! Perhaps they also polluted the search with search results relevant to your search term?

If you want to be completely anonymous on the web, use other means of browsing it. You can't visit other websites without them knowing where you come from if you are unprotected.

But perhaps I'm underestimating the stupidity of criminals - perhaps they'd just stick to unencrypted communication. However, if the usage of Tor for illicit activities is any hint, I'd say that is unlikely.

Yes, initially you might be able to catch a few bad guys who don't switch to the new service quick enough, but soon enough, all that is left is the ability for police & co. to read private messages of a population that is mostly innocent.