I really appreciate this attitude towards this because it recognizes that there are a lot of security teams out there that don't take a nuance view of CVEs. For instance, one time we had a security team that required us to patch a vmware support package that was installed by default on ubuntu, but the CVE required being ran on vmware when we were running on EC2. Arguing with them was pointless because they were not interested in determining if the CVE applied to us, only that it needed fixed.

Lots of teams that are supposed to be in charge of security don't ask "does this CVE affect us", but simply shift the burden of patching downward and outward. In some cases, like in the case of easy to update and centrally deploy SaaS products, that burden is more annoying and frustrating than difficult. In some cases, like when you have complicated deploy or have customer-controlled updates, those mandates cause a huge burden on teams not producing the decision to patch every low CVE.

We had pretty good luck in the end, but getting there produced a somewhat large app on our side that would manage the whole process, including but not limited to asking for multiple responses, comparing them to each other, finding consensus, and determining which users would consistently produce bad responses and stop them from responding. We got to a confidence that about 85-95% of the data was correct, which was good enough for the company.

Through that process, I learned a couple things about managing mturk, primarily about how changes to the cost-per-task would change the process. Initially, we thought that price would be a quality knob, but quickly learned that price was a speed know. The higher the price the faster the tasks would be taken and completed. Quality did not change significantly as the price went up or down.

Overall, I still have fondness to mturk, but it was really bare-bones experience that needed a lot of work to get working effectively.

We're talking about a few-hundred megabytes of data for all of the customers that these reports pull, but that's also for the past 15 years. We do have like 25k customers, which shrinks how much a customer can pull in even further. One last point is that we already de-normalize the report data into its own table specifically for these reports, so that's not something the data warehouse is doing for us.

I agree with your experience with QuickSight, it is exactly my experience. My preference is to continue using the reports we generate in the app, but I'm trying to wrap my head around cases where this ends up being the better direction.

From what I've been able to gather, the data goes from the production MySQL database to a secondary MySQL database using DMS. Then come the Glue jobs that ship the data out to a data lake in S3. After that there are several transformation jobs that I've been told convert the data into a "canonical" form, smoothing out all the differences between verticals. I think they said that next the data goes into a second data lake and has additional transformations performed. Finally the entire process gets the data to its final resting place in Redshift where QuickSight is used to create reports. I'm fairly certain I missed a couple steps because I just couldn't figure out the purpose of each step as they were describing the process.

Getting reports out of that process seems painful. Showing a report for an internal customer (sales or customer support for instance) means they need a QuickSight account and access to the specific report. Getting access to that for myself was not straightforward, which makes me think it is hand-managed by a dev.

For showing a report in product it feels worse. First the data team are about the only people that can create these reports because not only do the product devs not know this "canonical" form, but getting the development environment running consistently for product devs has been like pulling teeth. Once someone has written the report, they have to promote the report by copying it exactly, including an identical report id, to another region. Finally the report id is given to the product team to put into the product. Adding the report id to the product is the easiest part, but the data journey doesn't stop there. The product has to pass that report id and user information to a lambda the data team maintains that generates a URL for the product to embed with an iframe. And after all of that, the report doesn't come close to matching the look of the site.

Is this data warehouse setup normal? Is this a common way to handle in-product reports after a company invests in a data warehouse? There are a lot of what seem like redundant steps, as well as a lot of custom code for what I would expect to be built into these products.

Comments URL: https://news.ycombinator.com/item?id=48556530

Points: 4

# Comments: 7

The problem is that once I asked it "I'm thinking about A or B" twice, once with "I like A more but suspect B would be best" and a second time with them reversed. Not surprisingly, both times it chose the one I said I suspected was best as it's honest opinion.

I think its assumed in the LLM model business that the models themselves are not a good moat, the next model by another company is just as likely to be as good as the current model. So companies like Anthropic have to tighten the noose slowly to start recovering their costs. This appears to be one of those steps.