Discussion on dataprocessing layer
Pushing data into Aurora
People have used internal tools Data Flux, Data Path, that can help us push data from a batch path to tools like Aurora
We can use also Glue, and after version 10, Postgres introduced direct import from S3.
We should try to rely on the native tools
Two main datasets - ASIN Deep Dive, IDQ Deep Dive, conversion dashboard (Conversion dashboard has a request to have more ASINs on the dashboard)
Also have an open request for retail FastTrack
The idea is to set up the underlying infrastructure once, the effort should be very minimal to onboard a new metric
Q: Are we planning on using the Coral framework? A: We will not be using Coral, we will be using the native tools.
For the UI, we want to be reusing the existing business reporting, just duplicate the page, we don’t want to be making anything new for the front end
Q: With duplicating, do you mean that we have the same component in the page, or are we duplicating that somehow. A: You have two pages, business reporting and asin level reporting, just rename the names, but the layout and all the other options are the same. For the asin level reporting, we might need to think how to customize the columns that are shown in the report, because the existing thing only have static columns
JIYU Tech Spec: https://w.amazon.com/bin/view/Associates/JIYU/TechSpec/
Has a list of the pipelines, we can do some research Ripple: Distributed database, https://w.amazon.com/bin/view/BroadwayDataPlatform/BroadwayDataPlatform/Ripple
uid: 202007100900 tags: #meetings #amazon