amazon ec2 – Does Google Cloud Storage do parallelisation by itself?
Currently, I have a system that transcribes and summarises audio files with the Gemini API. Current configurations are those files that need to be processed, uploaded to DynamoDB, and then, through the system deployed in EC2, the files are processed.
The system needs to handle over 1000 file uploads. So, parallel processing and queuing are needed. If I change the project architecture to upload files directly to Google Cloud Storage (GCS) and process those files by connecting the Gemini API to the cloud project, will Google Cloud do the parallelisation by itself? According to my calculations, this architecture is cost-efficient too.
- If I do that, do I need to implement a backend that chunks files before uploading to the GCS?
- If I am creating a pipeline to upload files securely to the GCS, how should I implement it?
Read more here: Source link
