python – How to Create Conda Environment in AWS Lambda at Run Time?

I am using an AWS lambda as part of my CI/CD pipeline. The lambda checks out some code using the provided git url and commit hash in the event. Then, it goes into the source code and does a conda env create -f env.yml, and then runs a python file python flow.py run. Unfortunately, installing and activating a conda environment is proving to be problematic as conda creates cache files in the /home/sbx_user directory, and AWS Lambda doesn’t have option to write to any direction except /tmp. I get the following error as an example:

Creating Conda environment 'torch_model'...
CondaError: Cannot write to condarc file at /home/sbx_user1051/.condarc
Caused by FileNotFoundError(2, 'No such file or directory')
CondaError: Cannot write to condarc file at /home/sbx_user1051/.condarc
Caused by FileNotFoundError(2, 'No such file or directory')
CondaError: Error encountered while attempting to create cache directory.
Directory: /home/sbx_user1051/.cache/conda/notices
Exception: [Errno 30] Read-only file system: '/home/sbx_user1051'

One thing I’ve also tried is setting HOME as /tmp in hopes that the conda environment and cache files would be installed in the tmp folder, which the Lambda does have write access to. But this attempt was unsuccessful as well and returned the following error:

environment variables:
conda info could not be constructed.
KeyError('pkgs_dirs')

Any help/guidance would be greatly appreciated!

Read more here: Source link