Building Bridges: Navigating AWS Lambda’s Multi-Language Landscape with Docker

In a recent project, I encountered the need to seamlessly convert and re-encode an MP4 file into MP3 format, ensuring compatibility across various platforms. To tackle this conversion and re-encoding task, I found myself leveraging ffmpeg, a crucial tool that I had to ensure was seamlessly integrated into the execution environment. In this specific case, the environment was AWS Lambda, orchestrated within a Docker container.

Problem Statement: The application was written in nodejs and used a library called fluent-ffmpeg for convenience to interface with ffmpeg under the hood. However, this convenience came with a twist – the library mandated a Python version of 3.8 or higher. Ideally, I could make use of any AWS provided lambda runtimes to handle this task i.e Lambda nodejs specific base image or Lambda images for custom runtimes, but the node.js Image repository didn't include python version 3.8 or higher so yum install python3.8 or amazon-linux-extras enable python3.8 didn't cut it, likewise using and setting up a custom runtime image proved to be more challenging than expected.

Solution: Multi Stage Builds.

If you are familiar with Docker, you would be probably aware that multi stage builds enable us to reduce our final Docker image by retaining only the essential dependencies needed for our application to operate efficiently. Beyond just efficiency gains, this approach offers additional security advantages.

Let's walk through the Dockerfile that achieves this:

# Stage 1: Build Python environment
FROM public.ecr.aws/lambda/python:3.8 AS python-builder

# Install Python dependencies (if any)
# RUN pip install --upgrade pip && \
#     pip install <your-python-dependencies>

# Stage 2: Build Node.js environment
FROM public.ecr.aws/lambda/nodejs:16

# Copy Python executables from the builder stage
COPY --from=python-builder /var/lang/bin /var/lang/bin
COPY --from=python-builder /var/lang/lib /var/lang/lib

ENV LD_LIBRARY_PATH="/var/lang/lib:$LD_LIBRARY_PATH"

# Create symbolic links for convenience
RUN ln -sf /var/lang/bin/python3.8 /usr/local/bin/python3 && \
    ln -sf /var/lang/bin/pip3.8 /usr/local/bin/pip3

# Set the working directory
WORKDIR /var/task

# Copy Node.js application code and dependencies
COPY ./dist/ /var/task/
COPY package.* /var/task/

# Install npm production dependencies
RUN npm install --production

# Set the Lambda handler
CMD ["index.handler"]

Breaking Down the Dockerfile

Stage 1:

We start with the AWS Lambda Python 3.8 image as our builder stage or any Python version you need for your use case.
Here, you can install any Python dependencies your application requires.

Stage 2:

We switch to the AWS Lambda Node.js 16 image or your required Nodejs version.
We copy the Python executables and libraries from the Python builder stage to ensure Python 3.8 is available on the final image.
Symbolic links are created for ease of use, making python3 and pip3 point to the Python 3.8 versions.
Finally, we set the working directory and copy our Node.js application code and dependencies.

Using this approach, our application now functions fine as it has all the required dependencies. However, this is merely the tip of the iceberg when it comes to the potential of Lambda custom runtimes. The flexibility and extensibility offered by custom runtimes open up a realm of possibilities for tailoring your serverless environment to meet specific application requirements and preferences.