apache kafka – Create Flink Docker Container with pyflink
I’m new to Flink and not getting along with the documentation. I want to include Flink in my pipline, which consumes data from Kafka Topic to perform stream aggregations. I already tried several approaches but I’m either not able to submit the jobs, getting wrong results, errors during the build etc.
Can someone provide me with Dockerfile and Docker-Compose which build the required containers which are using pyflink and a python script for the stream consuming/aggregations etc? The goal should be that the job will be automatically submitted and starts running after the containers are build.
The Dockerfile provided in the documentation always throws an error during the build
FROM flink:latest
# install python3 and pip3
RUN apt-get update -y && apt-get install -y python3 python3-pip python3-dev && rm -rf /var/lib/apt/lists/*
RUN ln -s /usr/bin/python3 /usr/bin/python
# install PyFlink
COPY apache_flink*.tar.gz /
RUN pip3 install /apache_flink_libraries*.tar.gz && pip3 install /apache_flink*.tar.gz
Log message
=> ERROR [jobmanager 5/5] RUN pip3 install /apache_flink_libraries*.tar.gz && pip3 install /apache_flink*.tar.gz 1.1s
------
> [jobmanager 5/5] RUN pip3 install /apache_flink_libraries*.tar.gz && pip3 install /apache_flink*.tar.gz:
0.891 WARNING: Requirement '/apache_flink_libraries*.tar.gz' looks like a filename, but the file does not exist
0.909 Processing /apache_flink_libraries*.tar.gz
0.918 ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/apache_flink_libraries*.tar.gz'
0.918
------
failed to solve: process "/bin/sh -c pip3 install /apache_flink_libraries*.tar.gz && pip3 install /apache_flink*.tar.gz" did not complete successfully: exit code: 1
These are all the sources I tried to follow so far:
github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-python-example/Dockerfile
Read more here: Source link