Context
We are encountering an error [Errno 6] No such device or address when running a script in Airflow that involves GPG encryption. The script works fine in the development container but fails in the production container. Here’s a detailed summary of the issue and the steps taken so far:
Issue Description
- Error Message: [Errno 6] No such device or address
- Environment: Airflow running in Docker containers (Dev and Prod)
- Script: The script involves exporting data from Snowflake, encrypting it using GPG, and then sending it to an SFTP server.
Key Observations
- GPG Configuration:
- GPG home directory: /home/sbapp/.config/python-gnupg
- GPG binary: /usr/bin/gpg
- Warning in logs: gpg: WARNING: options in ‘/home/sbapp/.gnupg/gpg.conf’ are not yet active during this run
- Environment Variables:
- PGP_PUBLIC_KEY and GNUPGHOME are set correctly in the environment.
- File Paths and Permissions:
- The GPG home directory and its contents need to be readable and writable by the Airflow user.
- Permissions and ownership checks are performed in the script.
- Detailed Logging:
- Added detailed logging to capture environment variables, file paths, and permissions.
Steps Taken
- Added detailed logging to capture environment variables, file paths, and permissions.
- Verified Environment Variables:
- Ensured that PGP_PUBLIC_KEY and GNUPGHOME are set correctly in the Airflow environment.
- Checked File Paths and Permissions:
- Verified that the file paths used in the script are correct and accessible in the Airflow environment.
- Ensured that the GPG home directory and its contents are readable and writable by the Airflow user.
- Adjusted Directory Permissions:
- Set the correct ownership and permissions for the GPG home directory and its contents.
- Updated Script:
- Added detailed logging and error handling to the script.
- Ensured that the GPG object is initialized with ignore_homedir_permissions set to True.