Need Help with SFTP Error #6 using Environment Variables in Docker for Automated Data Transfer

GnuPG
1

Hey everyone,

I’m hitting a wall with an SFTP issue and could use some guidance. I’m working on automating a data transfer from Snowflake to an external partner via SFTP. The setup is running Python in a Docker container, designed to integrate with Airflow for daily automation. I’m using environment variables to securely manage sensitive info like SFTP credentials.

I’ve managed to replicate my production environment exactly in a local dev Docker container, running the same image, settings, and ect I keep getting this error:

ERROR - An unexpected error occurred during encryption: [Errno 6] No such device or address

Current Setup & Environment Variables

I’m loading critical info (like the SFTP host, port, username, and password) from environment variables. Here’s a quick look at how I’m doing that:

import os
import paramiko

SFTP_HOST = os.getenv(‘SFTP_HOST’)
SFTP_PORT = int(os.getenv(‘SFTP_PORT’, ‘22’))
SFTP_USERNAME = os.getenv(‘SFTP_USERNAME’)
SFTP_PASSWORD = os.getenv(‘SFTP_PASSWORD’)

Connecting to SFTP with Paramiko

I’m using Paramiko to handle the SFTP connection. Here’s the core setup for connecting and uploading:

try:
transport = paramiko.Transport((SFTP_HOST, SFTP_PORT))
transport.connect(username=SFTP_USERNAME, password=SFTP_PASSWORD)
sftp = paramiko.SFTPClient.from_transport(transport)

# File transfer logic
sftp.put(local_file_path, remote_file_path)

except paramiko.SFTPError as e:
print(f"SFTP Error: {e}")
finally:
sftp.close()
transport.close()

Docker & Airflow Environment

I’m running this script in Docker, which is orchestrated by Airflow in production. Here’s a snippet from the Dockerfile, showing how I’m loading the environment variables:

Dockerfile example

ENV SFTP_HOST=sftp.example.com
ENV SFTP_USERNAME=my_username
ENV SFTP_PASSWORD=my_password

Despite having an identical setup in both dev and production environments, the error only occurs in production. The specific error [Errno 6] No such device or address suggests an address or device issue, but since the credentials and connection details are the same, I’m at a loss for what could be different between the environments.

Has anyone encountered this kind of error with Paramiko or Docker, especially in Airflow-managed environments? Any insights or troubleshooting tips would be greatly appreciated!

2

Hi @Pythonnoob247,

can you say where GnuPG comes into play for your setup?
I’m asking because you’ve asked this in the GnuPG category.

Regards,
Bernhard

3
  1. Locating the GPG Binary: We first locate the GPG binary to ensure that the system can use GPG. The find_gpg_binary function checks standard paths to find an executable GPG binary.
gpg_paths = ['/usr/bin/gpg', '/opt/homebrew/bin/gpg']
gpg_binary = find_gpg_binary(gpg_paths)

if gpg_binary is None:
    logging.error("GPG binary not found.")
    exit(1)
  1. Initializing GPG: With the binary located, we initialize the gnupg.GPG object, which allows us to handle encryption and decryption within the script.
gpg = gnupg.GPG(gpg_binary=gpg_binary, use_agent=True)
logging.info("GPG initialized successfully.")
  1. Encrypting the Data File: After exporting data from Snowflake to a .txt file, we use GPG to encrypt this file before sending it over SFTP. This ensures the file remains secure and readable only to someone with the correct private key.
def encrypt_file():
    pgp_public_key = os.getenv('PGP_PUBLIC_KEY')  # Retrieve the PGP public key from an environment variable
    if pgp_public_key is None:
        logging.error("Environment variable 'PGP_PUBLIC_KEY' is not set.")
        return
    
    # Import the public key
    import_result = subprocess.run(
        [gpg_binary, '--import', public_key_file_path],
        capture_output=True,
        text=True
    )
    
    # Encrypt the file contents using the PGP public key
    result = subprocess.run(
        [gpg_binary, '--batch', '--yes', '--encrypt', '--recipient', key_id, '--output', ENCRYPTED_FILE_PATH, TXT_FILE_PATH],
        capture_output=True,
        text=True
    )
    
    if result.returncode != 0:
        logging.error(f"Encryption failed with return code {result.returncode}: {result.stderr}")
    else:
        logging.info("Encryption successful")
  1. Sending the Encrypted File over SFTP: With the data now encrypted, it’s safe to transfer over SFTP to the partner’s server. This way, even if the SFTP connection were compromised, the data itself would remain encrypted and protected.
1 Like
4

Suggestions:

  • do the same operation manually to see more debug messages. Also add --verbose so see more.
  • Add the display of stdout as well to your diagnostics
  • consider using the python wrapper to GPGME, which is the recommended interface to from python to GnuPG for automatic use. See https://wiki.gnupg.org/APIs

(I took the liberty to fix the formatting of your post).