CONNECTION RESET ERROR – BOOMI CONNECTING TO SFTP SERVER

 This blog is about connection reset error we received while connecting from Boomi atom to remote sftp. This issue can occur for any client connecting to a remote sftp server.

The sftp server we were trying to connect was old sftp server and it has been used in many integrations. We haven’t faced reset issue for other integrations flow except this new one. So, we first suspected that there was something wrong with the process itself. We looked closely at the process. The process has multiple processes – one main process and multiple child processes. Main process checks for availability of files and child processes transfer files to Boomi atoms for further processing. The first main process and few child processes succeed always but last few process would fail always (usually after processing 7 or 8 files). We tweaked the process to drill down the issue. Here are few thing we tried:

1. Disabled all other processes that connect to the same sftp server. Issue still occurred.
2. We checked logs of individual atoms to check if this was occurring in any specific atom. It was happening on all nodes.
3. We tried with different file sizes: smaller and bigger. There was no difference. So, we ruled out that issue was not due bigger files.
4. We placed less number of files at source (less than 7 where processes started failing). When files were less, all processes were succeeded.
5. We added little delay of 5 seconds between the child processes. Surprisingly, all process got completed.

From points 4 and 5, we hypothesized that problem was not with client (Boomi) and it lies upstream: either network or sftp server. We captured tcpdump both at Boomi and remote sftp. Below is the screenshot from wireshark and we could see that remote host sent reset packet.

Wireshark Screen Grab

SFTP team analyzed and told that they were not sending reset packet. We took tcpudump multiple times and it was failing after 7 or 8 transfers and immediately after receiving “Key Exchange Init” packet.

During screenshare session with SFTP Admin, Network and Firewall teams, firewall team observed that connections were reset by a firewall policy. There was some policy like Open SSH Denial of service that got triggered. This policy was sending reset packets to both client and server as it was assuming that too many connections in short time could be the act of some malicious software. As it was trusted traffic and both source and destination are internal, firewall team raised issue with firewall vendor. Vendor told that as “Key Exchange Init” packet was too small to really understand traffic, to be on safter side, firewall policy was configured to block traffic. After adding except rule in firewall, our issue got resolved. We couldn’t have resolved this issue without team effort of disparate teams.