Oracle MFT Error - Error occurred while polling the listening source

We can setup email notifications whenever an error occurs in MFT like failed connection to source or target etc. This is very useful feature of MFT. We had setup this monitoring for our MFT servers. We used to get below error frequently in production server while connecting to a 3rd party application.

 
Error occurred while polling the listening source XYZ
Cause: Unexpected error
occurred while polling the JCA source.
                
Action: Review the diagnostic log for the exact error, which begins with the
string 'ERROR:'. This error line, along with the preceding lines, indicates the
reason for this exception. Contact Oracle Support Services if the problem
persists.
The endpoint here was the target system to which MFT transfers file. We had setup retry at target so transfers were retried whenever this error occurs. So, it didn’t have direct impact on business. But we used to get around 100 alerts mails with these connection errors. It was tiring experience to read and check if this was real alert or false alarm.
 
We had setup a meeting with 3rd party application vendor and explained our scenario. They asked for sometime and came back that they didn’t find any issue from their end. Issue stopped after some time and all were happy. Unfortunately, it reared its head again after couple of weeks. We approached vendor and they told same thing. Issue used to exist for weeks and go away for weeks together.
 
We didn’t have direct contact with 3rd party vendor. It had to go through SME. Our SME for this area was known for perfection and he didn’t want to give up. He escalated to higher levels and everyone got into a call. We ran sftp commands on the call and around 30% of the sftp connections were failing. Which was same as failure rate we see normally. So we concluded that there was no issue with MFT.
 
With the information in hand and with experts on board, the vendor did real analysis for the first time. They came back and told us that they saw lot of failed login attempts to their test env. There was some common gateway hardware between test and production systems. These failed connections to test envs impacting common gateway and hence impacting production system too.
 
We were surprised that such a reputed vendor had such a setup. The troubling transfers were setup during POC stage and passwords got changed later. We should have cleaned these transfers immediately after POC. Anyhow, it was good lesson to everyone. We didn’t see these errors again after correcting passwords.
 
One question is still not answered, why issue used to absent for days together and reoccur again? The reason was test MFT servers used to go down due to OOM error (bug in MFT) and we used to restart them only when explicitly requested.???

Comments

Post a Comment

Popular posts from this blog

HOW WE REDUCED SOA OSB PROVISIONING FROM 4 DAYS TO 4 HOURS

 In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work Import of 10s of TLS Certificates into KSS Manual retiring and shutting down of services Monitoring setup Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-t...
Read more

NOT ABLE TO START RABBITMQ CLUSTER: CANNOT DECLARE A QUEUE ‘~S’ ON NODE ‘~S’: ~255P

We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error: 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”, 2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'” This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgi...
Read more

SOA SUITE 12.2.1.4 INSTALLATION: GOT EXCEPTION WHEN AUTO CONFIGURING THE SCHEMA COMPONENT(S) WITH DATA OBTAINED FROM SHADOW TABLE

  We planned to do out of place upgrade from soa suite 12.2.1.3 to 12.2.1.4. As part of this, we planned to automate end to end provisioning. While executing domain configuration using ansible and python scripts, we got following error in domain creation step. Got exception when auto configuring the schema component(s) with data obtained from shadow table. Failed to build JDBC Connection object: com.oracle.cie.domain.script.jython.CommandExceptionHandler. exception. Quick internet search gave few solutions like deleting domain-registry.xml file, verifying dehydration store password supplied to script etc. But none of them helped. After trying several options, I thought I might have done some stupid error prior to domain creation step. To track down the issue, though of checking what getDatabaseDefaults actually doing. I tried to grep for the getDatabaseDefaults function call inside oracle home. But couldn’t find the source code for the python function. As a final resort, thought of...
Read more