DevOps SRE Blog

I was configuring new datasource from admin console and got below error while activating datasource. Error An error occurred during activation of changes, please see the log for details. Error oracle.net.ns.NetException: Unknown host specified Error Substituted for the exception oracle.net.ns.NetException which lacks a String contructor, original message – Unknown host specified I have double checked datasource configuration and everything looked fine. My colleague, Vijay, suggested that there could be a problem with datasource other than current one. I went to jdbc config directory ($DOMAIN_HOME/config/jdbc directory and retrieved all hostnames using combination grep and awk commands. The suspicion came true, there was another hostname that was not getting resolved. We have untargeted datasoruces from all servers and activated changes and issue got resolved.

Oracle weblogic managed servers (SOA/OSB/ESS) were failing to restart after making few changes. The error written into logs was not straight forward and we spent couple of hours debugging the issue. Sharing this so that others can benefit from it. We are using Exadata as dehydration store. Network domain for applications we are managing got changed. As part of this change we changed datasource, ftp adapter config to refer to new domain. We made all required changes and tried to restart servers. We were able to bring up admin server but all managed servers failed to restart. Following error was written logs. Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request “42,336,427,916,066,804”. Underlying error is: “[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id “42,336,427,916,066,804...

Let us learn about SSL/TLS and how it applies to Weblogic and Fusion Middleware. 1) What is SSL/TLS? Transport Layer Secuity (TLS) and Secure Socket Layer (SSL) are cryptographic protocols used to securely communicate over a computer network. SSL was deprecated in 2014 after finding a vulnerability. TLS became new standard for secured http. TLS 1.3 is latest TLS version. 2) Why SSL/TLS? It is easy to intercept and read data transmitted in plain text over a network. Obviously we don’t like bank details, email or personal information fall in wrong hands. TLS protects from this by encrypting data between client and server.   3) How TLS works? TLS uses public key infrastructure to encrypt data exchange. TLS handshake at high level:      i)    Agree on version of TLS to use.      ii)   Agree on Cipher suites to use.      iii) Validate identity of the server using server certificate by client. In case of two way ...

There was an interesting issue that involved EBS, SOA and ODI with a client that I worked earlier. Customer is using Product Data Hub (PDH) application along with Integrated SOA Gateway (ISG) for publishing product details from master repository (EBS) to downstream systems.  Oracle SOA is used as integration layer and ODI is used to extract bulk data from PDH. All applications are deployed in Azure and Azure Application Gateway is used as Loadbalancer (LB). High level flow is as below: EBS Business Event -> SOA -> ODI PIM Webservice -> ODI Scenario -> EBS DB. Lately the customer has been observing stuck threads in SOA and ODI and response never comes back to SOA even after waiting for hours. I got interested in this problem, will let you know the reason at the end. We followed below procedure to find the root cause: The issue was replicated at will in lower envs. We observed stuck threads in both SOA and ODI when issue occurred. We took the thread dumps. We could see th...

We had an interesting problem over weekend and thought of sharing with all. One of my colleagues called me up and told that admin server in OSB cluster was not starting up and process was hanging. Here are the steps we followed to troubleshoot the issue. 1) He told that server had been up more than a week and he added JMX port before restarting the server. He was very confident that it was not causing issue. But I insisted to rollback this recent change and start again. Situation was not improved, server startup was hanging again. So, we rule out that this change was not causing issue. 2) We ran top command and checked if there was any pressure on resources. cpu, memory and load average were very low. 3) Checked log files for potential errors. Nothing useful information was found in logs too. 4) The symptoms were similar to recent issue I have worked on. So, we ran lsof command to find if server was waiting for any network connection ( lsof -a -i4 -i6 -itcp -p <pid> ).  lsof...

Recently I started working on a new client engagement. This setup was new to us. I was troubleshooting some issue and thought of checking log files. I was able to find server.log and diagnostic log files but surprisingly server.out files were missing all together. Here are the steps I have followed to troubleshoot the issue: 1) Checked if the managed server was started from console (node manager) or using startup scripts. As you know that stdout logs will be written to nohup if started from commandline. Server was started from node manager, so this possibility was ruled out. 2) Compared with logging config of an env where server.out logs are present. Didn’t find any difference. 3) As it is dev env, thought of checking it quickly restarting server. Restarted server and I could see server.out file and it was getting updated. 4) Returned to my desk after lunch, surprisingly the log was gone again. 5) Checked lsof output for this process and found interesting thing as below: [myid@abc...

This blog covers basics of Vim/vi that are useful in day to day activities of a weblogic/fmw admin. Navigation Commands : h – Move cursor left j – Move cursor down k – Move cursor up l – Move cursor right 0 – Move to first character of real line gj/gk – Move down or up by display lines instead of real lines g0 – To first character of display line (when line spreads multiple lines, useful in navigating log files) gg – Go to beginning of the file G – Go to last line of the file 8G/8gg – Go to 8th line w – Move to start of the word e – Move to end of the word b – Move backward to start of the word Movement by finding characte r: f{char} – Move to the next occurrence of {char} F{char} – Move backward to the previous occurrence of {char} t{char} – Move to the character before the next occurrence of {char} T{char} – Move backward to the character after the previous occurrence of {char} *...

Patching is a confusing topic for beginners. In this post, I will try to explain different terminologies. CPU – Short for Critical Patch Update. It comprises of security patches.   PSU – Short for Patch Set Updates. It includes security and priority fixes. Both patches are released quarterly on the Tuesday closest to the 17th day of the months of January, April, July and October. Both the patches are cumulative that means every patch includes fixes of the early patches. So, there is no need to apply earlier patches. Once a PSU has been applied on the system, the recommended method to apply all future CPU program security content is to apply future PSUs. In other words, once a PSU has been applied, it is not recommended to switch back to traditional N-apply patches. Reverting to CPU from PSU is complex and time consuming task and so it is not recommended. One-Off : If Oracle finds critical vulnerability that can’t wait till next patch release, it rel...

This is the final part of the series. Let us see some miscellaneous commands and tips. Hope this series helped you. 1) The utilities zcat, zgrep, zless, and zdiff, among others, serve the same purpose as cat, grep, less, and diff, respectively, but operate on compressed files. For example, if you want to search inside jar/zip file, you can use zgrep. 2)  apropos command can be used to find the appropriate command if we forget command name.  apropos search # gives all related commands with search string. 3) watch command allows to watch the output of a program change over time, which can be especially convenient when looking for changes to file sizes that reflect debugging output, download or file creation state, memory or disk usage, and so on. 4) History: If you want to remove a sensitive command from your history, you can simply edit your $HISTFILE history file and remove it. There is a trick you can use if you want to fly under the radar and never have a command recor...

Let us learn about tools that can help in troubleshooting performance related problems: 1) top : Top provides statistics about cpu, memory utilization, load averages, resident memory, virtual memory etc. The screen refreshes automatically.  i) top u oracle to see all the processes initiated by user oracle. ii) Press ‘1’ to see the individual cores. iii) top -p <pid> : To monitor only processes with a given process id. iv) top -H -p <pid>: Show all threads.We can press H on top output screen as well to get thread info. v) -b Batch mode. Useful for sending output from top to other programs or to a file. vi) To sort memory usage in ‘top’ output, press ‘SHIFT+m’  vii) List of process are sorted and displayed by default based on CPU utilization. We can change it by pressing “<” or “>”, sort column changes accordingly. top command consumes reasonable cpu. So, don’t run too many top sessions. 2) htop : htop is similar to top...

Let us learn about some miscellaneous commands in this part: 1) To find Linux version of OS: lsb_release -a 2) To find kernel version: uname -a 3) To find number cpu cores: lscpu cat /proc/cpuinfo 4)  more : cat command can be used to view the file contents. But if the file is more than few pages, reading through the buffer is difficult. The commands more and  less are more useful in these cases. We can go to next page by pressing space bar.  By pressing b we can go back by one page. We can search for a string by typing /, followed by the string similar to vi. We can find the next occurrence of the string by typing n. 5) less : less command has similar functionality as more but it is faster. 6) which   command can be used to find the location of a program. which perlwhich java 7) To read a file backwards, we can use tac (reverse cat) and less together: tac nohup.out | less 8) To find packages installed using rpm : rpm -qa | egrep "java-1.7.0-openjdk|java-1.8.0-openjdk 9) fuser i...

In this series of blog posts, I will provide useful Unix commands and tips that are useful in everyday sysadmin life in general and weblogic/oracle middleware admins in particular. I am writing these blogs based on personal notes gathered over years mostly from web. Unfortunately, I didn’t maintain source of the articles/websites. Extremely sorry for not giving due credits. Let us start the series with find command. I am dedicating whole blog post to find as it is versatile and useful. find command is used to search files/directories in a directory. Using find exec option e.g., renamme, copy. 1) To find all files and directories in a given directory and sub directories. find . # Finds all the files and directories in current directory.It searches current dir if none is specified find /abc/xyz # Finds all the files and directories in directory named /abc/xyz 2) We can limit search results only to either files or directories. find . -type f # limits the results to only files. f here...

strace is a Unix diagnostic tool used to trace system calls like read, write, connect etc. It comes handy for analyzing system hang and slowness issues. One of the key advantages is that we can use this tool without root access. But it comes with disadvantage too, it greatly slows down the process we are tracing. So don’t use it in production systems.    strace generates lot of output, at the beginning it will be quiet overwhelming. As you work on problems, you will get comfortable using the tool. Let us see some examples of strace.     1) To check what system calls a command is making:    strace <command> e.g., strace ls   2) If the process is already started and want to debug diagnose problems:    strace -p <pid>   3) We can save the output of strace to a file using -o option e.g., strace -o out.txt ls   4) To trace a bash script: strace -f -o trace.out bash <script name>   5) To trace specific system calls instead tracing all systems calls, we can use -e optio...

The first port of call for analyzing most of the issues is log files. Lot of beginners try to resolve an issue by looking at error message generated in application, console etc.That may not give you exact error always. So, it is better to look at log files as they provide lot more information to solve the problem. In this blog, let us look at different log files available in Weblogic/Middleware and how increase log levels. It is good practice is note down all log file locations of systems we are supporting and ways to increase log levels. Diagnostic Logs : Logs are written using ODL format. Most of the SOA, OSB and other application  built on top of weblogic are written to this log file. So, if you are working on any of those products, it is better to look at this log file first. The default log file location is: $DOMAIN_HOME/servers/$SERVER_NAME/logs. If the information written in the logs is not sufficient to anlayze the issue, we can increase logger levels from em console. We can al...

Scripts to monitor and managed Oracle Fusion Middleware applications are available at following github location:  https://github.com/RameshPoonati/oracleMiddlewareScripts Weblogic : Disable Diagnostic Modules Error Report Generation from logs Application/Deployment Monitoring http log monitoring JMS Bridge Monitoring JMS Consumer Monitoring Log Archiving JMS Queue Monitoring Server Monitoring Domain Startup/Shutdown Thread Dump Generation Thread Monitoring SOA Suite : Monitor Adapter Status Reference instance response time from audit trail ODI : Duplicate Execution Monitoring Oracle Service Bus (OSB) : OSB Proxy Status Change Shell Scripts : Disk Usage Monitoring

We had a SOA service which uses mail server to send notifications. One day, mail server went down and this triggered huge amount of log messages in server.out log file. Soon we got monitoring alert saying disk space is 90%. Out log files more than 1 GB size got created. We have had cleared lot of server.out log files. Surprisingly, it didn’t reclaim any space and disk utilization was keep on climbing.  We ran lsof command and grep for server.out file. [oracle@abcd logs]$ lsof -p 6913 | grep soa_server1.out lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing       Output information may be incomplete. java    6913 oracle    1w      REG              249,0 655131937   3283212 /u01/data/domains/soa_domain/servers/soa_server1/logs/soa_server1.out (deleted) java    6913 oracle    2w      REG            ...

There is a known bug in Oracle Managed File Transfer (MFT) which causes leaking of connections. This used to cause OOM error once in a month. We have an alert – which sends mail if heap usage is greater than 95% and 3 consecutive garbage collections not able to free much heap space. As we have two node cluster, we restart the server which reached heap utilization of more than 95%. On that particular day, the person on shift missed the alert. He checked alter after sometime and server health became ‘Not Reachable’. He tried to restart from console but he was not able to stop it and server status was showing as ‘RUNNING’. He tried to restart from command line even it didn’t work. He gave me a call. I faced similar issue when I was working for a different customer. So I was quickly able to identify issue.   Checked if there was any zombie process running (ps aux | grep Z). Yes, there was a zombie process. It was server process. Looks like serve...

Weblogic is nothing but a java process. So like any other java process we can take Thread and Heap dumps to analyze issues. Let us look at them one by one. Thread Dump : Thread dump gives a snapshot of threads at the time of taking dumps. It is primarily used to analyze performance and lock/resource contention issues. It is recommended to take 5-6 thread dumps in intervals of 10-15 seconds to understand if there is any issue. With single thread dump we won’t be able judge if it contention is momentarily or of long duration. There are multiple ways of taking thread dumps but my personal choice is jstack as we can take dumps even during most of the jvm hangs. 1) Using WLST : This method is not recommended as weblogic doesn’t respond during high loads. Also, wlst uses lot of resources. Following code snippet is taken from Oracle documentation: Place following code in a file code thread_dump.py. Please make changes according to your setup. serverName = ‘soa_server1’ ...

Our client uses both MuleSoft and Oracle Middleware integrations. There was a requirement for MuleSoft to connect to Oracle Weblogic JMS queue and read messages.  MuleSoft and Weblogic are in two different clouds. MuleSoft was using Weblogic full client and t3 protocol to connect to JMS queue. Also, they were using ip address in connection url as they had limitations using DNS. Mule was getting timeouts when they try to connect to weblogic JMS.  We all got into call to troubleshoot as this issue was hanging for a while. Our infra admin quickly took tcp dump and confirmed that request from Mule is reaching Weblogic VM. We have verified Weblogic configuration and noticed that Weblogic server was listening on host name but Mule was using ip address. We quickly changed listen address to ip address. Then, MuleSoft was able to connect JMS queues. Latter after resolving DNS issue, MuleSoft team used host name (Weblogic listen address too changed to host name). Mule was able to connect to ...

We updated DB Adapter outbound connection pool configuration and tried to deploy it and got below error: <20-Apr-2018 11:18:21 o’clock BST> <Failed to initialize the application “DbAdapter” due to error weblogic.application.ModuleException: weblogic.connector.exception.RAException: JNDI name is null or empty. weblogic.application.ModuleException: weblogic.connector.exception.RAException: JNDI name is null or empty.         at weblogic.application.internal.ExtensibleModuleWrapper.prepare(ExtensibleModuleWrapper.java:114)         at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)  Root Cause: Save outbound connection pool without XA datasource JNDI name and deployed with new plan. Resolution: Used below command to find entries without JNDI name and deleted thsoe entries. grep -A 1 “ JNDIName ” DB_Plan.xml | more