DevOps SRE Blog

  Expiring SSL certificates are one of the reasons for frequent production incidents and outages. One of the recent example is:  Microsoft Teams . In Java/Weblogic/SOA Suite world there are multiple places where certificates are stored: cacerts, jks trust & identity stores, KSS certificates, Load Balancer certificates etc. If your organization has license for monitoring tools like Oralce OEM, we can setup alerts easily. Our current client doesn’t have OEM. So, we have developed some custom scripts. These scripts can be found in   github .  Luckily, sample scripts for url certificate and jks certificate monitoring are available over internet. So, we modified little bit to suit our requirement. We were not able to find script for monitoring certificates stored OPSS/KSS. So, we have developed a wlst based script.

Usually asymmetric DR topology is (in which PROD and standby sites’ configurations differ) used to save cost of standby site. If number of instances at both sites are same but number of cpus differ, there won’t be any additional steps required during switchover. But if number of instances differ, switchover will fail due to missing configuration like machines and managed server etc. We got a 4 node weblogic cluster with restricted JRF (no daatbase) at production site and two 2 node cluster at standby site. We followed below additional steps to bring up weblogic domain at standby site. If you have SOA suite or webcenter etc the steps may differ. Removed MS3 and MS4 configuration from config.xml, entries between <server> tags. Removed node 3 and node 4 machine configuration from config.xml, Removed MS3 and MS4 from migratable targets section in config.xml Removed OHS3 and OHS4 configuration from system-components section in config.xml, entries between <system-component> tag. ...

Sometimes we as sysadmins, need to use Graphical User Interface (GUI) in addition most commonly used ssh terminals. The purpose may be to install software like oracle or accessing graphic applications that are installed only on remote machines. Unix based GUI technologies like X11 or VNC are always confusing to me and I used to rely on my colleagues. Recently, I got to install weblogic on Oracle Cloud Infrastructure (OCI). I invoked installer but no GUI window opened. My colleague, Rohit, configured everything on the VM for me. But this time I wanted to understand how these systems work under the cover so that I can troubleshoot these issues myself next time and save my colleagues’ time :-). In first part of the series, I will explain how to use X11 windows system for GUI in Oracle Cloud Infrasture (OCI). The same steps can be followed in any other Linux environment. In   second part , I will try to explain how things work behind the scenes. For accessing X11 GUI remotely we need t...

  We have two node weblogic domain with SOA, OSB, ESS and WSM clusters. Server on one of the nodes was going to FAILED state and was not returning to RUNNING state even after multiple restarts. Logs have following messages: Caused by: java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request "51,325,278,609,827,905". Underlying error is: "[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id "51,325,278,609,827,905" from server "wls_wsm1". Exception is: "files list is empty"." at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:92) at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:175)weblogic.deploy.internal.tar...

  This story goes back to Aug’20 when we got a new engagement with a client to manage their infrastructure. This infrastructure had been managed by different vendor on their platform. So, we had to make few changes like DNS server, network domain. We have implemented domain name changes on a Saturday for all SOA/weblogic application and everything went fine. Next Monday, business started using application hosted on weblogic fronteneded by OHS (collocated), users started getting 400 response code frequently. It was not 100% failure but some requests were going through. Someone in our team suggested restart of OHS. I tried to login to em console to restart OHS but log in was also failing with same 400 response code. I tried to restart using startComponent.sh script. It was failed due to misconfiguration of domain. The funnily named log files confused me further and I missed to notice right log file. All of a sudden I was under pressure as business was getting impacted. Luckily, my co...

We discussed how to open graphical applications in Linux in   Part 1 . In this second part, let us learn more about X11. This post is more of a personal notes for my future reference. For detailed technical information, please refer to relevant documentation. The X Windows system, also known as X11 or simply X, is a windowing system for Unix-like operating systems. X system uses client-server model. Here, X client is a remote application we want to access and X server is the program that displays windows in front of the user. This client–server terminology – the user’s terminal being the server and the applications being the clients – often confuses new X users, because the terms appear reversed. But X takes the perspective of the application, rather than that of the end-user: X provides display and I/O services to applications, so it is a server; applications use these services, thus they are clients. Unlike most earlier display protocols, X was specifically designed to ...

We got below error while configuring collocate OHS in weblogic domain without a database (restricted JRF): The Oracle HTTP Server (Collocated) [ohs] template could not be selected because of the following: CFGFWK-64079: Template conflict. The selection requires templates which conflicts with the current domain: Oracle Enterprise Manager : 12.2.1.3.0; Oracle JRF : 12.2.1.3.0; Common Infrastructure Engineering Runtime : 12.2.1; Oracle OPSS Metadata for JRF : 12.2.1.3; Oracle JRF : 12.2.1.3.0 CFGFWK-64071: Template conflict. The selection conflicts with templates already present in the domain: Oracle Restricted JRF : 12.2.1.3.0. Solution to this problem is to select  Oracle HTTP Server Restricted JRF  instead of Oracle HTTP Server (Collocated) as shown in below picture: If you are not aware of what is a Restriced JRF mode, here is snippet from oracle documentation: The Weblogic Server Domain in Restricted-JRF mode is similar in architecture and functionality to Weblogic Server Do...