DevOps SRE Blog

Kubernetes port forwarding feature can be used to access a pod port from local machine without accessing pod. This can be achieved by other means like creating service. But port forwarding is somewhat quick. Port forwarding can be enabled by running following command: kubectl port-forward <pod Name> <local port>: <pod port> e.g., kubectl port-forward my-pod 9000:80 I have a test  k8s cluster with single master node and two worker nodes. I have enabled port forward by using above command and invoked curl by opening another ssh session from master node. vagrant@master-1:~$ curl http://localhost:9000 curl: (52) Empty reply from server I was expecting some output but got “Empty reply from server”. Also observed, in ssh session where port forwarding is enabled, below error message. E0811 13:51:54.597132   13384 portforward.go:391] an error occurred forwarding 9000 -> 80: error forwarding port 80 to pod ce0a72c043baa73a773b0803c9b3dc365229e...

Recently I started working on a new client engagement. This setup was new to us. I was troubleshooting some issue and thought of checking log files. I was able to find server.log and diagnostic log files but surprisingly server.out files were missing all together. Here are the steps I have followed to troubleshoot the issue: 1) Checked if the managed server was started from console (node manager) or using startup scripts. As you know that stdout logs will be written to nohup if started from commandline. Server was started from node manager, so this possibility was ruled out. 2) Compared with logging config of an env where server.out logs are present. Didn’t find any difference. 3) As it is dev env, thought of checking it quickly restarting server. Restarted server and I could see server.out file and it was getting updated. 4) Returned to my desk after lunch, surprisingly the log was gone again. 5) Checked lsof output for this process and found interesting thing as below: [myid@abc...

This blog covers basics of Vim/vi that are useful in day to day activities of a weblogic/fmw admin. Navigation Commands : h – Move cursor left j – Move cursor down k – Move cursor up l – Move cursor right 0 – Move to first character of real line gj/gk – Move down or up by display lines instead of real lines g0 – To first character of display line (when line spreads multiple lines, useful in navigating log files) gg – Go to beginning of the file G – Go to last line of the file 8G/8gg – Go to 8th line w – Move to start of the word e – Move to end of the word b – Move backward to start of the word Movement by finding characte r: f{char} – Move to the next occurrence of {char} F{char} – Move backward to the previous occurrence of {char} t{char} – Move to the character before the next occurrence of {char} T{char} – Move backward to the character after the previous occurrence of {char} *...

This is the final part of the series. Let us see some miscellaneous commands and tips. Hope this series helped you. 1) The utilities zcat, zgrep, zless, and zdiff, among others, serve the same purpose as cat, grep, less, and diff, respectively, but operate on compressed files. For example, if you want to search inside jar/zip file, you can use zgrep. 2)  apropos command can be used to find the appropriate command if we forget command name.  apropos search # gives all related commands with search string. 3) watch command allows to watch the output of a program change over time, which can be especially convenient when looking for changes to file sizes that reflect debugging output, download or file creation state, memory or disk usage, and so on. 4) History: If you want to remove a sensitive command from your history, you can simply edit your $HISTFILE history file and remove it. There is a trick you can use if you want to fly under the radar and never have a command recor...

Let us learn about tools that can help in troubleshooting performance related problems: 1) top : Top provides statistics about cpu, memory utilization, load averages, resident memory, virtual memory etc. The screen refreshes automatically.  i) top u oracle to see all the processes initiated by user oracle. ii) Press ‘1’ to see the individual cores. iii) top -p <pid> : To monitor only processes with a given process id. iv) top -H -p <pid>: Show all threads.We can press H on top output screen as well to get thread info. v) -b Batch mode. Useful for sending output from top to other programs or to a file. vi) To sort memory usage in ‘top’ output, press ‘SHIFT+m’  vii) List of process are sorted and displayed by default based on CPU utilization. We can change it by pressing “<” or “>”, sort column changes accordingly. top command consumes reasonable cpu. So, don’t run too many top sessions. 2) htop : htop is similar to top...

Let us learn about some miscellaneous commands in this part: 1) To find Linux version of OS: lsb_release -a 2) To find kernel version: uname -a 3) To find number cpu cores: lscpu cat /proc/cpuinfo 4)  more : cat command can be used to view the file contents. But if the file is more than few pages, reading through the buffer is difficult. The commands more and  less are more useful in these cases. We can go to next page by pressing space bar.  By pressing b we can go back by one page. We can search for a string by typing /, followed by the string similar to vi. We can find the next occurrence of the string by typing n. 5) less : less command has similar functionality as more but it is faster. 6) which   command can be used to find the location of a program. which perlwhich java 7) To read a file backwards, we can use tac (reverse cat) and less together: tac nohup.out | less 8) To find packages installed using rpm : rpm -qa | egrep "java-1.7.0-openjdk|java-1.8.0-openjdk 9) fuser i...

Let us learn about tar and ps in this part of the Unix series: tar : tar is derived from Tape Archive when magnetic tapes were used to archive files. tar utility is useful in creating single archive file from multiple files maintaining owners, file types, permissions etc. It is mainly used to create backups and copy files from one computer to other. Let us see few examples: 1) To create tar file from a dir: tar -cvf <tar file name> <Source dir name> #Here, c stands for compress, v for verbose, f - file name of tar file. tar -cvf scripts.tar /opt/oracle/product/admin/scripts 2) To extract files from tar file: tar xvf scripts.tar 3) tar doesn’t compress files, it only creates archive file. It can be compressed using gzip utility. gzip scripts.tar # It creates a compressed file scripts.tar.gz gzip -d scripts.tar.gz  # to unzip 4) We can archive and compress with single command using ‘z’ option: tar -zcvf scripts.tar.gz scripts # Compress tar -zxvf scripts...

Let us learn about df, du, sftp and scp commands in this part of the series: du : 1) To know the percentage of disk space used at mount level:  df -kh 2) To sort disk usage in a directory in human readable format: du -sh * | sort -rh 3) If multiple mounts uses common path. du command on common path includes storage used in all mounts. If we want find disk used by only common directory path and not device mounted on common path, need to use –exclude clause of du as shown above. du -sh --exclude=/directory {target-directory/*} 4) To find largest directories:  du -S | sort -n | tail -20 5) To find directories with size greater than 1 GB: du -h --max-depth 50 /orasoa/common/ 2>/dev/null | grep '[0-9]G>' | sort -h 6) du usually don’t include hidden directories/files during disk usgage calculation. To include hidden files/dirs: du -sk * .[a-zA-Z0-9]* 7) -d option prevents du from following file system boundaries. For example, to determine disk usage on the root fil...

This is in continuation to the first  part in the series. Let us discuss grep and lsof in part 2: grep : 1) Case insensitive search: grep -i "string" FILE   2) Checking for full words, not for sub-strings using grep -w grep -w "serachword" file 3) Displaying lines before/after/around the match using grep -A, -B and -C. This is very useful if you want to see lines around search string: i) grep -A <N> "string" FILENAME # display N lines after match: ii) grep -B <N> "string" FILENAME # display N lines before match iii) grep -C 2 "Example" demo_text # display N lines around match 4) Searching in all files recursively using grep -r grep -r "ramesh" * 5) Invert match using grep -v grep -v -e "pattern" -e "pattern" 6) Counting the number of matches using grep -c. There is no need to pipe the output to wc. grep -c "pattern" filename grep -v -c this demo_file (Count of lines not matching patter...

In this series of blog posts, I will provide useful Unix commands and tips that are useful in everyday sysadmin life in general and weblogic/oracle middleware admins in particular. I am writing these blogs based on personal notes gathered over years mostly from web. Unfortunately, I didn’t maintain source of the articles/websites. Extremely sorry for not giving due credits. Let us start the series with find command. I am dedicating whole blog post to find as it is versatile and useful. find command is used to search files/directories in a directory. Using find exec option e.g., renamme, copy. 1) To find all files and directories in a given directory and sub directories. find . # Finds all the files and directories in current directory.It searches current dir if none is specified find /abc/xyz # Finds all the files and directories in directory named /abc/xyz 2) We can limit search results only to either files or directories. find . -type f # limits the results to only files. f here...

strace is a Unix diagnostic tool used to trace system calls like read, write, connect etc. It comes handy for analyzing system hang and slowness issues. One of the key advantages is that we can use this tool without root access. But it comes with disadvantage too, it greatly slows down the process we are tracing. So don’t use it in production systems.    strace generates lot of output, at the beginning it will be quiet overwhelming. As you work on problems, you will get comfortable using the tool. Let us see some examples of strace.     1) To check what system calls a command is making:    strace <command> e.g., strace ls   2) If the process is already started and want to debug diagnose problems:    strace -p <pid>   3) We can save the output of strace to a file using -o option e.g., strace -o out.txt ls   4) To trace a bash script: strace -f -o trace.out bash <script name>   5) To trace specific system calls instead tracing all systems calls, we can use -e optio...

Scripts to monitor and managed Oracle Fusion Middleware applications are available at following github location:  https://github.com/RameshPoonati/oracleMiddlewareScripts Weblogic : Disable Diagnostic Modules Error Report Generation from logs Application/Deployment Monitoring http log monitoring JMS Bridge Monitoring JMS Consumer Monitoring Log Archiving JMS Queue Monitoring Server Monitoring Domain Startup/Shutdown Thread Dump Generation Thread Monitoring SOA Suite : Monitor Adapter Status Reference instance response time from audit trail ODI : Duplicate Execution Monitoring Oracle Service Bus (OSB) : OSB Proxy Status Change Shell Scripts : Disk Usage Monitoring

We had a SOA service which uses mail server to send notifications. One day, mail server went down and this triggered huge amount of log messages in server.out log file. Soon we got monitoring alert saying disk space is 90%. Out log files more than 1 GB size got created. We have had cleared lot of server.out log files. Surprisingly, it didn’t reclaim any space and disk utilization was keep on climbing.  We ran lsof command and grep for server.out file. [oracle@abcd logs]$ lsof -p 6913 | grep soa_server1.out lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing       Output information may be incomplete. java    6913 oracle    1w      REG              249,0 655131937   3283212 /u01/data/domains/soa_domain/servers/soa_server1/logs/soa_server1.out (deleted) java    6913 oracle    2w      REG            ...

This happened some time around Apr’2019. This story has all features of a suspense movie with a chaotic first half, surprise intermission/interval and fast paced unraveling second half.     Our client has Oracle EBS/ERP and it is integrated with other applications using Oracle middleware (java based) in Azure cloud. ERP admins clone ERP test environment from prod once in a month. Post cloning they do bunch of config updates to test environment. They also do a service re-deployment in our middleware application from ERP. This service is like a gateway to ERP application.   Architecture diagram is as shown below. ERP application has two nodes and is behind loadbalancer (LB). SSL is offloaded at LB. While servicing requests, middleware contacts ERP application via LB. During deployment, ERP send executable jar file to middleware, middleware does lot of validations including validation of ERP endpoint which is exposed through LB (https). If all validations are passed, middlew...

We have 100+ Oracle middleware servers that are running on Azure cloud. Most of these servers were not used during weekend. So, guys in infra team thought of saving cost by shutting down MW servers. They asked us  to provide a single script to shutdown all machines and another script to start them. They would invoke these scripts through Ansible and shutdown VMs once all processes on the VM are stopped. We came up with a wrapper script that will invoke individual component startup scripts. We have two node cluster and wrapper script is placed on node 1. Node 1 will connect to node 2 using ssh and execute individual component scripts on that node. The order is start managed server node manager -> OHS NM ->  Start OHS etc with delay of 30 seconds. But ssh session that starts managed server node manager never terminates. Found this answer on stackoverflow helped.  We added nohup and routed output to /dev/null. Snippet of the code: echo “`ssh $userName@$secondNodeIP /bin/bash ...

One of our Oracle SOA servers were crashing lately and Oracle requested for core dumps to analyze issue further. In hotspot crash report, following message is written: Not able to take core dump as core file size is set to zero. Below are steps followed to generate core dumps on JVM crash: Step 1: Increased ulimit using following command: ulimit -c unlimited > /dev/null 2>&1 Step 2: Restarted server. But no luck, still no core dumps are generated. It was not effective as it is applicable to only for the current shell session and not to new shell session. Step 3: Updated the value in .bashrc file ‘ulimit -c unlimited > /dev/null 2>&1′ . This made all new sessions’ ulimit to new value. Again started soa server, still ulimit value for core dump file size is zero. Step 4: This was not expected. On analysis found that, soa server is taking its ulimit values from its parent process which is node manager process as server was started via node manager. Restarted node manage...

ctrl-a – move the cursor to the beginning of the current line ctrl-e – move the cursor to the end of the current line alt-b – move the cursor backwards one word alt-f – move the cursor forward one word ctrl-k – delete from cursor to the end of the line ctrl-u – delete from cursor to the beginning of the line alt-d – delete the word in front of the cursor ctrl-w – delete the word behind of the cursor ctrl + xx in command line toggles between start and end of the line. ctrl + ] <char>  – moves cursor to 1st occurrence of <char> to right alt + .   – change command keep last argument. (Same can be achieved using !$) !* – change command keep all arguments