Troubleshooting Essbase Agent Hangs on Windows Systems


The following steps should be taken when the Essbase Agent is in a hung/unresponsive state.  Recovery from the hang may result in a corrupt security file.  This Guide addresses periodically backing up the Security file.

There are 4 different areas that are covered:

  • Setting the debug for logs.
  • Taking mini dumps during the hang.
  • Taking java dumps.
  • Rebooting the machine.

Note – This is based on Essbase v11.1.2.x, however, can be used with previous versions. The pathing for files/logs will be different.

1. In the /Oracle/Middleware/user_projects/epmsystem1/EssbaseServer/essbaseserver1/bin/essbase.cfg file, add the following parameters:

AGENTLOGMESSAGELEVEL DEBUG
AGENTDISPLAYMESSAGELEVEL DEBUG
LOGINFAILUREMESSAGEDETAILED

2. In the /Oracle/Middleware/user_projects/epmsystem1/EssbaseServer/essbaseserver1/bin/logging.xml file in the section:

– <logger name=”oracle.EPMCSS” level=”NOTIFICATION:32″ useParentHandlers=”false”>
<handler name=”epmcss-handler” />
</logger>

Change the NOTIFICATION to TRACE:

– <logger name=”oracle.EPMCSS” level=”TRACE:32″ useParentHandlers=”false”>
<handler name=”epmcss-handler” />
</logger>

3. Restart Essbase after the changes have been made.

On the next occurrence of an Essbase agent or application hang:

1. Using ESSCMD or MaxL, try logging into the Essbase Agent. If there is no response from the Agent, it will appear to “hang”. Do not close this window.

2. Take a dump of the ESSBASE process:

a. Start TASK MANAGER.

b. Under the tab for PROCESSES select ESSBASE.exe process.

c. Make note of the Process ID (pid) that the ESSBASE.exe process is running on. If not displayed, Select View -> Select columns from the menu bar. Check PID (Process Identifier).

d. Right-click on ESSBASE.exe and select “Create Dump File”. When the dump is complete, it will show the path and filename of the dump file, for example C:\Users\EPMAdmin\AppData\Local\Temp\1\ESSBASE.dmp.

e. Repeat the above process for each ESSSVR.exe process. (Only needed for application hangs).

 

3.  Take  a Java dump of the ESSBASE process. (NOTE: Information taken from external site: https://access.redhat.com/solutions/19170)

After each ESSBASE dump file creation, get an Essbase Java stack trace:

a. Download PsTools from https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx and extract it. This file contains the PsExec utility.

b. Run the following in a cmd window from the location where you extracted PsExec (On Windows 2012, use “Run as administrator” to open the cmd window.)

PsExec.exe -s \Oracle\Middleware\jdk{version}\bin\jstack.exe -l {ESSBASE|ESSSVR process ID} > c:\temp\jstack_{pid}.txt

For example, if the ESSBASE process is run on process ID 3816:

PsExec.exe -s F:\Oracle\Middleware\jdk160_35\bin\jstack.exe -l 11196 > c:\temp\jstack_11196.txt

4. Check the MaxL/ESSCMD window. If the login is still not responding, repeat step 2-3 more couple of times with interval of 1-2 minutes. On each run, the ESSBASE.dmp filename will append so existing files will not be overwritten, for example, , ESSBASE (2).dmp, ESSBASE (3).dmp.

5. Note exact time of occurrence.

Attach the following:

Essbase.dmp files (should be 3)
jstack_{pid}.txt files (should be 3)
ESSBASE.log and ESSBASE_ODL.log with debug level messages.
essbase.cfg
opmn.xml
application.log (if needed)

 

After dump files have been attached:

1. Start TASK MANAGER
2. Start killing the ESSSVR processes, noting the PID as you kill them and checking the MaxL/ESSCMD Window after each. If killing the PID allows you to login, attach the log from that ESSSVR (application) process. The ESSSVR PIDs are located in the essbase.log. For example:

Application [Sample] started with process id [1928]

 

Other Windows 32-bit machines:

Install the following debugging tool on the Essbase server and run it against the ESSBASE.exe and ESSSVR.exe processes.  The debugger can be downloaded from Microsoft:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=E089CA41-6A87-40C8-BF69-28AC08570B7E&displaylang=en

After installing, from a ‘cmd’ window change to the x64 directory.  For example, C:\Tools\userdump8.1\x64.

1. Create the following directories on the same drive where the tool was installed.  These are for the output files:

minidumps_1
minidumps_2
minidumps_3

2. Run the following command:

userdump essbase.exe .\minidumps_1\essbasedmp1.mdmp
userdump esssvr.exe .\minidumps_1\esssvr1.mdmp

3.  Wait 5 minutes and run the dumps a second time, changing the output directory:

userdump essbase.exe .\minidumps_2\essbasedmp2.mdmp
userdump esssvr.exe .\minidumps_2\esssvr2.mdmp

4.  Wait 5 minutes and run the dumps a third time, changing the output directory:

userdump essbase.exe .\minidumps_3\essbasedmp3.mdmp
userdump esssvr.exe .\minidumps_3\esssvr3.mdmp

5.  Attach the following files to the Service Request:

  • All memory dump files from above commands, zip up each directory to separate files.
  • All application logs that were active at time of hang. These are located in the C:\Oracle\Middleware\user_projects\epmsystem2\diagnostics\logs\essbase\essbase_0\app\{appname} folder.
  • The Essbase Agent log, located in C:\Oracle\Middleware\user_projects\epmsystem2\diagnostics\logs\essbase\essbase_0\essbase.log.
  • The environment variables used.  From a ‘cmd’ window, type the following command, attach the envvars.txt file:
set > envvars.txt
  • The Essbase Configuration File, essbase.cfg, located in the C:\Oracle\Middleware\user_projects\epmsystem2\EssbaseServer\essbaseserver1\bin folder.
  • The Shared Services logs, SharedServices_Security_Client.log.  This file is usually located in C:\Oracle\Middleware\user_projects\epmsystem2\diagnostics\logs\essbase\essbase folder.

REBOOTING THE SYSTEM

1.  Before rebooting the system, bring up the Services Console:

  • select START -> RUN
  • type services.msc

2.  Locate the Essbase Server process, i.e. opmn_EPM_epmsystem2.

3.  Right click and select Properties.

4.  Change the Startup Type to Manual.

5.  Reboot the machine.

6.  Before starting Essbase, restore the last security file that was backed up in the Section “Backing Up The Essbase Security File”.

7.  Go back to the Services Console and start the Essbase service.

8.  Reset the service Startup Type to Automatic.

Thanks,

~KKT~

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s